The illumination based traditional source layout has the problem of uneven illumination, which results in blind effect of communication and greatly affects the reliability of communication system. In order to solve these problems, a 4 m×4 m×3 m room is taken as the model, and the illumination compensation technology is used in the common indoor light source layout to optimize the layout. A kind of light source layout which is composed of five light-emitting diode arrays is obtained. With this kind of layout, the power consumption of the system can be reduced, and at the same time the uniformity of illumination can be improved. In order to balance the reliability of visible light communication (VLC) system, the indoor illumination standard deviation and the average bit error rate (BER) on receiving plane in communication system are used to build a system optimization model function f(L,i). When f(L,i) reaches the minimum value, it can meet the requirements of the receiving plane illumination and the BER. The simulation results show that the f(L,i) can reach the minimum value when L=0.35 m and i=0.025 m. For the receiving plane, the minimum illumination value is 301.26 lx, the maximum illumination value is 389.90 lx, and the illumination uniformity is 93.24%. At the same time, the illumination standard deviation is 20.1, the power consumption is 140.5 W, and the BER is 6.39×10-7. The proposed system takes the uniformity of indoor illumination distribution on receiving plane and the communication reliability into account, which provides an optimization method for the light source layout of indoor VLC.