In order to solve the problem of multi-user interference and the subchannel strength generated by the block diagonalization (BD) algorithm in multi-user MIMO (MU-MIMO) indoor visible light communication, the bit error rate of the indoor MU-MIMO visible light communication system is optimized by using the substream selected BD algorithm. This paper establishes the channel model for MU-MIMO indoor visible light communication and compares the channel spatial correlation between the 4×4 MIMO and 8×8 MIMO in different indoor system layout modes by using the control variable method and taking different parameters of LED and PD distance, the system capacity and bit error rate performance of substream selected BD algorithm and BD algorithm are compared and analyzed. The results show that with the continuous enhancement of spatial correlation, the bit error rate performance decreases, and the substream selected BD algorithm can bring a gain of more than 4 dB compared with BD algorithm.