The error vector magnitude (EVM) and bit error rate (BER) performance of visible light communication (VLC) system will be deteriorated by the nonlinear characteristic of light emitting diode (LED). In this paper, the principle of optical orthogonal frequency division multiplexing (O-OFDM) VLC system based on nonlinear LED is introduced, and the BER expression of the system is derived. It is proposed to divide the LED nonlinear distortion into nonlinear distortion in the working area and clipping distortion outside the working area. For the clipping distortion, the subcarrier attenuation factor of the load information is calculated according to Bussgang theory, and then the clipping noise variance superimposed on the subcarrier is obtained. The nonlinear distortion in the working area is the equivalent to the driving signal plus the nonlinear distortion noise, and use the total average power of the nonlinear LED output deviates from the ideal linear signal minus direct current power as the noise variance. The influence of nonlinear distortion on constellation, EVM, and BER is analyzed by Monte Carlo simulation.