A novel method of subcarrier evolution was proposed to analyze the nonlinearity of an intensity-modulation direct-detection(IMDD) orthogonal frequency division multiplexing(OFDM) passive optical networks(PON). The nonlinearity was mainly caused by the subcarrier-to-subcarrier intermixing interference(SSII) and the chirp from the modulation. The fiber disperssion results in that the optical subcarrier can not keep the best initial phase matching condition which exacerbates the signal distortion. A new modified Volterra digital filter was also proposed to cancel the signal distortion of the IMDD OFDM PON. The modified Volterra digital filter included not only the same samples but also the different samples. The corresponding experiments had been carried out to verify the effectiveness of the proposed signal distortion cancelation method. The proposed modified Volterra filter could bring about 3 dB power benefits than the previous work to meet the forward error correction(FEC) limit of 10-3. The transmission penalty over 25 km at the FEC limit of 10-3 was lower than 0.5 dB. Finally, the system can reach a data rate of 24.5 Gbps, and the spectrum efficiency can achieve up to 7 bits·s-1·Hz-1. These experimental results prove that the modified Volterra filter can effectively compensate the linear and nonlinear distortion and therefore it can recover the signal at optical network unit (ONU).