Four-wave mixing (FWM) is the most important nonlinear effect in coherent optical orthogonal frequency-division multiplexing (CO-OFDM) system. The principle of CO-OFDM system based on wavelet transforms (WT-OOFDM) is analyzed, and effect of wavelet transforms on phase noise caused by FWM and amplified spontaneous emission (ASE) of optical amplifiers is discussed. For the conventional optical fiber G. 652, 100 Gbit/s WT-OOFDM system, the simulation results show that the variance of the total phase noise can be decreased by 15%, and the nonlinear phase noise has a 20% reduction while the transmission distance reaches 1500 km.