Due to the high peak to average power ratio in coherent optical orthogonal frequency division multiplexing system, a coherent optical orthogonal frequency division multiplexing system using the wavelet packet transform instead of the traditional fast Fourier transform was proposed. Simulation results show that the novel transformation has obvious inhibiting effect on peak to average power ratioand and gets better bit error rate performance than the traditional system using fast Fourier transform. With symmetry of the wavelet scaling function and wavelet function improving, the ability to avoid the phase shift improves in the process of signal processing and system performance ameliorates.At the complementary cumulative distribution function of 0.01, peak to average power ratio of using haar wavelet function can be optimized 2 dB and the optical signal to noise tatio of haar is improved by 1.5 dB at bit erro rate of 10-3 compared with that of traditional fast Fourier transform. On the basis of research, with the simpler algorithm structure, lifting wavelet transform instead of Mallat algorithm was proposed to apply to the coherent optical orthogonal frequency division multiplexing system system and the inhibition peak to average power ratio and bit erro rate performance is similar to the latter system.