Coherent optical orthogonal frequency division multiplexing (CO-OFDM) system is considered as a promising technology for high speed optical transmission. Polarization dependent loss (PDL) and polarization mode dispersion (PMD) influence system performance seriously in high speed transmission. The mathematical model and mechanism of transmission affected by first-order and second-order PMD and PDL are presented. Analysis and simulation results show that first-order PMD adds cosine factors for different OFDM subcarriers and in second-order PMD, CO-OFDM system performance is mainly influenced by depolarization rate which can moderate phase noise induced by chromatic dispersion. In addition, PMD can not only make subcarrier polarization strengthen with the increase of frequency, but also moderate noise induced by PDL. At the fiber link of 720 km, loss factor of 0.5 and without chromatic dispersion consideration, system Q value is improved about 2 dB over the case of no second-order PMD.