In this paper, a frequency-domain mathematical model is deduced for polarization-division multiplexing (PDM) coherent optical orthogonal frequency division multiplexing (CO-OFDM) system with transceiver in-phase and quadrature (IQ) imbalances. A novel training symbol structure is designed in which the mirror-subcarriers of pilot subcarriers were modulated with zero signal so that the channel distortion with transceiver IQ imbalances can be estimated. It proves that the channel distortion and transmitter IQ imbalances cannot be separated using the training symbols; therefore, the channel equalization method was used to recover the signals. Simulation results show that at 2 dB transmitter amplitude imbalances, 15° transmitter phase imbalances, 100 Gbit/s transmission rate, 1 040 km standard signal- mode fiber link, and 1×10^-3 bit error rate, 23.5 dB optical signal-to-noise ratio (OSNR) is necessary for the proposed method. However, the compared schemes cannot achieve effective transmission.