A mathematical model of the crosstalk in polarization division multiplexing (PDM) system is induced. An adaptive crosstalk elimination scheme is proposed, in which the radio-frequency (RF) power of the optical signal related with the crosstalk is used as feedback signal and particle swarm optimization (PSO) algorithm is applied in the control unit to adjust the polarization controller and eliminate the crosstalk. The effectiveness is demonstrated in 2×50 Gb/s polarization division multiplexing-differential quadrature phase shift keying (PDM-DQPSK) system. The results show that with the crosstalk elimination scheme the performance of the system is improved remarkably and the bit error rate is reduced greatly.