A digital signal proccessing (DSP) algorithm, nonlinear compensation based on Manakov equation, was applied in coherent 40Gbit/s polarization-division multiplexing quadrature phase-shift keying (PDM-QPSK) transmission systems to compensate the nonlinear impairment in optical fiber. The back propagation algorithm based on Manakov equation can reduce the interaction between nonlinear impairment caused by Kerr effect and polarization mode dispersion (PMD) while the traditional back propagation based on scalar nonlinear Schrdinger equation (NLSE) neglects the PMD and cannot compensate the impairment caused by PMD in the polarization-division multiplexing transmission systems. The perfomance of nonlinear compensation based on Manakov equation was verified. Both simulation and experimental results show that the Manakov equation based algorithm exhibits better performance of 400 km-long QPSK transmission comparing with NLSE, with a Q-factor improvement of approximately 3 dB at an OSNR of 18 dB.