In order to improve the phase noise tracking performance of coherent fiber communication system, a phase noise tracking algorithm——modified decision-directed recursive least square(DD-RLS) algorithm has been proposed. The modified DD-RLS algorithm and nonlinear pre-compensation algorithm are combined to suppress cross-phase modulation(XPM) in WDM coherent fiber system. Transmission simulation shows that the algorithm can effectively suppress the XPM effect in the system, it can track the nonlinear phase noise in coherent wavelength division multiplexing(WDM) optical fiber communication systems more efficiently than the traditional DD-RLS algorithm, and can improve the Q2 factor of the signal by 0.8 dB comparing with the traditional algorithm in a 1000 km transmission simulation with 11 channels.