Dense wavelength division multiplexing system is a final selection as an optical communication system because of its high speed and big capacities, but the fiber nonlinear effects and polarization mode dispersion limit severely the performance of the system when signal propagates at 40 Gbit/s in a single channel. The coupled nonlinear Schrodinger equations of a single channel in dense wavelength division multiplexing system are derived, considering all factors of group velocity dispersion, self-phase modulation, cross phase modulation, four wave mixing and polarization mode dispersion, and their number results are obtained with extended Split-step Fourier Method. To analyze the impacts of the fiber nonlinear effects and polarization mode dispersion on the optical communication system, the simulated results of a 8×40 Gbit/s dense wavelength division multiplexing system are discussed under different conditions respectively at last. It is found that nonlinear effects influence dense wavelength division multiplexing system more severely than single wave-length system because of the effects of cross phase modulation and four wave mixing. The degree of impacts of polarization mode dispersion and nonlinear effects on system is related to the power of the input signal. When the average power of the input signal is below 0.1 mW, the influence of polarization mode dispersion is dominating; while it is 1 mW, the impact of nonlinear effects is worse than polarization mode dispersion. But polarization mode dispersion, which can widen pulse and is similar to the small chromatic dispersion in G.655 fiber, can mitigate the impact of nonlinear effects.