The optical fiber transmission model, i.e., the nonlinear Schrodinger equation, is solved by split-step Fourier algorithm to research the effects of fiber-optical dispersion and nonlinearity on a new closed-loop dual-directionally chaotic secure communication system with three lasers. The effects of the actions of dispersion, nonlinearing, and combination action of dispersion and nonlinearity on the chaotic signal and the demodulated message quality are analyzed as the optical fiber transmission distance increases. The results show that: the dispersion leads to the broadening of chaotic signal, and the increase of broadening ratio, and reduces the synchronization performance of chaotic signal with the optical fiber transmission distance increasing, which results in the degradation of the demodulated message quality. Nonlinearity does not affect the time-domain waveform of chaotic signals, but the combination effects with dispersion make the chaotic signal change rapidly, as a result, the synchronization performance of chaotic signals varies rapidly, and the demodulated message quality sharply deteriorates.