The transmission of signal in optical fiber is affected by Kerr nonlinear damage, resulting in nonlinear spectrum broadening effect, resulting in information leakage out of band and incomplete information in the receiving end. The traditional nonlinear compensation method processes the signal at the receiving end by inverting the channel transfer function, but the effect is not ideal. In order to solve this problem, first, the optimization algorithm is used to find out the out of band adjoint signal which can compress the original signal to another Nyquist bandwidth in the digital reverse channel transmission; Second, the compressed Nyquist signal is sent in the sender; Finally, the original signal is recovered by Nyquist filtering at the receiving end. Simulation results show that the performance of the proposed algorithm is better than that of reverse transmission method in the band-limited systems, and the gain of the error vector amplitude is 3.17 dB in the standard single-mode fiber with the length of 800 km.