The nonlinear distortion caused by the Mach-Zehnder modulator (MZM) is one of the main limiting factors for the improvement of the transmission performance of the intensity modulation and direct detection (IM/DD) optical communication system. In order to solve the problem, an improved Volterra nonlinear equalization (VNLE) method is proposed. Compared with the traditional VNLE that uses the least mean square (LMS) to calculate the tap coefficients, the improved VNLE uses the least square (LS) method to obtain more stable convergence. The simulation results show that the VNLE based on LS has better performance when solving complex nonlinear damage. For 25 Gbaud 4-level pulse amplitude modulation (PAM4) signals, the improved VNLE can reduce the bit error rate (BER) to below 10-4 in a 7-km-long single-mode optical fiber transmission system. In addition, in order to make the BER below 10-3, the transmission distance that the improved VNLE can withstand is about 1.5 km longer than that of the traditional VNLE.