In order to improve the measurement precision of the homodyne laser interferometer for vibration measurement, we propose a real-time active nonlinear compensation method based on the pseudo-extremums. According to this method, the pseudo-extremums instead of the extremums are used to determinate the direct current (DC) offset and alternating current (AC) amplitude difference. Then the quadrature phase shift is suppressed by the operation for vector phase correction. Experimental results show that the method proposed can reduce the error caused by the bit error of digital signal transmission, laser power drift and Abbe error during measurement. Laser power drift and Abbe error make the elliptic trajectory of quadrature signals into a spiral shape. The peak-to-valley amplitude of residual error can be reduced to 0.8 nm by using the pseudo-extremums method with the operation for vector phase correction, which has a more effective suppression for nonlinear error than the conventional extremums method. Besides, the method proposed does not need complex calculation and remains good instantaneity.