Sinusoidal phase modulating technique is introduced in to the self-mixing interference in order to improve the displacement measurement accuracy of the self-mixing interferometer. Phase modulation is obtained by an electro-optic crystal in the external cavity and phase demodulation is achieved by Fourier analysis method. Error sources that can influence the displacement measurement accuracy such as modulating instability of the electro-optic crystal (EOM) and the multiple feedbacks effect in the external cavity have been simulated and the measurement accuracy of the new signal processing method is obtained theoretically. Experimentally, calibration results with a high precision commercial PZT show that a displacement measurement accuracy of nanometers scale is achieved in the common laboratorial environment.