Frequency sweeping interferometer is very sensitive to drift of optical path difference. Drift error is multiplied by an amplification factor to thousands of times, seriously affecting the measurement results, so it is necessary to remove or reduce drift error. For low-frequency vibration or slow drift of objective mirror under laboratory environment, the amplified drift errors of two consecutive forward and reverse scanning measurements are approximately equal in quantity but opposite in sign. According to these characteristics, a drift error compensation method by consecutive forward and reverse optical frequency scanning is proposed. Then, validation experiment is established to prove the feasibility of this method for vibration of objective mirror with frequency of 4.7 Hz and amplitude of 1 μm. The experimental results show that the standard deviation of 40 measurements decreases from 51.9 μm to 8 μm with distances up to 1543.3 mm, after applying this compensation method.