A new similarity demodulation method in fiber optic sensing system based on a frequency-swept laser and a Mach-Zehnder interference structure is proposed and demonstrated. The method is designed specifically to overcome the difficulty due to the finite frequency steps of the frequency-swept laser, which is not fine enough and will result in inaccurate demodulation result of the interference arm length difference. The method relies on the construction of a simulated function by finely subdivide in the frequency step range, and then compare it with the real interference signal, the simulated interference arm length difference corresponding to the maximum similarity is considered closest to the true result. The feasibility of the method is verified by simulation and experiments, which show that the demodulation range of the interference arm length difference of the system is 2~17 mm, and the minimum resolution of the demodulation result over this entire range is 1 μm, corresponding to a resolution of 1/10 000th of the full scale. In addition, the method also features high anti-interference ability, and can perform demodulation satisfactorily as long as the signal-to-noise ratio is greater than 6.87 dB. According to the principle of the new demodulation scheme, in applications that do not emphasize real-time performance, the resolution of the system can be further improved by increasing the subdivision.