Aimed at the phase ambiguity phenomenon existing in the φ-OTDR based vibration sensing system, the expression to describe the relationship between the Rayleigh backscattering power and the phase difference of optical interference fields, and the expression to describe the relationship between the phase difference of optical interference fields and the strain of optical fiber induced by vibration were derived. The cause and manifestation of the phase ambiguity phenomenon and the influence of the phase ambiguity phenomenon on system performance as well as the relationship between the maximum detectable strain and the spatial resolution of the sensing system were studied theoretically, and demonstrated by Matlab simulation. The results show that the phase ambiguity phenomenon is the main factor that limits the strain measurement range of the φ-OTDR based vibration sensing system, and it may cause mistakes in the measurement of vibration frequency and amplitude, and a failure in reporting large strain information. And the higher the spatial resolution, the larger the detectable strain range, which indicates that the φ-OTDR based vibration sensing system is very suitable for applications in the measurement with high spatial resolution. This study provides a theoretical support for the parameter design and optimization of the sensing system.