The phase grating position measurement system is the core of achieving ultraprecision manufacturing. The contrast of the self-referencing interference signal significantly influences the accuracy of measuring the position of phase gratings, which is directly determined by the polarization properties of beams. In this study, the Jones matrix and vector-form reflection law are used to trace the polarization of the transmitted beam in the self-referencing interferometer prism, and the Jones matrix model of the phase grating position measurement system is constructed; then, based on the physical light field numerical analysis software VirtualLab, the accuracy of the model is verified. Based on this model, the influence of the extinction ratio of the polarization beam splitter, the polarization state of the incident light, and the optical path difference of the prism on the contrast of the self-referencing interference signal is analyzed. The results show that to ensure that the contrast of the interference signal is better than 0.98 under the illumination beam of wavelength λ, the extinction ratio of the polarization beam splitter needs to be greater than 122; in addition, the incident light ellipticity ψ and ellipticity tan ε are (0.693 rad,0.878 rad) and (-0.093,0.093), respectively, and the prism optical path difference needs to be controlled within (-0.03λ,0.03λ). After comprehensive consideration, the value range of the incident light ellipticity is required to be 0.531δ±0.087 (δ is the phase difference from the reference interferometer prism). The analysis results can provide a theoretical basis for adjusting and compensating for the beam polarization characteristics in the phase grating position measurement system and are crucial for improving the contrast of the self-referencing interference signals and accuracy of the phase grating position measurement system.