In order to achieve the high-precision measurement of glass internal stress, a new method of magneto-optical modulation was proposed, and the system of internal stress measurement based on the magneto-optical modulation was established. First, measurement model of the system was derived according to the Mueller matrix description method of polarized light, by using separation of the direct current, fundamental frequency and the second harmonic component of detected signals, and "normalized" approach, the impact of the light source intensity fluctuation on the measurement result was eliminated. The direction of glass internal stress and the size of stress birefringence values were received by processing the respective signal components. By measuring the glass at different positions, the validity of the method is verified, and the measurement accuracy of internal stress direction is 5″, the measurement accuracy of stress birefringence value is below 0.5 nm/cm. This system has high stability and high accuracy.