In resonator micro-optic gyro (RMOG), square-wave frequency modulation implemented by applying serrodyne waveforms on phase modulators is one of the most frequently used techniques. However, in order to realize ideal square-wave frequency modulation, perfect linear phase modulation waveform is required, which is hard to achieve in analog systems. The influence of the nonlinearity of modulation waveform on the output of the resonator is analyzed. Moreover, the resonant curve shift and distortion induced by the second and third order modulation nonlinearity are theoretically calculated. Furthermore, the relationship between demodulated output and input rotation rates is presented. A RMOG experimental system is set up, and the generated triangle wave is tested and fitted to a nonlinear cubic curve. The coefficients of the second and third order nonlinearity are in the opposite directions, which counteracts part of the error. The testing result of the scale factor shows the feasibility of the modulation waveform generating method.