Square-wave modulation with various distortion will influence the outputs of the digital closed-loop Fiber Optic Gyroscope(FOG) when measuring angular velocity. Based on Fourier series, square-wave modulation models and signal demodulation models for closed-loop FOG including distortion noise were built in this paper. Simulation studies were analyzed for the output errors caused by different modulation distortion signals. A new demodulation method of bipolar return to zero pulse of square-wave for the elimination of errors in output signals of FOG was proposed. The simulation results show that the distortions, including phase distortion, pulse width distortion, harmonic distortion and noise comb pulse, have a great influence on outputs of FOG when using conventional demodulation methods of square-wave. Measuring the relative errors of angular velocity is up to 1%. But the errors are effectively eliminated when using bipolar return to zero pulse of square-wave demodulation, and are one order of magnitude lower than the errors in conventional demodulation methods. The simulation proves that the demodulation method of bipolar return to zero pulse of square-wave is very meaningful for improving measurement accuracy and stability of the digital closed-loop FOG.