With the rapid increase of electrical power demand, the voltage level of power grid also improved. The traditional current transformer is difficult to meet the needs of power grid because of its easy saturation, poor insulation, narrow frequency band, etc. The optical fiber current sensor can solve these problems. Therefore, it has wide application prospects in power monitoring and smart grid construction. However, the nonlinear response of magneto-optical crystal is the main factor that limits the application of the magneto-optic fiber current sensor. Nowadays, research on the nonlinear error of the Fiber Optical Current Sensors (FOCS) mainly focuses on the nonlinear dependence of the Verdet coefficient of the Magneto-optical (MO) materials on the external temperature. The change of Verdet coefficient will lead to the crosstalk of magnetic field and temperature, which is one of the main factors limiting the practical application of the FOCS. Nevertheless, when the temperature is fixed, the output signals of the MO materials is also not changed linearly with change of the external magnetic field. To improve the accuracy of the sensors, we studied the source of the nonlinear response of the MO crystal theoretically and experimentally. In this paper, we demarcated the output curve of garnet crystal from no magnetic field to saturated magnetic field. Through simulation and experiment, we proved that the nonlinear response of MO fiber optical current sensor is caused by the diffraction characteristics of magnetic domain of the MO material. The model of magnetic domain effect proposed by us conforms to the output response of the MO sensors. Through the dual-channel demodulation method, we can calculate the intensity of the output light through the 45° polarization beam splitter according to Malus law. And we obtain the Faraday rotation angle according to the demodulation algorithm of difference division sum. The theoretical results show that the influence of nonlinear error caused by magnetic domain effect can be ignored with the method of the two-channel demodulation. And in this way, the influencet of light source fluctuation can be eliminated. Simulation and experimental results show that the nonlinear error at a fixed temperature is mainly due to the voltage conversion coefficient mismatch of the two photodetectors and the assembly error of angle between polarizer and polarization beam splitter. Although the nonlinearity seems slight in macro performance, it is the key to affect the performance of the sensors in the case of high accuracy requirements. When the temperature changes, the nonlinear error of the MO materials is mainly due to the temperature dependence of Verdet coefficient on temperature. In order to solve the above problems and improve the accuracy of the MO fiber optical current sensor, we propose a nonlinear compensation model based on quadratic fitting. By comparing the four error models between the compensation model and the traditional demodulation model, the experimental results show that this method has good fitting accuracy and is suitable for nonlinear compensation of FOCS based on the MO material. In the actual experiment, we adjust the two magnifications to make the two conversion signals equal without applying the magnetic field. It means that the initial assembly angle error is compensated by adjusting the magnification. In this case, the result of the sensor output is the best. After that, the nonlinear law of the sensor is calibrated by the experimental data, and the external magnetic field is obtained by solving the quadratic function, which can significantly improve the accuracy of the sensor. This method has a simple structure and high fitting precision, and can meet the real-time acquisition of the MO fiber optical current sensor.