The transmission matrix of the magneto-optical (MO) nonlinear fiber Sagnac structure is deduced, which can be used to analyze the influences of fiber birefringence, MO effect, nonlinear phase shift, the polarization controller state and the total loss of the fiber loop on the transmittance of Sagnac. The MO modulation of Sagnac structure and functions of nonlinear optically-controlled-optical switch are experimentally investigated. The MO modulation degree can reach 17.05 dB under the applied magnetic field of 0.0177 T, which is in good agreement with the theoretical analysis. When the peak power of the clock pump control signal is 28.2 dBm, the extinction ratio of optically-controlled-optical switch can reach 25.8 dB. The extinction ratio of the switch can increase 0.7 dB by further applying the magnetic field. To some extent, the magnetic field response of the optically-controlled-optical Sagnac structure may be equally analyzed with the help of polarization control state.