According to the elliptical birefringence in magneto-optic (MO) fiber, the eigen and non-eigen nonlinear coupled-mode equations for guided optical waves are put forward, which are used for analyzing nonlinear effects and elliptical birefringence in MO fiber. The relation between the equivalent nonlinear coefficient and the ellipticity of eigen elliptical polarized light is deduced for self-phase modulation (SPM) case. The maximum nonlinear phase shift of SPM will decline with the increase of magnetic field, and then reveals the mechanism of magnetically controlled nonlinear optical devices. Our calculation results show that the eigen nonlinear couple-mode equations are especially applicable for split-step Fourier method, and the polarization dependence of nonlinearity reduces under large applied magnetic field for input linearly polarized light. The findings can apply to the control of nonlinear effects in optical communication field.