The steady-state solutions and eigenvalues of the nondegenerate optical parametric oscillator are obtained, and the instabilities of the system in both cases of resonance and non-zero detuning are investigated. The results show that the system is always in a steady state when there is resonating and it has instability when there is non-zero detuning. The evolution of eigenvalues versus input field amplitude portrayed by numerical simulation with no detuning, as well as the evolution of the three output modes versus time with non-zero detuning, are proposed.