Terahertz gyrotron can achieve high output power and has a certain frequency tuning range, thus it is an ideal high power terahertz radiation source for NMR spectroscopy system. A 263 GHz, TE_5,2 fundamental harmonic frequency continuously tunable gyrotron is designed, the corresponding frequency tuning range can reach 1.4 GHz through adjusting the magnetic field. The unstable oscillation state of the designed gyrotron is studied by using the time domain multi-mode multi frequency self-consistent nonlinear theory. The results show that in the magnetic field range of low order axial mode, when the operating current is greater than the starting current, the continuously tuned gyrotron enters the stable state, where the high order axial mode is suppressed, and the output power of TE_5,2 remains unchanged with time. When the current increases, the competition between axial modes causes the gyrotron to enter the unstable oscillation state from the stable state, the output power of TE_5,2 oscillates with time and the interaction efficiency decreases greatly. With the further increase of current, the gyrotron returns to another stable state different from that of low current. It is also found that the initial current increases with the increase of magnetic field. The research of this paper has a certain guiding significance for the development of continuously tuned THz gyrotron for DNP-NMR applications.