As the commercial CMOS (Complementary Metal Oxide Semiconductor) transistor model will lose the accuracy in a high frequency range, a nonlinear RCL transmission line model based on classic kinetic theory was developed with ADS software. The accuracy of proposed model and the working principle of the CMOS in THz range were discussed based on measured data. A simulation system for the nonlinear RCL transmission line model was constructed, and simulation results were compared with that of commercial model and the difference between the proposed model and the commercial CMOS model in THz range was analyzed. Then, the frequency responses of current CMOS transistors were tested, tested data were compared with those of the simulation data from the two kinds of models. The results demonstrate that the proposed model has been improved the prediction accuracy. Finally, the effects of channel size of transistor on the scattering effect of carriers were analyzed and conditions of transistor to turn on ballistic mode were given with 3σ rules. The results show that the difference between the two models mainly focuses on the inductance part, which could represent the momentum conservation of carriers in transistor channel and if the scattering effect could be neglected or not. Compared with the commercial model, the prediction accuracy for the optimal resonant frequency of a detector has improved by 0.3%, and that for the optimal working bandwidth of the detector has increased about 10%. This study provides a good foundation for the accurate establishment and simulation analysis of CMOS transistor models.