In order to improve the measurement precision of identification models, multisource information fusion technique was employed to identify four types of wheat grain (normal, worm-eaten, moldy, and sprouting wheat grains). The terahertz (THz) spectra of wheat grains with various degrees of deterioration were investigated; classification fusion models were constructed by using THz absorption spectra combined with refractive index spectra. The adaboost and SVM were used in the feature level fusion models. The results showed that the different wheat samples were identified with an accuracy of nearly 95%. Furthermore, fusion models results of wheat detection were compared with results from other methods, the comparisons showed that the recognition ratio of fusion models had a great improvement, and the SVM model outperformed the others.