The terahertz spectrum material identification method mainly relies on finding the different characteristics of the different spectra of the substance in the terahertz band to identify a specific substance. The methods of absorption peak extraction are commonly used spectral feature extraction algorithm. However, when the spectrum has no obvious characteristic absorption peaks or peak positions, and peaks are similar or difficult to distinguish, it is difficult to use the absorption peak characteristics to distinguish substances. Although machine learning and statistical learning techniques to identify terahertz spectra reduces the interference of absorption peaks, it often requires an artificial definition of features to cause classification errors. The deep learning method can automatically extract features, but it often requires complex preprocessing operations before recognition, and it is easy to lose some features in the feature extraction process, leading to classification errors. A method of terahertz spectrum identification based on wavelet coefficient graph and convolutional neural network is proposed. When using the terahertz spectrum signal for wavelet transformation, each row of the wavelet coefficient matrix has a corresponding relationship with the original spectrum signal. The absorption coefficient of the terahertz spectrum is expanded in the frequency domain through wavelet transformation to obtain different two-dimensional frequency-scale distribution diagrams, which are also known as wavelet coefficient maps. Then a convolutional neural network (CNN) is constructed to classify the wavelet coefficient graph, and the classification result of the terahertz spectrum material can be obtained. To verify the effectiveness of the proposed algorithm, the three sets of wavelet coefficient maps and the original spectral data were input into three different classifiers of CNN, Support Vector Machin (SVM), Multilayer Perceptron (MLP) respectively for comparison. From the experimental results, we can find the recognition of the algorithm in the three sets of data. The rates reach 100%, indicating that compared with traditional methods, the method in this paper can still accurately classify spectra without obvious characteristic absorption peaks, which proves the effectiveness of using convolutional neural networks to identify wavelet coefficient maps. To show the advantages of the proposed algorithm in this paper, we compared it with the wavelet ridge peak-finding recognition algorithm. The experimental results show that the proposed algorithm is hardly affected by peak frequency, peak position, and peak value. Whether to identify the starch without an absorption peak or to identify high similarity sucrose and glucose, a high recognition rate is achieved by the proposed algorithm, and the classification accuracy rate is up to 97.62%, which proves the superiority of the proposed algorithm. The proposed algorithm provides a new idea for identifying terahertz spectrum data and can also be extended to the identification of other spectrum substances.