Terahertz time-domain spectroscopy (THz-TDS) is an important method for rapid and nondestructive material identification due to its spectral fingerprint properties, which has broad application exploitation in the nondestructive inspection of drugs and explosives. Spectral identification is one of the most important aspects of the applied research of THz-TDS. Most existing spectral identification methods are machine-learning based classification of manually selected features or thresholding classification of absorption spectral peak. Those methods are not adapt well to low signal-to-noise ratio, because some materials have few or no spectral absorption peaks features in the terahertz waveband and spectra are affected easily by concentrations of samples, air humidity and noises. Meanwhile computational cost increases with data quantity and category. In recent years, with the rise of deep learning technology, the methods represented by CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) have been widely applied to fields such as computer vision and natural language processing where they have been shown to produce better results than traditional machine learning methods. Due to the strong nonlinear classification capability of deep learning technology, two networks respectively were designed based on RNN and CNN for spectral identification in this paper: one-dimensional spectral line classifier based on RNN and two-dimensional spectral image classifier based on CNN. To simulate the practical application scenario, over 20 000 terahertz time-domain spectra of 12 materials were measured in a non-vacuum environment as training-set and test-set. After analyzing the effects of concentrations of samples and air humidity on spectra, S-G(Savitzky-Golay) filter was introduced to reduce noises of spectra. Experimental results show that S-G filter could improve the identification accuracy, because processed spectra have more obvious feature compared with the unprocessed spectra; the proposed methods based on RNN and CNN are more accurate and faster on the test-set, compared with traditional machine learning algorithm k-NN (k-Nearest Neighbor); CNN demonstrated better robustness to noises than RNN on spectral identification task. Therefore, deep learning technology could be utilized for quick and effective identification terahertz time-domain spectra, which provide a theoretical and experimental basis for new nondestructive safety inspection techniques.