Terahertz spectral imaging not only includes the intensity information in two-dimensional image space but can also obtain spectral information in the terahertz band, constituting a three-dimensional data matrix. Due to the limitation and influence of the internal hardware of the Terahertz imaging system, the signals in the higher frequency band of the terahertz frequency domain have weak energy and low signal-to-noise ratio, resulting in the problems of low resolution and low contrast of the terahertz images. Therefore, this paper improves the spatial resolution and edge detail visibility of terahertz spectral imaging by using a three-dimensional data matrix and a suitable algorithm. In this paper, a three-dimensional portable Terahertz time-domain spectroscopy imaging system is built to realize the two-dimensional scanning of standard high-resolution plates. The signals collected by the system were compared in the time domain and frequency domain, respectively. The spatial resolution and depth of field of the imaging system were calibrated by combining the Rayleigh criterion and resolution scale, and the spatial resolution algorithm for improving THZ spectral imaging was studied. Then, aiming at the characteristics of low SNR, low contrast and complex noise causes in the high-frequency region of the Terahertz frequency domain, combined with the image denoising theory of deep residual learning, a terahertz image depth denoising network is proposed, which introduces the real “terahertz residual noise” in the imaging system in the training set. Finally, the reconstructed images are compared with the original images and the traditional terahertz denoising algorithm results. The denoising effects of different algorithms on the high-frequency images in the terahertz frequency domain are evaluated from subjective and objective aspects. Experimental results show that the limit spatial resolution of the proposed algorithm is about 157 μm, the saddle-peak ratio of the Rayleigh criterion at the limit spatial resolution of the denoised image is 0.623, and the overall image contrast is 46.635. The spatial resolution is about double that of traditional imaging methods, and the contrast is about 26% higher. The results of this study provide a new standard for high spatial resolution and high visibility THZ spectral imaging and provide a new solution to the problem of image noise in the higher frequency region of the THZ frequency domain.