Thermal infrared imaging has an essential application in face recognition, but it has certain limitations, such as low resolution, unclear details, and fuzzy boundaries. Herein, we describe the enhancement effect of polarization detection technology on the texture details of thermal infrared face imaging by analyzing the characteristics of the long-wave infrared polarization images of human faces. Based on the correction of the difference of Gaussian (DoG) edge feature image’s color gamut channel mapping weights, a RGB space fusion framework for the polarized thermal images of human faces is proposed. We use the histogram of oriented gradients (HOG) to obtain infrared polarization facial features and propose a face recognition method based on support vector machine (SVM). Experimental results show that, first, polarization detection technology can enhance the texture and details of the infrared thermal image of the human face, and that RGB color gamut fusion can improve the structural similarity of the long-wave infrared thermal image of the human face. Second, the overall quality index of polarized infrared thermal images is better than ordinary infrared thermal images. Finally, under the framework of this article, the accuracy for face recognition can reach 75.6% using the polarized infrared thermal images of the face.