To overcome the shortcomings of infrared thermography, a novel method of metal fatigue damage assessment based on the characteristics of surface infrared polarized thermal imaging is proposed in this paper. First, the theoretical basis of polarized thermal imaging for metal fatigue damage assessment is provided. Then, the change rule of surface micromorphology in the process of fatigue damage is experimentally explained. Next, the target image is segmented and analyzed using the Tsalis entropy algorithm. Finally, the feature selection is performed using principal component analysis and a nonlinear prediction model based on back propagation (BP) neural network is constructed. Experimental results show that during the process of metal fatigue damage, the results of model training, verification, and test have a good correlation with the actual experimental data, and the average prediction error of tension and tension fatigue damage for the specific test target Q235 plate is less than 20%.