In the process of collecting thermal images of infrared nondestructive testing (NDT) with light source as the excitation, due to the influence of uneven heating, environmental radiation and other factors, the collected thermal image sequence has problems such as high background noise, low contrast and poor display effect of defects, which are easy to cause the omission of defects. In order to improve the defect detection rate, infrared thermal image sequence processing technology based on Trust Region Reflective (TRR) algorithm was proposed. Firstly, the background noise surface with uneven heating was fitted by TRR algorithm, and the background surface obtained by fitting was subtracted from the original thermal images to remove the background noise caused by uneven heating. Then, Principal Component Analysis (PCA) algorithm was used to extract the defect feature information of the thermal image sequence after removing the background, so as to further improve the signal-to-noise ratio of the infrared thermal wave images. Finally, the defect region was segmented by region-growing algorithm. The experimental results show that a combination of these algorithms can effectively improve the signal-to-noise ratio of the infrared thermal image, thus improve the defect detection rate.