An infrared polarized thermal image segmentation algorithm based on image gray and information entropy fusion is proposed. First, the local average gray and variance weighted information entropy of the image are used to find out the potential region of multi polarization azimuth thermal image and register it; second, the improved fuzzy C-means (FCM) algorithm is used to segment the thermal image separately, and the result of set operation is used as the label of support vector machine (SVM); then, the data of target region and background region are trained to obtain SVM model and redivide the fuzzy region; finally, the segmented thermal image is achieved by morphological processing synthetically. Experimental results show that compared with the maximum entropy algorithm, OTSU algorithm, and FCM algorithm, the proposed algorithm can get higher segmentation accuracy and effectively improve the phenomenon of wrong segmentation.