An infrared pedestrian detection algorithm is proposed to solve the problem of small differences between pedestrians and backgrounds in gray scale images and the occurrence of occlusion in infrared images at night. First, a significant graph with the full coverage of the target is generated by the pedestrian semantic fusion method, and the region of interest is obtained by combining it with the original graph. Then, a two-branch classifier based on the improved histogram of the gradient feature is constructed. The fuzzy score of the classifier is used to determine the occurrence of occlusion and call the head template for the final detection. Experiments based on the LSI far infrared pedestrian dataset and independent datasets of pedestrians captured at night in winter and summer prove that the proposed method is robust and quick in detecting pedestrians under different environments. It can significantly reduce the rate of missed detection and realize a detection rate of 94.20%.