In infrared nondestructive testing, the proportion of defects is very different from that of background, and the low contrast region of infrared image has not been completely eliminated after image sequence enhancement, resulting in impaired accuracy of defect segmentation. In order to solve this problem, a defect segmentation method based on robust Otsu algorithm was proposed, which combined the relative threshold idea of local threshold segmentation method. Firstly, the mean value and the total gradient of the neighborhood were used to represent the category and spatial state of the pixels. Secondly, a point-block fusion statistical adjusted model on this basis was established for dynamically adjusting the gray scale values of the infrared image defects and non-defect regions. Finally, the improved two-dimensional histogram and its region division method based on gray value and neighborhood gray deviation was set for calculation of fitness function in genetic algorithm through which the optimal threshold could be determined from the mutative neighborhood size, then segmentation of defects could be achieved. The results show that this method improves the robustness of Otsu and the accuracy of defect segmentation.