In active infrared thermography technology, the extraction of defect information from infrared images is crucial. Traditional image processing methods can eliminate noise and improve image contrast, but several challenges remain, such as selecting the infrared image manually, subjectivity in the process of infrared image enhancement and segmentation, and information loss in the process of a single infrared image. To overcome these challenges, this study proposes a method for extracting defect information from infrared images based on time sequence information. First, concrete blocks with delamination are fabricated by indoor experiments. Then, active infrared thermal image detection technology is used to collect the infrared image data and temporal information is extracted for each pixel. Finally, the K-means method is used for defect feature extraction based on temporal information. The results show that the defect extraction method based on temporal information can extract hidden defect information. Furthermore, its hierarchical defect information extraction effect is better than that of the K-means method based on the spatial domain.