For the question of the segmentation for Forward-looking Infrared (FLIR) building images, a template- based method based on the traditional method is presented. Firstly, the 2D shape templets of building targets are made using the Digital Surface Model (DSM) data. Then, the image is processed by Gaussian filter, and is enhanced using the method of gray-scale mathematic morphology based on template dimension. Furthermore, after the introduction of idea about template shape transcendental information, silhouette information is described using the method of level set. Lastly, the additional energy of target silhouette transcendental information based on the C-V model makes the improvement of segmentation driven by image data and shape transcendental information simultaneity. Finally, target segmentation is realized. Experiment results show the proposed algorithm can precisely segment the buildings from FLIR images with a complicated background.