A method of extracting manmade object from natural background is presented. Operators of difference orientations based on image gray separated a image into some sub-images of different orientations. Post processing of every sub-image made these sub-images construct different difference orientations collection of line segment. Structure grammar based on local area of collection of line segmentation and geometric element and number of classes of geometric element could extract manmade object from natural background. This algorithm could be developed as real-time hardware trackors applied to automatic target capture. Finally, experimental results show that this method can extract manmade object in infrared image under conditions of complicated background and low contrast, and the correct detection ratio is more than 95%.