Detecting lines in a digital image is a crucial step in many applications, such as visual navigation, feature recognition and remote sensing image processing, In this paper, an algorithm which is based on edge orientation image is proposed for extracting straight lines from comples building image. Firstly, edges are detected based on Canny detector. Secondly, an edge direction coding strategy is proposed by detecting the local line in 9×9 window. Thirdly, edges are split into a series of straight lines, curves and points, and a new secondary image — the edge orientation image is generated. Finally, the straight lines are extracted from the edge orientation image by analyzing the direction distribution, and using the straightness errors evaluation strategy and the sparse line fitting algorithm. Experiment results show that the proposed algorithm is better than hough transform and phase grouping algorithm. It can detect lines correctly in complex building images with high accuracy rate, low false and low missing rate and good robustness.