In order to improve the automation level and the precision of building extraction, an automatic building extraction method based on shifted shadow analysis is proposed. It is guided by the principal line of segmentation-classification-optimization. The object oriented multi-resolution segmentation method is adopted to perform the initial image segmentation. The segmentation results are classified by the support vector machine (SVM) classifier into four categories, i.e., shadow, vegetation, building and bare land. The initial results are extracted. The shadow rate on the intersection boundary is designed to accurately validate the existence of buildings and remove the disruptions of non-buildings without shadows, and the final results are obtained. The large amount of experimental results validate that the proposed method is very effective, and the automation level is significantly improved. The completeness is more than 85%. The correctness and the F1-score can both reach more than 90%.The proposed method only needs data from images in the visible band and has a wide application range.