Building extraction from high-resolution remote sensing images is easily affected by the surrounding shadows and vegetation, leading to incorrect detection of buildings in these areas and the building contour would deviate considerably from its actual shape. Therefore, this paper proposes a regularization method for building contour based on bidirectional-driven adaptive segmentation and reconstruction to solve this problem. First, to reduce the jagged representation on the contour graphics display, a flat rotation transformation algorithm based on the minimum bounding rectangle (MBR) was adopted to rotate the building contour to the horizontal or vertical state. Then, MBR and Shi-Tomasi were combined to develop the bidirectional-driven adaptive segmentation algorithm, thereby allowing the building contours to be divided into small parts to form accurate local segments. Finally, a contour reconstruction algorithm based on local optimal weight fitting was proposed. The regularization of the building contour was achieved through attribute assignment, constraint recombination, optimal weight fitting, and coordination reconstruction of local segments. Compared with the initial extraction results, the visual experimental results of the regularized contour obtained using the proposed method are better. Furthermore, compared with the other four similar contour regularization methods based on grid filling, corners correction, suitable circumscribed rectangle fitting, and main direction, the proposed method provides an efficient contour regularization, a higher accuracy, and broader applicability. The proposed method can obtain more regular contours for typical complex buildings with different angle changes. This method can be used as a reference for post-processing regularization of building extraction.