The present research proposes an efficient scale-adaptive and fully convolutional network based on an encoder-decoder network, which represents a crucial innovation aimed at improving buildings extraction with various scales from remote sensing imagery with high spatial resolution. First, a multiple-input multiple-output structure is proposed to obtain multiscale features fusion and cross-scale aggregation. Then, a residual pyramid pooling module is deployed to learn deep adaptive multiscale features. Finally, the initial aggregated features are further processed using a residual dense-connected aggregated-feature refinement module. Pixel dependencies of different feature maps are investigated to improve the classification accuracy. Experimental results on the WHU aviation and the Massachusetts datasets show that compared with other methods, the method has a better extraction effect on buildings, and the training time and memory usage are moderate, which has high practical value.