A small building target detection algorithm is proposed based on FCOS neural network. Aiming at the problem of insufficient target features of small buildings extracted in the feature extraction stage of the FCOS algorithm, multi-scale convolution and deformable convolution were used to improve the ability of the network to extract features of small buildings. And through the improved SGE attention mechanism, the weight of interference noise in the feature map becomes lower. The improved network can extract more target feature information of small buildings and is more robust to noise. Experimental results obtained from the building dataset made by ourselves show that, under the same test environment, the network’s overall detection accuracy (mAP) of buildings under normal, dense and occluded conditions is improved by 1.7%, and that of small buildings is improved by 3.6%, reducing the missing and error detection of small building targets.