In view of the problems that the traditional neural network model tends to ignore difficult samples due to the unbalanced classification of remote sensing image semantic segmentation data, and the reasoning results are hollow and the segmentation accuracy decreases, a drill-shaped neural network semantic segmentation method is proposed. First, a new bridge module is defined to fuse the shallow and deep feature information, thus more building details can be captured by the network; second, in the deep learning segmentation model training, the multi loss function is used to improve the extraction of difficult sample information; finally, to balance the differences of category training, the feature information is extracted from remote sensing images at multiple levels, and the segmentation accuracy is improved. The experimental results show that the average intersection to union ratio of the proposed method reaches 0.849, the building missing rate and wrong recognition rate are less, and the segmentation accuracy is improved compared with the existing methods.