To accurately segment ground objects from a high-resolution remote sensing image, we propose a remote sensing image segmentation network based on multi-level feature optimization fusion that focuses on the fusion of feature maps at different levels in the feature extraction skeleton network, performs reasonable and effective extractions, and analyzes output feature map information by fusing different types of information in the network feature map. Simultaneously, layer-by-layer multi-scale coding and decoding modules are used to refine the shallow feature map that merges with the high-level feature map, and the different types of information are optimized to the high-level feature map. The hollow convolution pyramid is then used to extract the information of different receptive fields on the high-level feature map, and the output feature map of semantic segmentation is optimized. When conducting experiments on the ISPRS Vaihingen dataset, the overall accuracy of the proposed network reaches 90.34%, which effectively improves the accuracy of remote sensing image target detection when compared with the classical semantic segmentation network. Moreover, to prove the generalization of the proposed algorithm, a generalization experiment on the ISPRS Potsdam dataset is conducted; the overall accuracy of this algorithm reaches 91.47%, proving its effectiveness.