The remote sensing image semantic segmentation in rural areas is the basis for urban and rural planning, vegetation and agricultural land detection. Segmentation of a high-resolution remote sensing image of rural areas is difficult because of the complex image information. Herein, we designed a complete symmetric network structure that includes a pooled index and a convolution used to fuse semantic information and image features. The Bottleneck layer is constructed using 1×1 convolution and employed to extract the details and reduce the parameter quantity, deepen the filter depth to build an end-to-end semantic segmentation network, and improve the activation function to further enhance network performance. The experimental results show that the accuracies of the proposed method and the classical semantic segmentation networks U-Net and SegNet are 98.4%, 80.3%, and 98.1%, respectively on the CCF dataset. Thus, the proposed method achieves better performance than the other two methods.