In image segmentation based on DeepLabV3+, the different importance of features in different levels of feature images are ignored in the feature extraction stage, and a large amount of details are lost, resulting in poor segmentation effect. To solve this problem, an image semantic segmentation algorithm based on the combination of DeepLabV3+ and attention mechanism is proposed. Two low-level features are extracted in the backbone network Xception model as input features of the decoder to improve the accuracy of feature extraction. The channel attention module is used to effectively integrate high-level features and obtain rich context information. The spatial attention module is used to extract low-level features and filter background information to reduce the loss of details. The depthwise separable convolution is substituted for void convolution to effectively reduce the amount of parameters and improve the calculation speed. At the same time, the focus loss is used as the loss function to improve the final segmentation effect by reducing the internal weighting. Experimental results show that the mean intersection over union (mIoU) value of the proposed algorithm on PASCAL VOC 2012 dataset reaches 84.44%. Compared with the traditional algorithm and the improved algorithm based on DeepLabV3+, the proposed algorithm effectively improves the accuracy of feature extraction, reduces the loss of feature details, and improves the final segmentation effect.