Image semantic segmentation requires fine detail information and rich semantic information, but in the stage of feature extraction, continuous down-sampling operation will lead to the loss of spatial details of objects in the image. To solve this problem, a semantic segmentation algorithm based on double-branch structure is proposed, which can obtain rich semantic information effectively and reduce the loss of object details in feature extraction stage. One branch of the algorithm uses shallow network to retain high-resolution detail information which is helpful for object edge segmentation, and the other branch uses deep network for downsampling to obtain semantic information which is helpful for object category recognition, and then the effective fusion of the two kinds of information can generate accurate pixel prediction. Experimental results on Cityscapes and CamVid datasets show that the proposed algorithm achieves better segmentation performance under fewer parameters than existing semantic segmentation algorithms.