As a pixel-level classification technique, image semantic segmentation has been employed in the field of synthetic aperture radar (SAR) image interpretations. U-Net is an end-to-end image semantic segmentation network with a typical encoder-decoder architecture. Among them, the coding part mainly comprises a convolutional layer and a pooling layer, which can effectively extract the features of a target image; however, extracting information such as the target position and direction is difficult. Capsule network is a type of neural network that can obtain the target pose (position, size, and direction) and other information. Therefore, this study proposes an SAR image semantic segmentation method based on the U-Net and capsule network. Moreover, considering the small data set of SAR images, the U-Net encoder is designed to be identical to the visual geometry group (VGG16) to allow the trained VGG16 model to be directly transferred to the encoder. The effectiveness of the method is verified by conducting a segmentation experiment of building targets on two polarimetric SAR image data sets. Results show that the method can achieve improved precision, recall, F1-score, and intersection over union as well as reduce the training time of the network model when compared with the U-Net.