The segmentation accuracy of nonparametric semantic segmentation algorithm, is easily affected by the image retrieval accuracy and semantic category unbalanced dataset. To solve these problems, an image semantic segmentation algorithm based on convolutional neural network (CNN) feature and improved superpixel matching is proposed. Image features are obtained through CNN learning, and images are retrieved after reducing dimensions of the features, and then the accuracy of image retrieval set can be improved. Superpixels of the images in the retrieval set are weighted by using Gaussian kernel density estimation, which increases the superpixel matching accuracy of rare classes. Therefore, semantic segmentation accuracy of query image can be improved. The experimental results on SIFTflow and KITTI datasets show that both per-pixel and per-class rates of the proposed algorithm are the best among different algorithms.