In order to improve the accuracy of multi-spectral image semantic segmentation, a neural network model based on local binary pattern (LBP) feature enhancement is proposed. The model obtains two feature maps from a single infrared image by two LBP feature extraction operators with the size of 3×3 and 5×5, respectively. The RGB image, the infrared image, and the LBP feature maps are imported into a neural network model with a 34-layer residual network for semantic segmentation. The experimental results show that the proposed neural network model can achieve an average accuracy of 60.7% and an average intersection over union of 51.9% on the RGB-Thermal dataset. The results are superior to other comparative methods. At the same time, in the visualization results, the results of proposed model are also more clear and accurate.