The traditional spectral matching method is based on the spectral reflectivity. However, the process of obtaining the reflectivity of each point is complicated, and the calculation has some errors, which will affect the recognition accuracy. In order to solve this issue, the problem of mural pigment recognition has been transformed into multi-spectral image classification, and a convolutional neural network algorithm with strong advantages is used in image classification to process multi-spectral images. Meanwhile, a new convolution neural network model is designed, and a data preprocessing method of spectral feature reorganization is proposed. By adding two dropouts, the problem of over-fitting in the training process is prevented, and the classification of ancient mural pigments is realized. The experimental results show that compared with the statistical manifold support vector machine classification method and the convolutional neural network classification method without dropout, the proposed method has obvious advantages in classification effect and classification accuracy.