Murals are treasures in the long history of Chinese culture. It has high research value in history, science and art. Pigment is the material carrier of the main form of mural expression. Simultaneously, it is also an important part of murals. After a long period of disease and corrosion, the surface of the mural may be damaged to varying degrees, making it difficult for researchers to distinguish the type of pigments in the murals. The accurate identification of pigments is the premise of conservation and restoration of cultural relics. The traditional method needs to take samples from the murals, which will cause irreversible damage to the murals. In this paper, multispectral imaging technology and deep learning related classification algorithm are used to analyze and identify the pigments in mural multi-spectral images to assist researchers in mural identification and cultural relic restoration. Rich spatial and spectral information is included in mural multispectral images. In traditional algorithms, spatial or spectral information is used as a feature of mural multispectral image classification. This method leads to low classification accuracy of mural multi-spectral image. In order to improve the classification accuracy of mural multispectral images, the deep learning algorithm is used in this paper, which can make full use of the spatial and spectral information of multi-spectral images. In the actual shooting, due to the limitations of site conditions and protection requirements, mural spectral imaging data need to be collected quickly. The efficiency of data acquisition can be improved by using sparse channel imaging methods. However, this method can make the spectral reflectance curve of pigments appear nonlinear, which can affect the classification accuracy of mural multi-spectral image. In order to solve this problem, a pigment classification method for mural sparse multi-spectral images based on spatial spectral combination features is proposed in this paper. The nonlinear spectral features are extracted by using the hyperbolic tangent activation function in the Long Short-term Memory (LSTM). Firstly, the spectral reconstruction of the mural multi-spectral image is carried out. Then the one-dimensional spectral vector is input into LSTM, which can actively learn under unsupervised conditions to reduce the influence of spectral curve nonlinearity on the classification accuracy. In order to solve the problems of high spatial resolution and strong correlation between adjacent pixels in multispectral imaging, the linear correction function in Convolution Neural Network (CNN) is used to map the feature map to nonlinear space. The activation function is added after the convolution operation of each layer to improve the nonlinear expression ability of the mural multi-spectral image. The spatial spectral unity of mural multi-spectral images can not be fully utilized, if only spatial or spectral features are used. For this reason, the multi-scale fusion strategy combining spectral and spatial features is used to eliminate the influence of spectral nonlinearity and spatial correlation on the classification results. Firstly, a Spatial Spectral Joint Feature Network Model (SSJF) is established to train pigment samples. Then, the loss function is designed by cross entropy, and the gradient is updated by back propagation algorithm. Finally, the softmax classifier is used to output the probability of each pigment. The experimental results show that the pigments in the paint board and self-made murals can be correctly classified through SSJF. The OA and Kappa coefficients reached 97% and 0.97, respectively, which effectively improved the pigment classification accuracy of the sparse multi-spectral image of the mural.