It is difficult to search the spectrum line in handwritten music spectrum, so in order to improve the robustness of the handwritten music spectral line deletion algorithm, a method based on multidimensional local binary pattern recognition and XGBoost model is proposed. The local binary pattern operator is designed and improved based on the characteristics of music score image, and from which the multidimension local binary pattern feature operator is extracted. Therefore, a high-dimensional feature vector is formed and the optimal XGBoost model is selected to identify the music spectral line location, then the line is deleted. The research results show that F-measure of this method is 97.19% on the test data, which illustrates that the method has a high accuracy and recall rate. F-measure is 96.43%, 98.36% and 96.79% respectively on three different test subsets, which shows that it has good robustness. Compared with existing lightweight spectrum line deletion algorithm, the F-measure of this method is relatively improved.