Optical music recognition plays an important role in the field of music information retrieval and computer aided instruction. For traditional frameworks, the processing steps are complicated, and the accuracy is low. Moreover, deep learning algorithm-based model training takes a long time and shows large recognition error for difficult notes. In this work, an improved convolutional recurrent neural network is proposed. First, different noises were added to the original score to expand the score image and improve the robustness of the training model. Then, the multi-scale residual convolutional neural network was used to extract note features to improve the subsequent recognition accuracy. Finally, bi-directional simple recurrent units were adopted to recognize note features and accelerate convergence of the algorithm in the training stage. Experimental results show that the average symbol error rate of the proposed network model has been reduced to 0.3234%. Thanks to the faster converging rate, the training time is about one third of that of traditional convolutional recurrent neural network.