The deep learning method based on convolutional neural network plays a very important role in promoting the automatic detection of rail surface damage. Therefore, a method based on convolutional neural network for rail surface damage detection is proposed. First, a branch network is added between the contraction path and extension path of the classic U-Net can assist U-Net to output the ideal segmentation graph. Then, the type-I RSDDs high-speed railway track dataset is taken as the test sample, and the test sample is amplified by means of data enhancement and fed into the improved U-Net for training and testing. Finally, the evaluation index is used to evaluate the proposed method. The experimental results show that the detection accuracy of the proposed method reaches 99.76%, which is 6.74 percentage higher than the highest level of other methods, indicating that the proposed method can significantly improve the detection accuracy.