In order to solve the problems of multiple types of road surface diseases, large scale changes in the scale, and small sample data sets in road surface disease detection, a road surface multi-scale disease detecting algorithm based on improved YOLOv4 is proposed. First, the depth separable convolution method is used to replace the ordinary convolution method in the CSPDarknet-53 backbone network, which reduces the amount of network parameter calculations. Then, the loss function of YOLOv4 is improved based on the focal loss, which solves the problem of low detection accuracy caused by the imbalance of positive and negative samples in the process of network training. Finally, the YOLOv4 network is pre-trained with the help of transfer learning ideas, and the data set is expanded using methods such as flipping, cropping, brightness conversion, noise disturbance and other methods, so as to solve the problem of over-fitting of network training caused by insufficient samples of road surface disease. Experimental results show that, compared with original YOLOv4 detection network, the mean average precision of road surface disease detection based on YOLOv4+DC+FL algorithm can reach 93.64%, which increases by 3.25%. The detection time is 35.8 ms per picture, which is reduced by 7.9 ms.