It is difficult to quickly and accurately identify wheelset tread defects using traditional image processing algorithms. We propose an algorithm to accomplish this using a dual deep neural network. The dual network is divided into a tread-extraction network and a defect-identification network. Based on the characteristics of the treads as a big target, we analyze and test the SSD network, and apply this network to extract the tread area from wheelset images. To improve the efficiency of tread defect recognition, after the tread image is extracted, we optimize the YOLOv3 network structure to obtain M-YOLOv3 for the characteristics of medium and small tread defect targets. The experimental results show that when extracting tread areas, the average precision (AP) of the SSD algorithm is the highest (99.8%). When identifying tread defects, the AP of the M-YOLOv3 network reaches 89.9%. Compared with the original YOLOv3, the image computing time of the M-YOLOv3 network is reduced by 7.1%, with the AP showing only a 0.6% loss. The results demonstrate the proposed algorithm’s high detection accuracy.