An improved bilinear convolutional neural network (B-CNN) model is proposed to solve the problem of fine-grained classification of crack images of sleeper block shoulder. Using this model, the global information in the image features of the global average pooling link is first used to capture the width information of the fine crack. Then, the fusion of different levels is performed to enhance the ability of feature expression to obtain effective width features and fine-grained classification. Experimental results show that compared with the B-CNN model, the classification accuracy of this model improves by 2 percentage. In terms of the false negative rate, the normal category reduces by 2.3 percentage, and the obvious crack category reduces by 4.55 percentage. Compared with the baseline VGG-D (Visual Geometry Group Network-D) model (6.11 percentage classification accuracy), the normal false negative rate reduces by 7.39 percentage, and obvious crack category reduces by 8.39 percentage. Furthermore, the feature extraction rate for the original is 18.51%, whereas that of our proposed model is 45.31%, which shows that the proposed model can satisfy the need for rapid and accurate imaging of the shoulder for double block-type sleeper crack image classification to meet engineering requirements.