Traditional methods of detecting and recognizing metal fatigue cracks by ultrasonic infrared thermal images mainly extract relevant thermal characteristics of infrared thermal images through image processing algorithms and match crack characteristics. This process is tedious and the recognition rate is low. Additionally, the effective characteristics need to be manually selected. Taking the advantages of active thermography and Convolutional Neural Network (CNN) in metal structure testing and automatic defect recognition, a vibrothermography crack detection and recognition method based on CNN was proposed. The specimens (metal platesin this work) were tested and thermal data sets were obtained by the proposed CNN-based vibrothermography. The designed convolutional neural network was applied to the feature extraction, recognition and classification of vibration-induced infrared thermal images with different crack sizes. In addition, the proposed method was compared with two common image classification network models and support vector machine. Experimental results show that the designed convolutional neural network can recognize and classify metal fatigue cracks with an accuracy of 100% on the experimental data sets, which is better than other network models and support vector machine, and can effectively detect and recognize metal fatigue cracks.