A general automatic fault recognition algorithm based on sparse-coding-based spatial pyramid matching and Genetic Algorithm Optimized Support Vector Machine (GA-SVM) was proposed for fault detection of the bogie block key, dust collector, and fastening bolt in the Trouble of moving Freight car Detection System (TFDS). First, the image of a sample was divided into patch areas in different scale spaces, and the Scale-Invariant Feature Transforms (SIFT) of each patch area was extracted. Sparse coding was then performed by iteratively learning dictionaries using the SIFT features of randomly extracted samples. Second, principal component analysis was used to define the contribution of the encoded features towards fault recognition accuracy and reduce the dimensionality of the coding features. Then, the SVM classifier was trained using the reduced dimension features after coding and optimization with the genetic algorithm. Finally, the trained classifier was used to detect the bogie block key, dust collector, and fastening bolt faults from their images. The experimental results show that the algorithm can adaptively recognize the three different kinds of faults. The fault recognition rates were 97.25%, 99.00%, and 97.50% for bogie block key, dust collector, and fastening bolts, respectively. This technique is robust to noise and illumination changes and can meet the actual detection requirements of a vehicle's structural faults.