Aiming at the problem of low accuracy of traditional infrared image target classification methods, a method of SVM based on PSO combined with multi-feature fusion is proposed. HOG and LBP are used to describe the contour features and local textures of targets in infrared images. The method shows the characteristics of infrared image from different aspects, so there is a certain complementarity in the expression of image features. After feature extraction, the convex hull algorithm is used to calculate the sample data, and some representative sample data are obtained, so as to improve the efficiency of classification calculation. In the training of classification model, PSO is used to optimize SVM to find the optimal penalty factor and kernel parameters of SVM, so as to improve the accuracy of classification model. The experimental results show that the accuracy of the multi-feature fusion classification model is nearly 10% higher than that of the single-feature classification model, and the classification accuracy of the final SVM model optimized by PSO is as high as 99%.