In view of the problems of large matching calculation amount, too single constraints in the process of matching, and high rate of false matching in the extraction of image feature vector using the existing scale invariant feature transform algorithm, an improved matching algorithm is proposed. Considering the problem of feature description, selective independent component analysis algorithm is adopted to reduce the dimensionality of the feature vector for the decrease of the number and the dimension of the feature vector. In order to solve the problem of higher mismatching rate, a direction constraint is added to the constraint conditions, namely matching twice through the direction of the feature vector and the Euclidean distance to reduce the mismatching rate. The particle swarm algorithm is used to find the extremum of the function to reduce time consumption of the algorithm. The experimental results show that the improved algorithm can effectively increase the matching accuracy.