Beginning with the analysis of the real-time performance and robustness of scale-invariant feature transform (SIFT), the paper optimized traditional algorithm in both aspects of matching time and accuracy. A multi-scale feature extraction algorithm was proposed which is based on moving target. This algorithm was devised in two improved ways. One way was that the size of the template could automatically adjust along with the change of the scale factor. The other was that a bidirectional feature points matching was introduced into the algorithm. With the improved algorithm, the weakness of the real-time performance and robustness was amended successfully for match between the images and the moving target. Finally, experiment showed that with the improved algorithm, the sequential frame images based on time-axis could be matched quickly and accurately.