According to the problem of error accumulation and matched feature points loss in the optical flow feature tracking method, a predictive frame and key frame algorithm framework is proposed based on the new Harris-SIFT feature representation method. The proposed target tracking algorithm was realized by combining optical flow motion estimation and local feature recognition. Predictive frame uses pyramid decomposition and recursive algorithm to compute the motion vectors from optical flow field characteristics. The proposed algorithm gets motion vector of the target and eliminates false matching point from motion vector histogram; when the number of matched point is less than 5, the key frames uses the Harris-SIFT feature point for local feature matching, and affine model was used for accurate target positioning and attitude correction. The experiment results show that the proposed algorithm still can continue to achieve reliable tracking in complex background, target occlusion or temporarily lost case.