In view of the fact that the traditional tracking learning detection (TLD) algorithm has poor robustness, low tracking success rate and low computing efficiency, a TLD tracking algorithm combining binary robust invariant scalable keypoints (BRISK) feature points and region prediction is proposed. In the tracker, the BRISK feature point is combined with the conventional pixel points used for tracking, and they are used for target tracking together. Due to the fast extraction of BRISK feature points, the total computing time of the tracker is reduced. In the detector part, the combination of Kalman filter and Markov model direction predictor greatly reduces the number of sub-image blocks sent to the detector, and enhances the identification ability for similar targets, thereby improving the speed and accuracy of the detector. The experimental results show that, compared with the traditional TLD algorithm, the tracking accuracy of the proposed TLD algorithm is improved by about 64.4%, and the running speed is increased by about 39.6%, and its robustness is better.