Aiming at the tracking drift problem due to object model update falsely in the online learning tracking algorithms, a simple but efficient solution is proposed. In the target area point trackers are uniformly sampled, which are assessed based on texture description in two consecutive frames point trackers and then the initial location of target is completed. Multi-dimensional feature spatio-temporal context model is used to output the precise position of object by the confidence map, the model update rate is decided combining with the confidence map and a multiscale update mechanism is proposed. Experimental results show that the proposed algorithm can complete the robust tracking under the condition of background interference, fast motion, occlusion, illumination changing and scale changing. In the video sequence of 320 pixel×240 pixel, the average tracking speed can keep in 55.1 frame/s, which meets real-time application requirement.