The Spatio-Temporal Context(STC) tracking algorithm has defects in feature representation and scale adaptive strategy. Undesired conditions, i.e. abrupt deformations, partial occlusions or scale variations of the object appearance, would severely degrade the performance of the tracker. In this paper, based on the improvement of the STC algorithm, an algorithm is proposed for merging template response according to the STC model and color histogram response to locate the target object. The color statistics-based model has a good complementary nature to the STC model. The STC tracker combining the color histogram response can be inherently robust to both motion blur and deformations. Moreover, another scale search strategy which is based on a multi-scale pyramid model is adopted to replace the scale module in STC tracker, and makes scale estimation more accurately and adaptively. Extensive experimental results on large-scale benchmark sequences show that the proposed algorithm exhibits better tracking performance and adaptability under the complex environment of different influencing factors while running at 134.2 frames/s on average.