In video tracking, the use of a single feature to describe the target is difficult to adapt to the changes in complex scenes. Futhermore, the scale change, deformation, occlusion of target and other factors will lead to tracking failure. In order to improve the robustness of tracking, an adaptive feature fusion and model updating tracking algorithm is proposed based on the circulant structure of with kernel, and the scale updating mechanism is also introduced. Firstly, the response maps are calculated using the gray and local binary pattern features of the target respectively and fused by the weights assigned according to the peak to sidelobe ratio(PSR), and the best location is estimated. The PSR of the fused response map is also used to judge the tracking quality to decide whether to update the model. Finally, according to the scale pyramid constructed with the histograms of oriented gradients features extracted around the target location,the scale correlation filter is trained to estimate the optimal scale of the target. The experiment selects the sequences with illumination variations, occlusion and scale changes from the visual tracker benchmark datasets. The results show that the proposed algorithm can track the target robustly in complex scenes, and the distance precision and success rate are also superior to the compared algorithms.