In this study, we propose a multi-scale kernel correlation filter algorithm for visual tracking based on the fusion of adaptive features to promote the robustness of visual tracking in complex scenarios and tackle the tracking failure problems that can be attributed to illumination variation, target deformation, scale variation, occlusion, etc. First, two kernel correlation filters are separately trained using two different features. Then, the peak side-lobe ratio of the responses and the correlation filter response consistency of two consequent frames are considered to be the weight factors for feature fusion. Meanwhile, an adaptive strategy is adopted to fuse two responses for estimating the position. Next, multi-scale image patches are sampled to construct a scale pyramid based on the estimated position center, and the Bayesian method is employed to estimate the optimal scale of the target. Finally, the tracking model is updated according to the confidence of the tracking result to prevent the deterioration of the model. 51 video sequences are selected for conducting tracking evaluation, and the visual tracking algorithms that exhibited excellent performances in recent years are compared with our proposed algorithm. The experimental results demonstrate that the proposed algorithm effectively reduces the interferences, including the illumination variation, target deformation, scale variation, and occlusion. High tracking accuracy and success rate can be achieved using the aforementioned sequences, and the overall performance of our algorithm is observed to be better than those of the comparison algorithms.