To improve the robustness of object tracking in complex situations, a new algorithm based on adaptive feature updating is proposed. First, hierarchical deep and hand-crafted features are simultaneously extracted from the object, and multiple fusion feature experts are constructed through multi-feature fusion by using different linear combination methods. Second, the credibility score of each expert is computed and the highest score is selected as the tracking feature of the current frame. A position correlation filter is then constructed to predict the frame's target position. Finally, the reliability of the tracking result is detected. When this reliability is found to be lower than a certain threshold, the fusion feature updating mechanism is initiated, and the temporal and semantic informations are added to the re-track, which reduces the error accumulation of the model. The proposed algorithm is tested on OTB-2013 and OTB-2015 datasets, and the obtained results are compared with those of 9 recently developed popular algorithms. Our proposed algorithm demonstrates a higher success rate and better robustness in complex situations, such as fast motion, background clutter, motion blur, and deformation, than existing algorithms.