The description of a target’s appearance greatly influences the performance of a correlation filter tracker. It is difficult to obtain an accurate description of target’s appearance using a single feature. Therefore, the target appearance description based on multiple features can improve the tracking performance in complex scenes. For robust object tracking in complex scenes, we propose an object tracking algorithm based on multiple features with adaptive fusion and context-aware. First, we introduced a context-aware framework and extracted single-layer convolution features from four context image patches around the target to establish the background information. As a single feature may not accurately describe the target appearance, two correlation filters were used for feature extraction. The first filter extracted three-layer convolution features as deep features through a convolutional neural network, and the second filter extracted information from the directional gradient histogram and color histogram to obtain shallow features. Then, the deep and shallow features were adaptively fused. Finally, the average peak-to-correlation energy was used to evaluate the confidence of the response and we decided whether to update the model. The proposed algorithm was evaluated on the OTB-2013 benchmark, and the results show that it achieves excellent performance regarding accuracy and success rate and shows superior tracking performance compared with other state-of-the-art tracking algorithms.