Aiming at the tracking drift problem caused by the RGB feature, similar appearance and scale change in complex scenes, an improved method of distractor-aware object tracking based on multi-feature fusion and scale-adaption is proposed. Firstly, the distractor-aware object models are established base on the RGB feature and the modified (histogram of oriented gradient) HOG feature, which are extracted from object, surrounding background, and distractors. Secondly, the candidates are extracted by dense sampling in likelihood maps, which are obtained by calculating every pixel in the search region. The locations of the target and the distractor are obtained by vote score and distance score, also the model updating method is given. The RGB feature is extracted to establish a model scale, and the multi-scale feature pyramid method is used to get templates at different scales. The optimal scale is obtained by comparison between the model scale and the template scales. The experimental results indicate that the proposed algorithm can well adapt to environmental variation including distractors, partially blocking and scale variation and outperforms the compared tracking methods in terms of the distance precision and overlap precision.