To achieve robust target tracking in a highly dynamic scene, a single target tracking algorithm with multi-feature fusion in multi-scale models is proposed. The proposed models can adaptively fuse multiple features to achieve real time tracking in complex scenes. To tackle the problems of target significant deformation and occlusion, the proposed algorithm computes the confidence of the observation and uses it to update the reference models adaptively. The tracker is tested on ten representative sequences in a standard tracking benchmark. Compared with some other state-of-the-art algorithms, the results demonstrate that the tracking precision has been improved in the highly dynamic scenes with target significant deformation.