For the problems about robust tracking and precision scale estimation of the visual objects in the complex tracking conditions, a target scale adaptive robust tracking algorithm based on the fusion of multilayer convolutional features is proposed. First, the multilayer convolutional features are extracted from the target candidate area using VGG-Net-19 deep convolutional network architecture. By constructing the two-dimensional location filters by correlation filtering algorithm and fusing the multilayer convolutional features, the center location of the target is determined. Then, through the multi-scale sampling of target, the histogram of oriented gradient features are extracted to construct the one-dimensional scale filter to achieve the optimal scale estimation. The experimental results show that the proposed algorithm gains the best success rate and precision compared with the six state-of-the-art methods. Meanwhile, this algorithm achieves an adaptive tracking to the fast scale changing of target effectively, and possesses a fast tracking speed.