Most tracking algorithms cannot solve the problem of scale variation and the existing scale solutions are redundant and fixed. To solve the problems, a fast and novel scale estimation method based on kernel correlation filtering framework is proposed, which is coarse-to-fine. Considering that the peak value of the response graph is not stable, the Euclidean distance of the detection response graph and the expected output graph are used as the reliability of the peak value. The product is taken as the final comparison result. Firstly, three scale factors are used to determine the direction of scale variation, and then solve the optimum in the direction of scale variation. The proposed algorithm is experimented on 26 benchmark sequences with scale variation attribute of OTB-100, and is quantitatively and qualitatively compared with other existing advanced tracking algorithms. The results show that the proposed method can solve the scale variation problem well. The proposed method is 18.8% higher in mean distance precision and 19.6% higher in area under curve than those of the kernel correlation filter. The tracking speed is 2.5 times of the accurate scale estimation for robust visual tracking, and is 6 times of the scale adaptive kernel correlation filter tracker with feature integration.