A kernel correlation filter (KCF) with step-by-step association framework is proposed to aim at the various challenges, such as camera sudden movement, occlusion, false detection and appearance similarity in multi-target tracking algorithms. Firstly, the accurate detection results are obtained by utilizing a target detector based on the convolutional neural network. Then, a fast tracker based on the KCF algorithm is established for each target by weighted fusion of the tracking results of the three features to predict the motion state of the target. In addition, in order to effectively reduce the number of fragmented trajectories, this algorithm associates the trajectories step by step through the confidence of tracklets, and uses the online random fern to re-detect the target in the case of occlusion. Finally, the scale in the KCF algorithm is adaptively updated by using the associated successful detection information. Experimental results illustrate that the proposed algorithm displays powerful and efficient tracking performance under various complicated conditions compared with the existing excellent algorithms.