Due to the less information of distant target, it is always challenging to accurately track the target in the task of infrared dim small target tracking. To improve the accuracy, based on correlation filtering framework, the side window filtering method which can extract the edge features of small infrared target is introduced, and an algorithm of distant target tracking is proposed. Specifically, the side window filtering method is used to process the searching area of the current target, this method could restrain the negative influence of background edge on dim small target location. Next, the correlation filters tracking model is constructed with temporal and spatial regularities to achieve accurate target tracking. To verify the performance of the proposed algorithm, six groups of real infrared dim small target image sequences were used for experiments, and the algorithm is compared with other typical algorithms such as KCF, SRDCF and STRCF. The experimental results show that the algorithm could effectively solve the problems of fast motion, low resolution and strong light background in infrared dim small target tracking tasks, getting higher accuracy with image sequences and complex background.