In this paper, an infrared point target detection algorithm based on two filters in spatially varying clutter is presented. The target and clutter model is analyzed correctly, and very small (point) target under spatially varying clutter background is successfully detected with the algorithm, when the target and clutter are both moving across the image scene. This method can reduce the cost of computation, track the target in real-time, enhance the SNR, and suppress the clutter effectively.