A multi-target tracking framework is designed to detect small infrared targets under complex backgrounds when the signal-to-noise-ratio (SNR) is low. The infrared image’s optical flow field is calculated, based on which the mathematic methods inclnding segmentation of threshold and mathematical morphological filter are employed to detect the infrared target from the backgrounds. Based on the results, with the target moving continuously, the neighborhood predict arithmetic is used to we eliminate the noise generated in the process of detection. The track prediction method based on Kalman filter is given to solve the problems of target missing and track crossover. This framework avoids the interference of the noise and the confusion of multi-track through the moving characteristic of targets. The infrared sequence images got from long-wavelength infrared camera verify that this framework is effective to track the small infrared targets under complex backgrounds.