The traditional methods for the detection of the small dim targets could not detect the small dim targets accurately and efficiently in IR image sequence with complex background. A novel algorithm for detecting small dim target is proposed to solve the above problems. Through analyzing the infrared radiation characteristics and sequential motion characteristics of the small dim targets in IR image sequence, the difference between the small dim targets and interference points can be found. The candidate targets can be detected by searching maximum gray pixels, removing noise, removing the targets with slowly intensity change and removing the targets without having a 2D Gaussian distribution. The spatial-temporal pipeline is designed according to target position change. The position of the targets in the next frame could be predicted along the direction of the spatial-temporal pipeline. The real targets can be detected according to the energy change and area change of the targets in the IR image sequence. Experimental results show that the accuracy and efficiency of our proposed method is improved significantly comparing with the traditional small dim targets detection methods.