For the problem of small target detection in infrared image, a method based on the local saliency is proposed. The feature on the local gray-scale of small targets in infrared image is analyzed, and a criterion is proposed to check the feature of peak value. Based on the criterion to check peak value and the characteristic of small target on time domain, a fast algorithm is designed. Firstly, local max points are selected and the follow-up computing is limited to these points to reduce the computation. Then peak values are checked based on the decline of gray-scale. Finally, false targets caused by noise are removed based on the continuity between frames. Experiments show that this algorithm has a high processing speed, and can effectively filter out the random noise in the image.