The median filtering is a nonlinear signal processing technology based on the order statistical theory, it can effectively suppress the noise. The improved median filtering algorithm and the traditional 3×3 median filter are compared: the filtering mould board is composed of 2 FIFOs and 7 registers, simple and practical; by using 6-level comparison, 9 medians are found fast, which can improve the speed of finding median; in order to keep the image details, the obtained median and the original data are compared with the threshold to selectively replace. Using the algorithm, 3×3 window successfully achieves the median filtering of the image on the 640×512 element infrared focal plane array (IRFPA). This algorithm is simple and fast, it can protect the signal details well at the same time of filtering the noise, especially the pulse noise.