The cross-entropy value illuminates the information difference between the template image and the real-time image on the average. In this paper, aiming at solving the problem of the radiation distortion and geometry distortion in the system of the photoelectric pod, the image matching tracking algorithm based on cross-entropy is proposed. Since the cross-entropy minimum rule is propitious to image matching with abundant information, the algorithm mentioned is provided with the very good antinoise properties, and the capability of resisting radiation distortion and geometry distortion.The results of simulation experiment show that this algorithm has steady matching tracking ability and well adaptability when radiation distortion happens. So it is a practical algorithm.