For radiation measurement using infrared imaging systems, gray value drift caused by ambient temperature variation is a significant source of error. By studying the effects of ambient temperature on the output gray value of cooled infrared systems, a drift compensation method is proposed. The sources of gray value drift are analyzed, and the relationship between drift and ambient temperature is derived to establish a theoretical model. A simple drift correction method is proposed. The radiometric calibration experiment is carried out for verification. The results indicate that proposed method can not only improve the measurement accuracy of the infrared imaging system, but also achieve drift compensation under arbitrary integration time, which improves the efficiency of radiometric calibration and measurement.