On the basis of analyzing the design theory of infrared imaging spectrometer based on binary optical element (BOE), in order to eliminate the limitation, which the focal length vary with incidence wavelength in infrared optical system with binary optical component as transparent imaging element, the hybrid refraction/diffraction zoom optical system combining the off-axial three-mirror reflective system is introduced. Furthermore, arming at the principle of infrared hot detection of military objects, the concretely design example of infrared imaging spectrometer was given. The design results show that The system possesses simple structure, high acceptance of light energy, long back-working distance, and better aberration correction. In addition, the system has lower requirements for materials and higher detection precision to infrared focal plane of the stare array detector.