The panoramic band optical system has been widely used in the field of ultra-large field of view optics due to the characteristics of real-time imaging around the view range. For the traditional panoramic ring optical system integrating the refraction and reflection surfaces in a block lens, the light is repeatedly folded and reflected inside the lens to limit the size reduction of the head unit. Moreover, the infrared lens material has high density, low transmittance, and poor refractive index temperature stability, which is contradictory to the needs of high stability and lightweight optical remote sensors. Based on the aberration theory, the initial structure design method of the head unit of the infrared optical system with a panoramic ring and two mirrors is discussed in this paper. Q-Type aspherical surface was introduced into the panoramic head unit to increase the optimization variables. A panoramic band infrared optical system with two mirrors as the head unit is designed to describe the difficulty in aspherical surface processing. The modulation transfer function of the system is better than 0.5 at Nyquist frequency (20 lp/mm). The energy concentration of full-field pixels (within 25 μm×25 μm) is better than 65%, and the image quality evaluation results show that the image quality is good. This design has great improvement in reducing the size of the system and improving the efficiency of optical design optimization and meets the application requirements of real-time imaging with large field of view.