Acousto-optic tunable filter (AOTF) imaging spectrometer can acquire image, spectrum and polarization information simultaneously. It has been successfully applied to the aerospace, agriculture, forestry, medicine, food safety, and other fields, thus having good development prospects. Optical system is one of the key parts in an AOTF imaging spectrometer to obtain information. Therefore, the optical systems of common AOTF imaging spectrometers are studied in this paper, and the working principle of AOTF and the general design scheme of the spectrometers are introduced. Furthermore, the optical design parameters of the whole system, such as the front, collimating and imaging sub-systems are calculated, and the common-aperture optical system is simulated. The front optical system has an off-axis three-mirror structure based on free-form surface, which matches with the aperture stop of the collimating system, and the collimating and imaging systems are off-axis three-mirror systems with front offset of the aperture stop. The simulation results show that the system has an MTF values close to the diffraction limit and greater than 0.57 at the bands of 0.4--1.0 μm, 1.0--3.0 μm, and 3.0--5.0 μm, a distortion less than 5%, and good imaging quality, in the case of the working band of 0.4--5.0 μm, F number 5, field of view angle of 5.3°.