Reflective optical system has the characteristics of no chromatism, compact structure, small size, excellent imaging performance and so on, which makes it very popular in space remote sensing detection field. Firstly, based on the aberration expressions of coaxial four-mirror reflective optical system, the evaluation objective function of imaging performance is established taking root mean square of system wave aberration as standard. Then, the developed adaptive variant probabilities genetic algorithm is applied to solve the evaluation function, and a set of suitable optical parameter solutions of initial structure of the coaxial four-mirror reflective optical system are obtained. After that, based on the above solution system structure, the proper off-axis operation is performed on it to eliminate the obscurations, four optical surfaces are set as even aspheric surfaces and the aberration of the system is optimized by using the optical design software Zemax. Finally, an off-axis four-mirror reflective optical system with good imaging performance is designed, which has a focal length of 800 mm, a working wavelength of 0.4～1.6 μm, a full field of view angle of 1° and an pupil diameter of 80 mm. The results show that the proposed research method provides an effective means for designing the kind of off-axis reflective optical systems, and has strong practical value.