For a kind of high resolution space camera, on the basis of calculation and analysis of the optical system parameters, three kinds of optical system: coaxial three mirror (TMC) system, two imaging off- axis three mirror (Cook TMA) system and one imaging off-axis mirror (Wetherell TMA) system have been designed by Zemax software, and they can satisfy the requirements of the indicators. The advantages and disadvantages of the designed optical systems are analyzed, on the basis of considering of the fabrication, testing, calibration and the technology development of space camera, the Wetherell TMA system is selected as the final plan. The shape size of the camera is 1600 mm×1230 mm×545 mm, the focal length of the system is 4375 mm, the relative aperture is 1:9.94, and the viewing angle is 2°. The property of the optical system is simulated and validated, the optical transfer function, aberration and distortion are calculated and analyzed, each indicator is excellent. Finally, the system aperture and stray light elimination are analyzed and the tolerance of the optical system is allocate reasonably.