In order to meet the needs of space-borne hyper-spectral imaging, the parameters of optical system are determined based on analyzing application requirements. Rectangular aperture is used, instead of circle aperture, and doublet prism and single prism are used as dispersion elements to design visible near-infrared (VNIR) and short-wave infrared (SWIR) spectrometer respectively. Telecentric three-mirror anastigmat (TMA) is adopted to design the telescope objective. Principle of cemented prisms used to compensate the nonlinear dispersion of single prism is discussed. TMA and spectrometers are designed with Zemax optical software, and the field splitter along the track direction is used to achieve imaging of VNIR and SWIR respectively. According to the overall analysis, the optical system meets the indicators of spectral resolution and ground resolution. Using the rectangular aperture not only can reduce the size in longitudinal dimension, but also is helpful to the aberration correction, which is conducive to the practical application of engineering. The VNIR band dispersion result shows that doublet prism is a useful method to correct the nonlinear dispersion of prism.