A hyper-resolution spatial heterodyne spectrometer (SHS) used for hydroxyl radical OH detection with limb observations from a satellite is presented. The SHS instrument, consisting of two 1-D imaging channels in two orthogonal observation directions at different time for the same volume with the aid of the movement of the satellite platform, can get the three-dimensional distribution of OH with a series of observation radiation intensities. Each channel, including cylindrical telescope, collimating lens, a monolithic spatial interference unit and imaging lens, has vertical spatial resolution without scanning parts, and it can be field widened with prisms in the interferometer arms to increase the throughout of the instrument. An optical system for measuring mesospheric OH with a spectral resolution of 0.01 nm over the 1.6 nm ultraviolet passband 308.2-309.8 nm is designed, and its optimizing process and results are shown. The ground-based experiment with the prototype instrument is carried out with the use of the light from the sun tracker. The measurement results show that the optical system design meet the needs of OH detection requirements, and the instrument experimental performances are in agreement with the theoretical spectral characteristics. Our research provides a basis for remote sensing of mesospheric atmosphere detection in satellite.