Aiming at resolution and weak signal detection of Raman spectrometer, an optical system for Raman spectrometer is designed using transmission volume holographic grating. The transmission volume holographic grating is designed based on the coupled wave theory of Kogelnik. The diffraction efficiency of the grating at blaze wavelength reaches 90% and the average efficiency is higher than 80% in the range of 800~1000 nm. According to the aperture and field of view, Tessar-type split triplet is used as the basic structure of the focusing lens. The spot diagram root mean square (RMS) radius of the optimized system is less than 6 μm and the resolution reaches 0.3 nm. The aberration of the transmission grating system designed in this paper is small, which leads to high spectral resolution. By using volume holographic grating with high efficiency and low stray light, the system will perform better in weak signal detection. The system can be used in Raman spectrometer and improve its resolution and weak signal detection ability.