In order to accurately and easily obtain four wavelength-dependent Stokes parameters, the intensity modulated polarization spectral imaging technology was used. The process of the restoration of polarization spectrum was calculated and derived from polarization spectrum intensity modulation theory and Fourier transform demodulation analysis, and the basic structure of the system was obtained. According to the modular design concept, the forward telescope objective lens and Offner spectral dispersion system of the spectrometer optical system were designed, which achieved a spectral resolution of 1.24 nm in the 400-1 000 nm spectral range. And then the front polarization spectrum modulation module was designed based on spectral system parameter, so a complete design example was given. Finally, the reasonability of the polarization spectrum modulation module design and feasibility of the Fourier transform demodulation method were verified by experiments, which lays the foundation for the recovery of polarization spectrum.