With the wide application of broadband imaging spectrometer in the ocean remote sensor, the depolarizer in front of the optical system must satisfy the high requirement. Based on the theory of matrix optics, Lyot depolarizer is introduced. According to the numerical analysis, the relationship between the residual polarization and the angle and thickness of the depolarizer is analyzed in the wide wavelength range. Combined with the residual polarization and resolution requirement of the optical system, a depolarizer used in a broadband imaging spectrometer is designed. With in the wavelength from 400 nm to 950 nm, when the spectral resolution is 4 nm, the residual polarizations of the depolarizer are less than 2% for arbitrary polarization of linear polarized.