Compressed sensing-based spectral imaging systems need to decode the sampled data by a proper algorithm to obtain the final spectral imaging data. Traditional decoding algorithms based on single sparse domain transformation will lead to loss of spectral details. Addressing this problem, a solution is proposed by using transformation of two sparse domains. A signal was decomposed into a low frequency part and a high frequency part, sparse restoration was performed according to the characteristics of different frequencies, and then decoding was performed to obtain high-precision restored signals. In data verification, the OMP algorithm was firstly used to restore the spectral information profile in the frequency domain, then the IRLS algorithm was applied to compensate the spectral details in the spatial domain. The impact of different sparse transformations on parameter settings was analyzed, and the JDSD of different algorithm combinations was tested. Test and simulation results on 500 kinds of spectral data show that the joint solution of double sparse domains can greatly improve the fidelity of spectral restoration. With a sampling rate of 20%, the SAM and GSAM indexes are increased from 0.625 and 0.515 by traditional methods to 0.817 and 0.659, respectively. In the case of 80%sampling rate, the SAM and GSAM indexes are increased from 0.863 and 0.808 of traditional methods to 0.940 and 0.897, respectively. JDSD algorithm can maintain high-precision details such as spectral absorption peaks,which is of great significance.