Abstract In consideration of technical characteristics of the infrared hyperspectral interferometric spectrometer, we analyze the formation mechanism of nonlinear responses of its detector. The influences of second-order and third-order nonlinear responses on spectra are studied by means of simulating interference data with high-order nonlinear errors. An iterative method is proposed to minimize the out-of-band distortion by cross-iteration, thus the correction coefficient is determined to correct the non-linear responses. The interference data observed by the blackbody at different temperatures are corrected by the cross-iteration method, and then the spectra are recovered. The spectral response of the undisturbed wavenumber is fitted to the blackbody radiance. The results show that the second-order nonlinearity response has a major impact on the out-of-band data, while the third-order nonlinearity response mainly affects the in-band data. So the in-band data will still have residual errors when only the second-order nonlinear response is corrected. The cross-iterative method can correct the nonlinear response of the detector, and the accuracy produced by the third-order nonlinearity correction is improved about 7.26% compared to the second-order correction. The corrected goodness of fit is improved by about 0.4%, and the corrected interference data are more accurate than before.