In view of the real-time data processing requirements of Fourier transform infrared imaging spectrometer, a real-time spectrum recovery system on a FPGA chip for the spectrometer was studied which integrates the functions of interferogram nonuniformity correction, spectrum recovery and spectral radiation calibration. The system has made a standardized design of the input signals. It can be run in line way, output target's spectrum in real-time and be embedded in the real-time data processing system for various types of infrared/visible Fourier transform imaging spectrometer which has some specialties such as small volume, fast computing speed, good stability, high reliability and easy to upgrade. The system can establish a good technical foundation for the real-time target recognition based on the spectral characteristics.