The infrared hyperspectral image data collected by the passive Fourier transform infrared (FTIR) scanning remote sensing imaging system has spatial and spectral information and can be used to identify, quantify and visualize toxic and harmful gases in the atmospheric environment. The system has high spectral resolution and non-contact and long-distance detection advantages. However, the single-frame image has a small number of pixels, and some have gas absorption or emission, which cannot be directly used for target detection in infrared hyperspectral images.This paper proposes an adaptive matched filter (AMF) detection method for leaking gas based on multi-frame infrared hyperspectral image data in the same area in a short time. The background spectra without target gas feature are screened out and used for maximum likelihood estimation of the background in the detection area and then applied to target gas leak detection in subsequent frames. The infrared hyperspectral image collected by the remote sensing experiment of SF₆ has four frames (120 pixels/frame) scanned in total. The data containing the target gas feature in the first three frames are removed, and the remaining background spectrum is used to calculate the maximum likelihood estimation of the background. The AMF detection of SF₆ is implemented on the fourth frame of infrared hyperspectral data pixel by pixel, and the result is compared with the SF₆ column concentration image retrieved by the nonlinear least square method. To verify the performance of multi-frame background in different detection spaces, adaptive subspace detection (ASD) based on orthogonal subspace, adaptive cosine detection (ACE) based on hybrid space, and maximum likelihood ratio detection based on oblique subspace (OGLRT) detects the fourth frame data separately. Compared with the SF₆ column concentration image, the results show that the multi-frame background is suitable for detection methods in different spaces. In addition, to study the influence of the target absorption spectrum on the background space. Adding multiple spectra containing the absorption of SF₆ to the background space after ROC curve inspection, the results show that mixing target features in the background space will reduce the detection performance of the AMF method.The false color image of AMF detection value can also be applied to a passive FTIR scanning remote sensing imaging system. The leakage source and diffusion tendency are more obvious than the column density false color image. The detection method based on hyperspectral data relies on the statistical feature of the overall background. Compared with the inversion algorithm of a single-pixel spectral band, it greatly reduces the dependence of the background.The AMF leak gas detection method based on the multi-frame background can be well applied to the passive FTIR scanning remote sensing imaging system and meet the requirements of online monitoring.