The simulation of realistic infrared scene in space infrared down-looking system has always been a difficult problem in the research. A new kind of infrared scene simulation method of space down-looking system based on remote sensing inversion was presented in this paper. Firstly, infrared texture of the terrain was extracted from the data of infrared remote sensing inversion by using matching algorithm. Secondly, an infrared detecting model of down-looking system was proposed, which simulated resampling of infrared texture on FPA. Finally, by using radiation and transmission theory, scene radiance image in front of the entrance pupil was calculated. The simulation results show that the proposed method can realistically simulate infrared texture feature of earth surface. In this paper, thermal infrared remote sensing technology is applied to infrared scene simulation, which greatly simplifies the simulation process of large scale terrain scene, and provides an intuitive and effective new method for infrared scene simulation of space down-looking system.