Infrared scene simulation can provide the necessary sample images for target recognition and tracking, and has very important meaning for algorithm's training, evaluation and reliability test. Infrared simulation based on geometric modeling is difficult to provide the realistic simulation effect. Moreover, its computation is very large. The simulation method of infrared scene image with image based rendering (IBR) technique that is free of geometric modeling is proposed. According to the characteristic of infrared image, the nonlinear iteration algorithm for fundamental matrix computation and the dynamic programming stereo matching algorithm with feature constrains are proposed. The simulated images are more realistic because the reference images for simulation are real images. Furthermore, without intricate geometric modeling,the method can provide real time realization. The experiment result shows the method can produce realistic infrared images.