Aiming at the new generation of infrared staring imaging sensors, primary physical effects modeling of imaging system elements was accomplished based on thermal infrared radiative transfer and energy conversion in imaging physical processes. The simulation model, which is different from those qualitative models available now, quantitatively represents the modulation characteristics of input information by design parameters of imaging system elements. Through building the signal response model, spatial transfer model, spatial sampling model, and noise characterization model for infrared imaging sensors, a perfect simulator could be established and high-fidelity simulation could be realized for the imaging process. For credibility validation of the simulation model, an experiment platform was constructed to capture the system performance curve and output image of a real infrared camera. Subsequently the similarities of performance curves and output images between the simulator and the real device were calculated and compared, and a quantitive evaluation of the simulation fidelity was made. Finally, the credibility of the simulation model was validated by the fidelity data.