Considering the existence of a large number of outliers caused by the difference of infrared spectral radiation properties of cloud background and using different sensors, a new multi-modal registration method with the introduction of outliers rejection mechanism was proposed in order to realize the accurate registration of dual band infrared images. First, outliers was robustly estimated by computing dense SIFT flow. Then, through gradient based framework with the cost function of normalized correlation coefficient the accurate registration of dual band infrared images was achieved. Experimental results show that the registration parameters can converge to the global optimization fleetly after the rejection of outliers, and the algorithm can still maintain high registration accuracy to dual band infrared images with poor correlation.