To solve the problem that fusion algorithm is not optimal selection along with the change of difference characteristics when dealing with infrared polarization and infrared intensity of dual-mode image, a method of set-valued mapping fusion is presented. Multi-group of polarization and intensity of infrared images as training samples, different classification of low-level features and fusion algorithms are used to construct difference characteristics class set and fusion algorithm class set. Fusion effective measure distribution of each characteristic corresponding to fusion algorithms is constructed. Mean aggregating for multi-group images fusion effective measure distributions in order to establish the set-valued mapping relationship between difference characteristics and fusion algorithms of the global image. Difference characteristics of test images are extracted. Pick out corresponding algorithms according to the set-valued mapping. Then, get the fusion result by the combination of the algorithms. Experimental results show that the fusion method can select the optimal fusion algorithm, and have the highly complementary characteristics fused effectively.