To solve the problem that infrared polarization and intensity image fusion algorithm is not optimal selection along with the change of difference characteristics, a method of establishing the set-valued mapping between difference characteristics set and fusion algorithms set was presented. Different characteristics set was formed with difference characteristics which was obtained by the analysis and extraction of image characteristics, and fusion algorithm set consisted of typical fusion algorithms. Fusion effective measure of each characteristic corresponding to fusion algorithms was calculated by data envelopment analysis and then constructed to be a distribution. Multi-group fusion effective measure distributions were synthesized in order to establish the set-valued mapping of difference characteristics and fusion algorithms set. Experimental results show that the set-valued mapping can select the optimal fusion algorithm, and have the highly complementary characteristics fused effectively.