In specific environments, when an infrared sensor cannot detect a target, it is necessary to integrate polarization and infrared technologies. To obtain a clearer fused image, this study adopted a method based on a multiscale structure and feature image fusion to realize infrared and polarization image fusion. The algorithm decomposed the infrared image and polarization map into three independent parts: average intensity, signal intensity, and signal structure. An arctangent weight function was proposed for fusion in the average intensity part, the signal intensity adopted the maximum fusion principle, and the signal structure adopted a weighted average square based on the power function of the signal intensity for fusion, and finally, the fused image was reconstructed. To fuse faster and reduce computational complexity, the decomposition process was replaced with mean filtering, and the final fused image was obtained by upsampling and downsampling. To obtain a better fusion image, better fusion parameters were selected through an experimental comparison of different fusion parameters. Experiments showed that by using the proposed arctangent weight function and fusion parameter setting, the four evaluation indexes had advantages over the traditional multiscale algorithm and subjectively retained more texture details, improved contrast, and suppressed artifacts.