To fully extract and fuse typical features of infrared and visible images, an image fusion algorithm based on spatial multi-scale residual network is proposed. Firstly, the source image is input into an encoder network composed of spatial multi-scale residual modules, and through the task of image reconstruction, the encoder network is trained to automatically obtain important features. Then, a feature pyramid and a channel self-attention are introduced, the output of basic layer and detail layer by the endoder are fused to reduce scale noise, and the fused image is reconstructed by the decoder. Finally, qualitative and quantitative experiments on public datasets are carried out, and it is demonstrated that the imporved algorithm outperforms the alternatives on highlighting infrared image targets and preserving visible image texture details. Compared with the DDcGAN algorithm, the standard deviation and average gradient of the proposed algorithm have been improved by 12.91% and 47.41%, respectively.