In order to make full use of the information extracted from the middle layer and prevent information from losing excessively, a new image fusion method based on a convolutional auto-encoder and a residual block is proposed, which is composed of an encoder, a fusion layer, and a decoder. First, the residual network is introduced into the encoder, the infrared and visible images are fed into the encoder, and the convolution layer and residual block are used to obtain the feature map of the image. Then, the obtained feature map is fused by using an improved fusion strategy based on L1-norm similarity, which is integrated into a feature map containing the salient features of the source image. Finally, the loss function is redesigned and the decoder is used to reconstruct the fused image. The experimental results show that compared with other fusion methods, the method effectively extracts and preserves the deep information of the source image, which makes the fusion result have certain advantages in both subjective and objective evaluation.