Convolutional neural network (CNN) is gradually applied to the field of image fusion because of its excellent performance. For the fusion task of infrared image and visible image, because there is no label data, unsupervised learning modeling is of great significance. To solve this problem, an unsupervised end-to-end depth fusion algorithm is proposed. The algorithm can directly predict the fused image containing significant information of the source image from the input infrared source image and visible light source image. The proposed algorithm constructs an auto-encoder network and uses real datasets for training. The loss function used in the network is the image structure similarity index measure (SSIM), which is widely used in image fusion tasks. Specifically, an improved non reference image evaluation index is designed to calculate the loss function, so as to achieve the purpose of unsupervised training of the network. In addition, the attention mechanism is introduced into the model to further improve the fusion results. The proposed algorithm is compared with many fusion algorithms, experimental results show that the fusion results of the proposed algorithm are very competitive in both subjective evaluation and objective index evaluation.