The current image fusion methods mainly use image energy features to fuse layer content, and ignore the significant information of image, resulting in low contrast in the fusion image. In this paper, a method is proposed, which will fuse visible and infrared images based on the significant information of image. Firstly, a smooth transform is designed to decompose the visible and infrared images by using the L0 and L1 norms, and obtain the base layer and detail layer images with good edge features. Then, by using the frequency tuning method, the significant information in the infrared image is obtained to establish the fusion model of the base layer image, and get the fusion base layer image. Through the information entropy features of image, the fusion model of detail level image is constructed, and the fusion detail level image is obtained from the information relevance of different detail level images. The fusion image is obtained by summing the fusion detail layer image and the fusion base layer image. Experimental results show that this algorithm can better fuse visible and infrared images than current algorithms; its fusion results can not only highlight the target information, but also have better contrast.