Aiming at the problems that the edges of the targets object are blurred and the details are lost in the fusion process of infrared and visible images, an infrared and visible image fusion method based on rolling guidance filter (RGF) and convolution sparse representation (CSR) is proposed. First, RGF and Gaussian filter are used to decompose the matched source images. Second, aiming at the base layers, the comparison saliency map and weight matrix are constructed to fuse. Thereafter, aiming at the detail layers, the alternating direction multiplier method (ADMM) is used to solve the convolution sparse coefficients and the feature response coefficients are fused using CSR fusion rule. Finally, the graph of the fusion result is rebuilt. The experimental results show that the proposed method can avoid losing the details caused by blurring at the edge of the objects, preserve the contrast and the edge texture information of the source images, and improve the objective evaluation indexes.