In the traditional image fusion method based on sparse representation, image blocks are used as units for dictionary training and sparse decomposition. The representation ability of dictionary atoms for image features is insufficient if the internal connection between the image blocks is not considered. Moreover, the sparse coefficients are inaccurate. Therefore, a fused image is not desirable. In view of the abovementioned problem, this paper proposes a fusion method based on the group K-means singular value decomposition (K-SVD) for visible and infrared images. Considering the image non-local similarity, this method constructs a structure group matrix using similar image blocks, and then, dictionary training and sparse decomposition are performed in the units of the structure group matrix by group K-SVD. Thus, this method can effectively improve the representation ability of dictionary atoms and the accuracy of the sparse coefficients. The experimental results show that this method is superior to the traditional sparse fusion method in terms of subjective and objective evaluation.