In order to solve defect of the traditional wavelet transform, a novel image fusion algorithm based on patch-based directional wavelet was proposed. The image was divided into many patches, and the directional wavelet was obtained by training each patch, and then the sparse coefficients of input images were acquired by sparsely represented using the directional wavelet. The fused image was reconstructed by transforming the coefficients reversely, and the simulation experiment was used to test performance of the algorithm. The experimental results show that the proposed algorithm makes the image fusion transition effect more natural compared with the traditional algorithm such as DTC, FFT and DWT. Because it’s image fusion quality evaluation indexes are better in the information entropy and the average gradient. At the same time, the image fusion is faster and it can meet the demand of real-time image processing system.