Aiming at the problem that the traditional visible and infrared image fusion methods usually present missing directional and anisotropic information in the fusion image, a two-level decision rule based infrared and visible image fusion method is proposed. This method fuses anisotropic diffusion (AD) and extreme value rules. Firstly, an AD filter is used to decompose each input source image into an approximation layer and a detail layer. Then, the two-level fusion decision rules are applied to the detail layer and the approximation layer, respectively, to preserve the spectral information and structural information. Finally, the structural similarity index (SSIM), average gradient (AG), and information entropy (IE) can be used to evaluate the image quality of the fused image. Experimental results show that compared with other existing image fusion methods, the fusion image of the proposed method in this paper can obtain more information and effectively reduce the artifacts, which verifies the effectiveness of the proposed method.