This study proposes an asymmetric infrared and visible image fusion method based on a contrast pyramid to save the feature information of infrared image and the detail information of visible image simultaneously. First, the contrast pyramid is used to decompose the high-frequency and low-frequency information of the infrared and visible images; then, the high-frequency part is fused by taking the largest absolute value, and the low-frequency part is processed differently by the method based on bilateral filtering. Second, the inverse transform of the contrast pyramid was used to obtain the fused image. Subjective visual and objective index evaluations were conducted on the fused image. The results show that the algorithm performs well in highlighting the target feature information and retaining detailed feature information.