• Journal of Infrared and Millimeter Waves
  • Vol. 40, Issue 5, 696 (2021)
Wen-Qing ZHU1、2、3, Xin-Yi TANG1、3、*, Rui ZHANG1、2、3, Xiao CHEN1、2、3, and Zhuang MIAO1、2、3
Author Affiliations
  • 1Shanghai Institute of Technical Physics,Chinese Academy of Sciences,Shanghai 200083,China
  • 2University of Chinese Academy of Sciences,Beijing 100049,China
  • 3Key Laboratory of Infrared System Detection and Imaging Technology,Chinese Academy of Sciences,Shanghai 200083,China
  • show less
    DOI: 10.11972/j.issn.1001-9014.2021.05.017 Cite this Article
    Wen-Qing ZHU, Xin-Yi TANG, Rui ZHANG, Xiao CHEN, Zhuang MIAO. Infrared and visible image fusion based on edge-preserving and attention generative adversarial network[J]. Journal of Infrared and Millimeter Waves, 2021, 40(5): 696 Copy Citation Text show less
    Architecture of the proposed EAGAN. CA Block:channel attention block,SA Block:spatial attention block,BN:batch normalization,FC:fully connected layer,Conv:corresponding kernel size(k),number of feature maps(n)and stride(s)indicated for each convolutional layer
    Fig. 1. Architecture of the proposed EAGAN. CA Block:channel attention block,SA Block:spatial attention block,BN:batch normalization,FC:fully connected layer,Conv:corresponding kernel size(k),number of feature maps(n)and stride(s)indicated for each convolutional layer
    Architecture of Attention Block. GAP:Global Average Pooling,GMP:Global Max Pooling,r:scaling factor,Conv:corresponding kernel size(k),number of feature maps(n)and stride(s)indicated for each convolutional layer
    Fig. 2. Architecture of Attention Block. GAP:Global Average Pooling,GMP:Global Max Pooling,r:scaling factor,Conv:corresponding kernel size(k),number of feature maps(n)and stride(s)indicated for each convolutional layer
    Qualitative comparison of different algorithms on 4 typical infrared and visible image pairs. From top to bottom:visible image,infrared image,fusion results of ASR,GFF,GTF,DenseFuse,FusionGAN,RCGAN and our algorithm.
    Fig. 3. Qualitative comparison of different algorithms on 4 typical infrared and visible image pairs. From top to bottom:visible image,infrared image,fusion results of ASR,GFF,GTF,DenseFuse,FusionGAN,RCGAN and our algorithm.
    Qualitative comparison of different algorithms on 5 typical infrared and visible image pairs from TNO dataset. From left to right:Duine sequence,Nato_camp_sequence,Kaptein_1123,men in front of house and soldier_behind_smoke_3. From top to bottom:visible image,infrared image,fusion results of ASR,GFF,GTF,DenseFuse,FusionGAN,RCGAN and our algorithm.
    Fig. 4. Qualitative comparison of different algorithms on 5 typical infrared and visible image pairs from TNO dataset. From left to right:Duine sequence,Nato_camp_sequence,Kaptein_1123,men in front of house and soldier_behind_smoke_3. From top to bottom:visible image,infrared image,fusion results of ASR,GFF,GTF,DenseFuse,FusionGAN,RCGAN and our algorithm.
    Qualitative comparison of different algorithms on 4 typical infrared and visible image pairs from INO dataset. From left to right:ParkingSnow,GroupFight,MultipleDeposit,ClosePerson. From top to bottom:visible image,infrared image,fusion results of ASR,GFF,GTF,DenseFuse,FusionGAN,RCGAN and our algorithm.
    Fig. 5. Qualitative comparison of different algorithms on 4 typical infrared and visible image pairs from INO dataset. From left to right:ParkingSnow,GroupFight,MultipleDeposit,ClosePerson. From top to bottom:visible image,infrared image,fusion results of ASR,GFF,GTF,DenseFuse,FusionGAN,RCGAN and our algorithm.
    Attention weight maps:(a)the infrared image;(b)the visible image;(c)the fused result of our proposed EAGAN;(d)Output result of the third attention block;(e)Channel Attention weight map;(f)Spatial Attention weight map
    Fig. 6. Attention weight maps:(a)the infrared image;(b)the visible image;(c)the fused result of our proposed EAGAN;(d)Output result of the third attention block;(e)Channel Attention weight map;(f)Spatial Attention weight map
    The effect of attention mechanism on fusion results:(a)fusion result of the network without attention mechanism;(b)fusion result of our algorithm.
    Fig. 7. The effect of attention mechanism on fusion results:(a)fusion result of the network without attention mechanism;(b)fusion result of our algorithm.
    Fusion results when the loss function of the generator changes:(a)ℒG=λ1ℒperceptual;(b)ℒG=λ2ℒedge;(c)ℒG=ℒEAGANG;(d)ℒG=ℒEAGANG+λ1ℒperceptual;(e)ℒG=ℒEAGANG+λ2ℒedge;(f)ℒG=λ1ℒperceptual+λ2ℒedge;(g)result of EAGAN.
    Fig. 8. Fusion results when the loss function of the generator changes:(a)G=λ1perceptual;(b)G=λ2edge;(c)G=EAGANG;(d)G=EAGANG+λ1perceptual;(e)G=EAGANG+λ2edge;(f)G=λ1perceptual+λ2edge;(g)result of EAGAN.
    ASRGFFGTFDenseFuseFusionGANRCGANOurs
    EN6.927.267.267.246.847.147.30
    SCD1.271.230.981.710.861.191.54
    SF13.1113.609.1211.979.359.5015.62
    EI0.220.220.160.200.160.180.27
    Table 1. Quantitative comparison of different algorithms on RoadScene dataset
    ASRGFFGTFDenseFuseFusionGANRCGANOurs
    EN6.446.846.936.876.356.777.08
    SCD1.611.360.971.791.301.411.67
    SF8.939.558.318.546.597.4111.59
    EI0.130.140.130.140.110.130.19
    Table 2. Quantitative comparison of different algorithms on TNO dataset
    ASRGFFGTFDenseFuseFusionGANRCGANOurs
    EN6.947.147.027.096.626.977.23
    SCD1.401.291.031.691.021.181.53
    SF16.8017.3314.7214.3412.7113.1219.40
    EI0.250.260.210.220.190.210.30
    Table 3. Quantitative comparison of different algorithms on INO dataset
    ENSCDSFEI
    RoadScene无注意力机制方法7.261.5216.020.27
    本文方法7.301.5415.620.27
    TNO无注意力机制方法6.931.6311.620.19
    本文方法7.081.6711.590.19
    Table 4. Comparison of effects of attention mechanism on fusion results
    Wen-Qing ZHU, Xin-Yi TANG, Rui ZHANG, Xiao CHEN, Zhuang MIAO. Infrared and visible image fusion based on edge-preserving and attention generative adversarial network[J]. Journal of Infrared and Millimeter Waves, 2021, 40(5): 696
    Download Citation