Multimodal Image Fusion Based on Generative Adversarial Networks

Xiaoli Yang; Suzhen Lin; Xiaofei Lu; Lifang Wang; Dawei Li; Bin Wang

doi:10.3788/LOP56.161004

Journals >Laser & Optoelectronics Progress >Volume 56 >Issue 16 >Page 161004 > Article

Laser & Optoelectronics Progress
Vol. 56, Issue 16, 161004 (2019)

Multimodal Image Fusion Based on Generative Adversarial Networks

Xiaoli Yang¹, Suzhen Lin^1、*, Xiaofei Lu², Lifang Wang¹, Dawei Li¹, and Bin Wang¹

Author Affiliations

¹ School of Big Data, North University of China, Taiyuan, Shanxi 0 30051, China

² Jiuquan Satellite Launch Center, Jiuquan, Gansu 735000, China

show less

DOI: 10.3788/LOP56.161004 Cite this Article Set citation alerts

Xiaoli Yang, Suzhen Lin, Xiaofei Lu, Lifang Wang, Dawei Li, Bin Wang. Multimodal Image Fusion Based on Generative Adversarial Networks[J]. Laser & Optoelectronics Progress, 2019, 56(16): 161004 Copy Citation Text

show less

Fig. 1. Structure of residual block

Download full size

Fig. 2. Framework of method

Download full size

Fig. 3. Network structure of generative model

Download full size

Fig. 4. Network structure of discriminative model

Download full size

Fig. 5. Pre-selection maps of label images. (a) Longwave infrared; (b) shortwave infrared; (c) visible light; (d) LP; (e) DWT; (f) NSCT; (g) NSST

Download full size

Fig. 6. Effect of learning rate on generator loss

Download full size

Fig. 7. Effect of learning rate on discriminator loss

Download full size

Fig. 8. Effect of different λ on image quality. (a) λ=0; (b) λ=0.01; (c) λ=0.1; (d) λ=1

Download full size

Fig. 9. Effect of different λ on generator loss

Download full size

Fig. 10. Effect of λ on objective evaluation index of fused image. (a) The first set of fused images; (b) the second set of fused images; (c) the third set of fused images

Download full size

Fig. 11. Image fusion results. (a) Longwave infrared; (b) shortwave infrared; (c) visible light; (d) DTCWT_SR; (e) NSST_NSCT; (f) CNN; (g) CSR; (h) proposed method

Download full size

Layer	Filter size /step	Output size
Conv1	3×3 /1	128×128×64
Res(7 units)	3×3 /1	128×128×64
	3×3 /1	128×128×64
Conv9	3×3 /1	128×128×256
Conv10	3×3 /1	128×128×1

Table 1. Parameters of generator

Layer	Filter size/step	Output size
Conv1	3×3 /1	128×128×64
Conv2	3×3 /2	64×64×128
Conv3	3×3 /2	32×32×256
Conv4	3×3 /2	16×16×512
Conv5	3×3 /1	16×16×256
Conv6	1×1 /1	16×16×128
Res	1×1 /1	16×16×64
	3×3 /1	16×16×64
	3×3 /1	16×16×128
Fc	-	1

Table 2. Parameters of discriminator

Fusion method	SD	AG	Con	CC	IE	MI	VIFF
LP	49.019	4.211	40.812	0.424	7.223	5.608	0.466
DWT	44.772	3.826	36.650	0.342	7.175	5.415	0.442
NSCT	41.123	4.061	34.816	0.437	6.953	5.222	0.469
NSST	40.904	4.149	34.902	0.441	6.932	5.201	0.467

Table 3. Label image selection table

Image	Fusion method	SD	AG	Con	CC	IE	MI	VIFF
No. 1	DTCWT_SR	35.471	3.150	23.916	0.409	6.968	4.869	0.505
	NSCT_NSST	19.621	3.163	11.911	0.406	6.147	2.853	0.522
	CNN	34.928	2.885	23.254	0.408	6.961	4.610	0.363
	CSR	12.081	1.953	7.570	0.418	5.534	2.663	0.359
	Proposed method	38.011	8.129	27.793	0.431	6.977	2.552	0.301
No. 2	DTCWT_SR	26.274	7.245	25.927	0.143	5.984	1.492	0.367
	NSCT_NSST	23.268	7.270	14.815	0.306	6.026	1.542	0.376
	CNN	29.696	4.985	22.754	0.016	2.373	1.312	0.211
	CSR	25.271	5.543	15.989	0.322	6.032	2.540	0.322
	Proposed method	25.527	4.374	13.731	0.381	6.057	2.567	0.461
No. 3	DTCWT_SR	21.854	5.187	15.038	0.427	6.475	1.256	0.415
	NSCT_NSST	22.692	5.370	15.7166	0.471	6.806	1.542	0.419
	CNN	38.590	4.961	25.609	0.441	6.910	2.799	0.392
	CSR	23.054	3.868	15.726	0.499	6.450	2.391	0.382
	Proposed method	41.089	4.310	30.929	0.593	6.938	3.093	0.420
No. 4	DTCWT_SR	47.492	3.545	23.176	0.422	6.315	2.773	0.298
	NSCT_NSST	35.373	3.605	15.136	0.445	6.831	2.531	0.317
	CNN	54.597	3.163	29.778	0.436	6.283	2.862	0.351
	CSR	38.587	2.217	17.656	0.452	6.796	3.331	0.375
	Proposed method	32.400	8.452	22.767	0.457	6.938	2.055	0.384
No. 5	DTCWT_SR	40.006	5.058	32.554	0.025	7.308	2.976	0.407
	NSCT_NSST	24.886	5.147	18.059	0.489	6.615	1.812	0.428
	CNN	40.559	3.951	34.331	0.271	7.316	2.133	0.346
	CSR	26.239	2.526	20.707	0.515	6.642	3.007	0.277
	Proposed method	40.677	5.974	33.890	0.569	7.321	2.106	0.473
No. 6	DTCWT_SR	54.458	3.038	44.020	0.042	7.744	2.986	0.524
	NSCT_NSST	25.990	3.047	18.728	0.455	6.686	2.254	0.552
	CNN	45.549	2.538	35.269	0.193	7.333	1.839	0.208
	CSR	28.803	1.523	21.125	0.429	6.691	2.858	0.160
	Proposed method	46.425	5.643	36.462	0.476	7.453	2.993	0.648
No. 7	DTCWT_SR	54.848	3.555	36.984	0.044	7.420	3.485	0.484
	NSCT_NSST	27.527	3.550	21.490	0.460	6.813	1.807	0.504
	CNN	45.880	2.945	37.129	0.334	7.104	2.627	0.440
	CSR	27.853	1.834	21.168	0.472	6.701	2.968	0.326
	Proposed method	47.024	4.259	37.523	0.500	7.487	2.461	0.606
No. 8	DTCWT_SR	55.890	2.940	47.062	0.771	4.357	3.612	0.169
	NSCT_NSST	52.610	4.872	47.647	0.868	4.832	3.906	0.380
	CNN	87.236	6.208	73.907	0.721	5.688	3.933	0.294
	CSR	54.208	4.753	47.214	0.794	4.245	3.383	0.285
	Proposed method	92.152	15.279	76.870	0.943	4.333	4.947	0.505

Table 4. Comparison of evaluation index of fusion results

Xiaoli Yang, Suzhen Lin, Xiaofei Lu, Lifang Wang, Dawei Li, Bin Wang. Multimodal Image Fusion Based on Generative Adversarial Networks[J]. Laser & Optoelectronics Progress, 2019, 56(16): 161004

Download Citation

Set citation alerts for the article

Tools

Set citation alerts for the article

Save the article for my favorites

Paper Information