Soft Histogram of Gradients Loss: A loss Function for Optimization of the Image Fusion Networks

Yuxin Long; Wenjie Lai; Huaiyuan Zhang; Hongbo Zhang; Chengshi Li; Ziji Liu

doi:10.3788/LOP230882

Journals >Laser & Optoelectronics Progress >Volume 60 >Issue 24 >Page 2411001 > Article

Laser & Optoelectronics Progress
Vol. 60, Issue 24, 2411001 (2023)

Soft Histogram of Gradients Loss: A loss Function for Optimization of the Image Fusion Networks

Yuxin Long, Wenjie Lai, Huaiyuan Zhang, Hongbo Zhang..., Chengshi Li and Ziji Liu^*|Show fewer author(s)

Author Affiliations

College of Optoelectronic Science and Engineering, University of Electronic Science and Technology, Chengdu 611731, Sichuan, China

show less

DOI: 10.3788/LOP230882 Cite this Article Set citation alerts

Yuxin Long, Wenjie Lai, Huaiyuan Zhang, Hongbo Zhang, Chengshi Li, Ziji Liu. Soft Histogram of Gradients Loss: A loss Function for Optimization of the Image Fusion Networks[J]. Laser & Optoelectronics Progress, 2023, 60(24): 2411001 Copy Citation Text

show less

Fig. 1. Structure of three training networks . (a) Structure of NestFuse ^[3]; (b) structure of Res2Fusion^[7]; (c) structure of UNFusion^[8]

Download full size

Image comparison fused by different networks on the TNO dataset. (a) (b) Source images; (c) NestFuse; (d) NestFuse-HOG; (e) Res2Fusion; (f) Res2Fusion-HOG; (g) UNFusion; (h) UNFusion-HOG

Fig. 2. Image comparison fused by different networks on the TNO dataset. (a) (b) Source images; (c) NestFuse; (d) NestFuse-HOG; (e) Res2Fusion; (f) Res2Fusion-HOG; (g) UNFusion; (h) UNFusion-HOG

Download full size

Fig. 3. Image comparison fused by different networks on the RoadScene dataset. (a) (b) Source images; (c) NestFuse; (d) NestFuse-HOG; (e) Res2Fusion; (f) Res2Fusion-HOG; (g) UNFusion; (h) UNFusion-HOG

Download full size

Fig. 4. Fusion image comparison before and after loss function applied in NestFuse network. (a) (b) Source image 1; (c) image of source image 1 fused by NestFuse network; (d) image of source image 1 fused by NestFuse-HOG network; (e) (f) source image 2; (g) image of source image 2 fused by NestFuse network; (h) image of source image 2 fused by NestFuse-HOG network

Download full size

Fig. 5. Image comparison fused by different networks on the TNO dataset. (a) (b) Source images; (c) NestFuse; (d) NestFuse-grad; (e) NestFuse+grad; (f) NestFuse-perceptual; (g) NestFuse-HOG

Download full size

Fig. 6. Image comparison fused by different networks on the RoadScene dataset. (a) (b) Source images; (c) NestFuse; (d) NestFuse-grad; (e) NestFuse+grad; (f) NestFuse-perceptual; (g) NestFuse-HOG

Download full size

Network	Image intensity loss	Structural similarity loss
NestFuse	MSE	MS-SSIM
Res2Fusion	MSE	MS-SSIM

Table 1. Original loss function of the network

Network	EN	SD	MI	SCD	MS-SSIM	$F M I_d c t$	$F M I_w$	$Q_{a b f}$	VIF
NestFuse	6.9198	82.7523	13.8397	1.7335	0.8625	0.3580	0.4371	0.4870	0.7865
NestFuse-HOG	6.9435	84.5295	13.8870	1.7491	0.8684	0.3426	0.4323	0.4892	0.7995
Res2Fusion	6.7963	77.6068	13.5927	1.7365	0.8681	0.3745	0.4373	0.4760	0.7164
Res2Fusion-HOG	6.8087	78.5593	13.6173	1.7398	0.8682	0.3721	0.4362	0.4760	0.7271
UNFusion	6.9070	83.6454	13.8139	1.7097	0.8505	0.3506	0.4380	0.4940	0.8061
UNFusion-HOG	6.9210	84.8536	13.8419	1.7131	0.8500	0.3486	0.4412	0.4964	0.8165

Table 2. Quantitative comparison on the TNO dataset

Network	EN	SD	MI	SCD	MS-SSIM	$F M I_d c t$	$F M I_w$	$Q_{a b f}$	VIF
NestFuse	7.4418	79.6857	14.8835	1.6504	0.8555	0.3446	0.4395	0.5077	0.9009
NestFuse-HOG	7.4604	80.4892	14.9209	1.6688	0.8597	0.3230	0.4332	0.5062	0.9172
Res2Fusion	7.3300	73.6833	14.6599	1.6476	0.8630	0.3697	0.4326	0.5083	0.7806
Res2Fusion-HOG	7.3469	74.5431	14.6938	1.6526	0.8616	0.3665	0.4318	0.5099	0.7947
UNFusion	7.4137	79.4455	14.8275	1.6355	0.8442	0.3386	0.4361	0.5102	0.9033
UNFusion-HOG	7.4231	80.0186	14.8462	1.6369	0.8416	0.3392	0.4406	0.5119	0.9127

Table 3. Quantitative comparison on the RoadScene dataset

Network	EN	SD	MI	SCD	MS-SSIM	$F M I_d c t$	$F M I_w$	$Q_{a b f}$	VIF
NestFuse	6.9198	82.7523	13.8397	1.7335	0.8625	0.3580	0.4371	0.4870	0.7865
NestFuse-grad	6.9091	82.2634	13.8181	1.7390	0.8652	0.3443	0.4337	0.4868	0.7781
NestFuse+grad	6.9297	83.5418	13.8593	1.7433	0.8665	0.3489	0.4347	0.4851	0.7928
NestFuse- perceptual	6.8998	81.2038	13.7995	1.7347	0.8630	0.3500	0.4353	0.4878	0.7684
NestFuse-HOG	6.9435	84.5295	13.8870	1.7491	0.8684	0.3426	0.4323	0.4892	0.7995

Table 4. Quantitative comparison on the TNO dataset

Network	EN	SD	MI	SCD	MS-SSIM	$F M I_d c t$	$F M I_w$	$Q_{a b f}$	VIF
NestFuse	7.4418	79.6857	14.8835	1.6504	0.8555	0.3446	0.4395	0.5077	0.9009
NestFuse-grad	7.4475	79.8023	14.8950	1.6603	0.8585	0.3270	0.4353	0.5061	0.9032
NestFuse+grad	7.4394	79.3834	14.8789	1.6566	0.8577	0.3345	0.4366	0.5056	0.8977
NestFuse- perceptual	7.4368	79.3249	14.8735	1.6522	0.8559	0.3346	0.4375	0.5076	0.8947
NestFuse-HOG	7.4604	80.4892	14.9209	1.6688	0.8597	0.3230	0.4332	0.5062	0.9172