Object Detection via Multimodal Adaptive Feature Fusion

Xiaoqiang Gao; Kan Chang; Mingyang Ling; Mengyu Yin

doi:10.3788/LOP230856

Journals >Laser & Optoelectronics Progress >Volume 60 >Issue 24 >Page 2410003 > Article

Laser & Optoelectronics Progress
Vol. 60, Issue 24, 2410003 (2023)

Object Detection via Multimodal Adaptive Feature Fusion

Xiaoqiang Gao¹, Kan Chang^1、2、*, Mingyang Ling¹, and Mengyu Yin¹

Author Affiliations

¹School of Computer and Electronic Information, Guangxi University, Nanning 530004, Guangxi, China

²Guangxi Key Laboratory of Multimedia Communications and Network Technology, Nanning 530004, Guangxi, China

show less

DOI: 10.3788/LOP230856 Cite this Article Set citation alerts

Xiaoqiang Gao, Kan Chang, Mingyang Ling, Mengyu Yin. Object Detection via Multimodal Adaptive Feature Fusion[J]. Laser & Optoelectronics Progress, 2023, 60(24): 2410003 Copy Citation Text

show less

Fig. 1. Structure of YOLOv5

Download full size

Fig. 2. Structure of MAFFNet

Download full size

Fig. 3. Structure of SFF

Download full size

Fig. 4. Structure of JAM

Download full size

Fig. 5. Qualitative comparison with other algorithms in FLIR dataset. (a) Truth image; (b) CFT; (c) ProbEn; (d) MAFFNet

Download full size

MAFF module	Parameter quantity /10⁶	GFLOPs	FPS /（frame·s^-1）	mAP50
MAFF module	Parameter quantity /10⁶	GFLOPs	FPS /（frame·s^-1）	Person	Car	Bicycle	All
Without MAFF module	76.47	195.8	25	0.864	0.912	0.652	0.809
MAFF_1	76.34	194.1	24	0.868	0.914	0.676	0.819
MAFF_2	75.95	194.1	24	0.872	0.910	0.681	0.821
MAFF_3	74.37	197.1	24	0.867	0.910	0.668	0.815
MAFF_1+MAFF_2	75.82	192.4	23	0.882	0.918	0.701	0.834
MAFF_1+MAFF_3	74.24	192.4	24	0.879	0.913	0.690	0.827
MAFF_2+MAFF_3	73.85	192.4	23	0.879	0.914	0.736	0.843
MAFF_1+MAFF_2 +MAFF_3	73.72	190.8	23	0.886	0.922	0.740	0.849

Table 1. Influence of the number of MAFF modules on object detection results

Module	Parameter quantity /10⁶	GFLOPs	FPS /（frame·s^-1）	mAP50
Module	Parameter quantity /10⁶	GFLOPs	FPS /（frame·s^-1）	Person	Car	Bicycle	All
JAM	76.47	195.8	23	0.857	0.905	0.685	0.816
SFF	73.72	190.7	24	0.883	0.917	0.721	0.840
JAM+SFF	73.72	190.8	23	0.886	0.922	0.740	0.849

Table 2. Influence of single component of fusion module on object detection results

Method	Parameter quantity /10⁶	GFLOPs	FPS /（frame·s^-1）	mAP50
Method	Parameter quantity /10⁶	GFLOPs	FPS /（frame·s^-1）	Person	Car	Bicycle	All
JAM+SFF-GAP	73.72	190.8	23	0.888	0.919	0.722	0.843
JAM+DSFF	74.23	191.2	21	0.881	0.912	0.734	0.844
JAM-ECA+SFF	73.72	190.8	23	0.891	0.920	0.725	0.845
JAM+SFF	73.72	190.8	23	0.886	0.922	0.740	0.849

Table 3. Influence of different component of fusion module on object detection results

Method	Backbone	Data	mAP50
Method	Backbone	Data	Person	Car	Bicycle	All
YOLOv5	CSPDarkNet	RGB	0.581	0.781	0.407	0.590
YOLOv5	CSPDarkNet	Thermal	0.791	0.887	0.538	0.739
CFR^［25］	VGG16	RGB+T	0.745	0.849	0.578	0.724
GAFF^［27］	VGG16	RGB+T				0.727
GAFF^［27］	ResNet18	RGB+T				0.729
CFT^［28］	CSPDarkNet	RGB+T	0.822	0.890	0.640	0.784
ProbEn^［36］	ResNet101	RGB+T	0.877	0.901	0.735	0.838
YOLOBase（ours）	CSPDarkNet	RGB+T	0.864	0.912	0.652	0.809
MAFFNet（ours）	CSPDarkNet	RGB+T	0.886	0.922	0.740	0.849

Table 4. Performance comparison of different algorithms in FLIR dataset

Data	Method	Detector	Parameter quantity /10⁶	GFLOPs	FPS /（frame·s^-1）	mAP50
RGB	YOLOv5	YOLOv5	46.64	114.6	38	0.590
Thermal	YOLOv5	YOLOv5	46.64	114.6	38	0.739
RGB+T	CFT^［28］	YOLOv5	206.26	13732.5	14	0.784
RGB+T	ProbEn^［36］	Faster R-CNN	107.18	339.3	17	0.838
RGB+T	YOLOBase（ours）	YOLOv5	76.47	195.8	25	0.809
RGB+T	MAFFNet（ours）	YOLOv5	73.72	190.8	23	0.849

Table 5. Comparison of complexity of various models

Method	Backbone	Data	mAP50	mAP75	mAP
YOLOv3^［3］	DarkNet	RGB	0.859	0.379	0.433
YOLOv3^［3］	DarkNet	Thermal	0.897	0.534	0.528
YOLOv5	CSPDarkNet	RGB	0.908	0.519	0.505
YOLOv5	CSPDarkNet	Thermal	0.946	0.722	0.619
CFT^［28］	CSPDarkNet	RGB+T	0.975	0.729	0.636
CCIFNet^［37］	ResNet50	RGB+T	0.976	0.726	0.641
MAFFNet	CSPDarkNet	RGB+T	0.977	0.783	0.671

Table 6. Performance comparison of different algorithms in LLVIP dataset

Xiaoqiang Gao, Kan Chang, Mingyang Ling, Mengyu Yin. Object Detection via Multimodal Adaptive Feature Fusion[J]. Laser & Optoelectronics Progress, 2023, 60(24): 2410003

Download Citation

Set citation alerts for the article

Tools

Set citation alerts for the article

Save the article for my favorites

Paper Information