Multimodal Remote Sensing Image Classification with Small Sample Size Based on High-Level Feature Fusion

Qi He; Yao Li; Wei Song; Dongmei Huang; Shengqi He; Yanling Du

doi:10.3788/LOP56.111001

Journals >Laser & Optoelectronics Progress >Volume 56 >Issue 11 >Page 111001 > Article

Laser & Optoelectronics Progress
Vol. 56, Issue 11, 111001 (2019)

Multimodal Remote Sensing Image Classification with Small Sample Size Based on High-Level Feature Fusion

Qi He¹, Yao Li¹, Wei Song¹, Dongmei Huang^1、2、*, Shengqi He¹, and Yanling Du¹

Author Affiliations

¹ College of Information Technology, Shanghai Ocean University, Shanghai 201306, China

² Shanghai University of Electric Power, Shanghai 200090, China

show less

DOI: 10.3788/LOP56.111001 Cite this Article Set citation alerts

Qi He, Yao Li, Wei Song, Dongmei Huang, Shengqi He, Yanling Du. Multimodal Remote Sensing Image Classification with Small Sample Size Based on High-Level Feature Fusion[J]. Laser & Optoelectronics Progress, 2019, 56(11): 111001 Copy Citation Text

show less

Multimodal deep learning network. (a) Pixel-level fusion strategy; (b) feature-level fusion strategy; (c) high-level feature-level fusion strategy

Fig. 1. Multimodal deep learning network. (a) Pixel-level fusion strategy; (b) feature-level fusion strategy; (c) high-level feature-level fusion strategy

Download full size

Fig. 2. Multimodal remote sensing image datasets. (a) High-resolution remote sensing images; (b) low-resolution remote sensing images

Download full size

Network structure	Formula	Output	Feature size
cov1	X'_cov1_k=ReLU(K_cov1_k*x_i+b₁)	$\begin{matrix} {\{X'_{cov 1 k}\}}_{k = 1}^{16} \end{matrix}$	208×208
pool1	X'_pool1=f $\begin{matrix} (X'_{cov 1 k} + b_{2}) \end{matrix}$	X'_pool1	104×104
cov2	X'_cov2_k=ReLU(K_cov2_k*X'_pool1_k+b₃)	$\begin{matrix} {\{X'_{cov 2 k}\}}_{k = 1}^{16} \end{matrix}$	104×104
pool2	X'_pool2=f $\begin{matrix} (X'_{cov 2 k} + b_{4}) \end{matrix}$	X'_pool2	104×104
ful1	X'_ful1=X'_pool21±X'_pool22±…±X'_pool2104+b₅	X'_ful1	1×173056
Z'_ful1	Z'_ful1=W₁X'_ful1±W₂Y'_ful1+b₆	Z'_ful1	128×128
ful2	Z'_ful2=Z'_ful1×K_ful1+b₇	Z'_ful2	1×128

Table 1. Network structure of feature extractor

Parameter	8kernels	16kernels	32kernels	8/batch	16/batch	32/batch
Time /s	1356.9	2456.2	4828.1	1431.4	2452.2	4418.32
Accuracy /%	95.3	90	92.6	50	90	82.2

Table 2. Influence of MRSIC model parameter setting on classification accuracy

Network structure	Dataset	Amount	Accuracy /%
THRSI-LeNet	NW	1000	83.2
THRSI-LeNet	NW	500	74.6
TLRSI-LeNet	UC	1000	77.3
TLRSI-LeNet	UC	500	67.2
H&L-LeNet	NW、UC	1000	82.5
HUL-LeNet	NW、UC	1000	72.0
NanoNet	UC	500	77.1
VGG	NW、UC	500	70.4
In Fig. 1(b)	NW、UC	1000	85.0
MRSIC	NW、UC	1000	90.5
MRSIC	NW、UC	500	90.0