Author Affiliations
1School of Software, Liaoning Technical University, Huludao 125105, Liaoning , China2Department of Basic Teching, Liaoning Technical University, Huludao 125105, Liaoning , Chinashow less
Fig. 1. MHFNet structure diagram
Fig. 2. Structure diagrams of two attention mechanisms. (a) MHSA; (b) MSIA
Fig. 3. Various FFN structure diagrams. (a) FFN; (b) IRFFN; (c) GDFN; (d) MGFN
Fig. 4. Structure diagrams of two token extractors. (a) STE; (b) CTE
Fig. 5. Scene instances of two datasets. (a) Airplane; (b) beach; (c) ship; (d) cloud; (e) island; (f) bridge; (g) parking; (h) center; (i) storage tank; (j) playground
Fig. 6. Confusion matrix of AID dataset under 50% training ratio
Fig. 7. Confusion matrix of NWPU-RESISC45 dataset under 20% training ratio
Fig. 8. Visual CAM comparison. (a) Basketball court; (b) bridge; (c) church; (d) freeway; (e) roundabout; (f) ship; (g) parking
Parameter | Value |
---|
CPU | i7-12700 | GPU | NVIDIA GeForce RTX 3090Ti 24 GB | Language | Python 3.9.12 | Operating system | Windows 11 | Deep learning framework | PyTorch 1.11.0+CUDA 11.3 |
|
Table 1. Experimental configuration
Parameter | Value |
---|
Epochs | 300 | Initial learning rate | 0.0002 | Final learning rate | 0.00001 | Optimizer | AdamW | Batch size | 128 | Warmup | 20 | Weight decay | 0.003 | Random seed | 42 |
|
Table 2. Key parameter setting
Method | Parameters /106 | NWPU(1∶9) | NWPU(2∶8) | AID(2∶8) | AID(5∶5) |
---|
VGGNet16[27] | 138.36 | 76.47 | 79.79 | 86.59 | 89.64 | GoogleNet[27] | 54.40 | 76.19 | 78.48 | 83.44 | 86.39 | ResNet50[29] | 23.58 | 86.23 | 88.93 | 92.39 | 94.96 | DenseNet121[30] | 7.98 | 88.31 | 90.47 | 93.76 | 94.73 | ViT-B[11] | 85.63 | 87.59 | 90.87 | 91.16 | 94.44 | PVT-M[31] | 43.32 | 90.51 | 92.66 | 92.84 | 95.93 | DeiT-B[32] | 85.97 | 91.86 | 93.83 | 93.41 | 96.04 | Swin-B[33] | 89.74 | 91.80 | 94.14 | 94.86 | 97.80 | EMTCAL[34] | – | 91.63 | 93.65 | 94.69 | 96.41 | GCSANet[35] | 8.11 | 93.39 | 94.65 | 95.96 | 97.53 | MGSNet[36] | – | 92.40 | 94.57 | 95.46 | 97.18 | EMSCNet(ViT-B)[37] | 173.64 | 93.58 | 95.37 | 96.02 | 97.35 | MHFNet | 41.85 | 94.09 | 95.73 | 97.12 | 98.63 |
|
Table 3. OA of different models on AID dataset and NWPU-RESISC45 dataset
Module | Parameters /106 | NWPU(1∶9) | NWPU(2∶8) | AID(2∶8) | AID(5∶5) |
---|
None | 31.18 | 92.27 | 94.10 | 95.38 | 96.93 | ATM | 32.53 | 92.88 | 94.74 | 96.05 | 97.57 | ATM+MSIT | 41.85 | 94.09 | 95.73 | 97.12 | 98.63 |
|
Table 4. Impact of MSIT and ATM on model performance
Option | Parameters /106 | NWPU(1∶9) | NWPU(2∶8) | AID(2∶8) | AID(5∶5) |
---|
N=0 | 32.53 | 92.88 | 94.74 | 96.05 | 97.57 | N=1 | 37.68 | 93.79 | 95.57 | 96.74 | 98.37 | N=2 | 41.85 | 94.09 | 95.73 | 97.12 | 98.63 | N=3 | 46.02 | 93.82 | 95.66 | 96.89 | 98.46 | N=4 | 50.18 | 93.74 | 95.59 | 96.76 | 98.38 |
|
Table 5. Impact of number of MSITs in each branch on model performance
Module | Parameters /106 | NWPU(1∶9) | NWPU(2∶8) | AID(2∶8) | AID(5∶5) |
---|
None | 32.53 | 92.88 | 94.74 | 96.05 | 97.57 | MSIA | 38.43 | 93.73 | 95.41 | 96.75 | 98.30 | MSIA+MGFN | 41.76 | 93.97 | 95.65 | 96.94 | 98.52 | MSIA+MGFN+LRC | 41.85 | 94.09 | 95.73 | 97.12 | 98.63 |
|
Table 6. Effectiveness of various components in MSIT
Method | Parameters /106 | FLOPs /G | NWPU(1∶9) | NWPU(2∶8) | AID(2∶8) | AID(5∶5) |
---|
MHFNet(ResNet50) | 34.61 | 4.71 | 92.63 | 94.58 | 95.39 | 96.94 | MHFNet(ViT-B) | 96.17 | 56.30 | 93.28 | 94.83 | 95.21 | 96.47 | MHFNet(Swin-B) | 102.38 | 16.09 | 93.74 | 95.80 | 96.83 | 98.92 | MHFNet(FasterNet-S) | 41.85 | 5.12 | 94.09 | 95.73 | 97.12 | 98.63 |
|
Table 7. Impact of different feature extractors on model performance
Category | OA | Precision | Recall | F1 score | Category | OA | Precision | Recall | F1 score |
---|
1 | 99.63 | 99.26 | 100.00 | 99.63 | 16 | 100.00 | 100.00 | 99.55 | 99.77 | 2 | 99.12 | 100.00 | 100.00 | 100.00 | 17 | 95.04 | 95.93 | 94.19 | 95.05 | 3 | 100.00 | 100.00 | 100.00 | 100.00 | 18 | 100.00 | 100.00 | 100.00 | 100.00 | 4 | 98.81 | 99.78 | 99.38 | 99.58 | 19 | 99.83 | 99.04 | 98.59 | 98.81 | 5 | 99.44 | 99.00 | 99.50 | 99.25 | 20 | 99.45 | 99.39 | 98.22 | 98.80 | 6 | 99.47 | 99.22 | 90.81 | 94.83 | 21 | 99.61 | 100.00 | 100.00 | 100.00 | 7 | 97.22 | 97.67 | 100.00 | 98.82 | 22 | 99.80 | 100.00 | 100.00 | 100.00 | 8 | 97.08 | 97.61 | 98.37 | 97.99 | 23 | 96.33 | 96.35 | 95.60 | 95.97 | 9 | 97.87 | 97.23 | 100.00 | 98.60 | 24 | 98.09 | 98.78 | 98.11 | 98.44 | 10 | 97.94 | 98.78 | 100.00 | 99.39 | 25 | 96.91 | 97.72 | 97.83 | 97.77 | 11 | 98.91 | 98.72 | 100.00 | 99.36 | 26 | 97.32 | 99.59 | 98.65 | 99.12 | 12 | 99.08 | 100.00 | 100.00 | 100.00 | 27 | 95.15 | 95.25 | 99.60 | 97.38 | 13 | 98.52 | 99.65 | 99.41 | 99.53 | 28 | 99.44 | 99.16 | 96.19 | 97.65 | 14 | 97.66 | 98.15 | 100.00 | 99.07 | 29 | 100.00 | 100.00 | 99.01 | 99.50 | 15 | 99.18 | 99.26 | 100.00 | 99.63 | 30 | 99.28 | 100.00 | 100.00 | 100.00 |
|
Table 8. OA, precision, recall rate, and F1 score for each category in AID dataset
Category | OA | Precision | Recall | F1 score | Category | OA | Precision | Recall | F1 score |
---|
1 | 98.67 | 99.47 | 100.00 | 99.73 | 24 | 91.96 | 91.57 | 88.27 | 89.89 | 2 | 97.60 | 98.75 | 98.16 | 98.45 | 25 | 97.38 | 98.16 | 100.00 | 99.07 | 3 | 94.14 | 94.56 | 94.93 | 94.74 | 26 | 97.01 | 98.93 | 96.35 | 97.62 | 4 | 98.88 | 99.15 | 91.62 | 95.24 | 27 | 96.87 | 97.42 | 94.12 | 95.74 | 5 | 98.45 | 99.43 | 96.17 | 97.77 | 28 | 90.71 | 90.91 | 84.75 | 87.72 | 6 | 97.57 | 98.16 | 97.33 | 97.74 | 29 | 95.48 | 96.88 | 100.00 | 98.42 | 7 | 100.00 | 100.00 | 99.81 | 99.90 | 30 | 96.93 | 96.14 | 92.13 | 94.09 | 8 | 86.48 | 86.92 | 88.05 | 87.48 | 31 | 93.26 | 93.68 | 93.69 | 93.68 | 9 | 98.55 | 98.19 | 100.00 | 99.09 | 32 | 96.79 | 98.76 | 93.47 | 96.04 | 10 | 98.81 | 98.45 | 99.27 | 98.86 | 33 | 96.17 | 96.81 | 93.19 | 94.97 | 11 | 89.90 | 89.98 | 96.76 | 93.25 | 34 | 97.52 | 98.06 | 94.13 | 96.05 | 12 | 93.19 | 93.12 | 92.11 | 92.61 | 35 | 97.58 | 98.03 | 97.56 | 97.79 | 13 | 91.30 | 91.22 | 98.65 | 94.79 | 36 | 100.00 | 100.00 | 100.00 | 100.00 | 14 | 97.27 | 98.15 | 98.70 | 98.42 | 37 | 96.15 | 98.56 | 97.31 | 97.93 | 15 | 96.31 | 97.07 | 90.62 | 93.73 | 38 | 99.22 | 100.00 | 99.23 | 99.61 | 16 | 99.23 | 100.00 | 100.00 | 100.00 | 39 | 95.57 | 96.27 | 97.56 | 96.91 | 17 | 85.18 | 85.50 | 97.29 | 91.01 | 40 | 93.40 | 94.90 | 98.22 | 96.53 | 18 | 99.63 | 100.00 | 100.00 | 100.00 | 41 | 95.70 | 96.52 | 100.00 | 98.23 | 19 | 93.41 | 93.19 | 93.86 | 93.52 | 42 | 86.51 | 86.44 | 87.74 | 87.09 | 20 | 92.49 | 92.26 | 92.81 | 92.53 | 43 | 93.97 | 93.20 | 98.08 | 95.58 | 21 | 93.57 | 93.04 | 97.32 | 95.13 | 44 | 98.19 | 99.65 | 97.16 | 98.39 | 22 | 91.40 | 91.29 | 95.85 | 93.51 | 45 | 97.45 | 98.47 | 89.91 | 94.00 | 23 | 96.13 | 97.50 | 95.96 | 96.72 | | | | | |
|
Table 9. OA, precision, recall rate, and F1 score for each category in NWPU-RESISC45 dataset