Global-local combined features to detect pain intensity from facial expression images with attention mechanism

Jiang Wu; Yi Shi; Shun Yan; Hong-Mei Yan

doi:10.1016/j.jnlest.2024.100260

Journals >Journal of Electronic Science and Technology >Volume 22 >Issue 3 >Page 100260 > Article

Journal of Electronic Science and Technology
Vol. 22, Issue 3, 100260 (2024)

Global-local combined features to detect pain intensity from facial expression images with attention mechanism

Jiang Wu^1,2, Yi Shi², Shun Yan³, and Hong-Mei Yan^1,2,*

Author Affiliations

¹Yangtze Delta Region Institute (Huzhou), University of Electronic Science and Technology of China, Huzhou, 313001, China

²MOE Key Laboratory for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 611731, China

³College of Engineering, University of California at Santa Barbara, Santa Barbara, 93106, USA

show less

DOI: 10.1016/j.jnlest.2024.100260 Cite this Article

Jiang Wu, Yi Shi, Shun Yan, Hong-Mei Yan. Global-local combined features to detect pain intensity from facial expression images with attention mechanism[J]. Journal of Electronic Science and Technology, 2024, 22(3): 100260 Copy Citation Text

EndNote(RIS)

BibTex

Plain Text

show less

Fig. 1. Overall pre-processing pipeline: (a) original image, (b) facial localization, (c) facial alignment, and (d) facial cropping.

Download full size | View in the Article

Fig. 2. Framework of the proposed GLA-CNN.

Download full size | View in the Article

Fig. 3. Sample frames and their corresponding PSPI scores at the UNBC-McMaster Shoulder Pain database.

Download full size | View in the Article

Fig. 4. Number of pictures per PSPI code class at the UNBC-McMaster Shoulder Pain database.

Download full size | View in the Article

Fig. 5. Confusion matrix based on the proposed GLA-CNN model.

Download full size | View in the Article

Fig. 6. Samples of attention map from “No Pain” to “Strong Pain” by different models at the UNBC-McMaster Shoulder Pain database.

Download full size | View in the Article

PSPI score	Pain level	Number of images
0	No pain	2928
1	Weak pain	2909
2	Mild pain	2351
≥3	Strong pain	3102

Table 1. Number of image for four pain levels at the UNBC-McMaster Shoulder Pain database² according to the PSPI score.

View in the Article

Model	Accuracy	F1-score	Recall	Precision
Baseline(VGG-16)	51.72%	36.65%	34.55%	44.36%
LANet	55.53%	35.38%	33.78%	42.92%
GANet	55.51%	37.22%	33.91%	43.03%
GLA-CNN	56.45%	36.52%	34.08%	43.23%
GLA-CNN-BiLSTM	54.54%	33.15%	31.07%	42.02%

Table 2. Comparison of the average performance of proposed GLA-CNN with VGG-16, separated local and global modules under 5-fold cross validation.

View in the Article

Ref.	Method	Level	Accuracy
[42]	Head analysis	4	28.10%
[43]	LBP	4	34.70%
[43]	LPQ	4	23.40%
[43]	BSIF	4	23.70%
[44]	CNN	4	35.30%
[16]	CNN	4	51.10%
[45]	MRN	4	46.18%
[46]	MA-NET	4	55.16%
[47]	Swin transformer	4	54.40%
[48]	POSTER V2	4	52.24%
This work	GLA-CNN	4	56.45%

Table 3. Comparison of the proposed model with the state-of-the-art at the UNBC-McMaster Shoulder Pain database.

View in the Article

Model	GA	LA	Accuracy	F1-score	Recall	Precision
VGG-16	×	×	51.72%	36.65%	34.55%	44.36%
GNet	×	×	53.42%	33.06%	30.18%	41.32%
GANet	√	×	55.51%	37.22%	33.91%	43.03%
LNet	×	×	53.28%	35.76%	33.07%	43.77%
LANet	×	√	55.53%	35.38%	33.78%	42.92%
GLA-CNN	√	√	56.45%	36.52%	34.08%	43.23%

Table 4. Comparison of different network performance with/without attentional mechanisms (GA and LA stand for global attention and local attention, respectively).