Multi-Angle Facial Expression Recognition Algorithm Combined with Dual-Channel WGAN-GP

Yuan Deng; Yiping Shi; Jie Liu; Yueying Jiang; Yamei Zhu; Jin Liu

doi:10.3788/LOP202259.1810013

Journals >Laser & Optoelectronics Progress >Volume 59 >Issue 18 >Page 1810013 > Article

Laser & Optoelectronics Progress
Vol. 59, Issue 18, 1810013 (2022)

Multi-Angle Facial Expression Recognition Algorithm Combined with Dual-Channel WGAN-GP

Yuan Deng, Yiping Shi^*, Jie Liu, Yueying Jiang, Yamei Zhu, and Jin Liu

Author Affiliations

School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201620, China

show less

DOI: 10.3788/LOP202259.1810013 Cite this Article Set citation alerts

Yuan Deng, Yiping Shi, Jie Liu, Yueying Jiang, Yamei Zhu, Jin Liu. Multi-Angle Facial Expression Recognition Algorithm Combined with Dual-Channel WGAN-GP[J]. Laser & Optoelectronics Progress, 2022, 59(18): 1810013 Copy Citation Text

show less

Fig. 1. Overall structure of proposed model

Download full size

Fig. 2. Dual-channel generator network structure

Download full size

Fig. 3. Discriminator network structure

Download full size

Fig. 4. Generation effect of frontal expression images on KDEF dataset

Download full size

Fig. 5. Generation effect of frontal expression images on Multi-PIE dataset

Download full size

Fig. 6. Confusion matrix of expression recognition under different angles on KDEF dataset. (a) 0° rotation; （b） 45° rotation；（c） 90° rotation

Download full size

Fig. 7. Partial error samples on KDEF dataset

Download full size

Fig. 8. Confusion matrix of expression recognition under different angles on Multi-PIE dataset. (a) 0° rotation; (b) 30° rotation; (c) 60° rotation; (d) 90° rotation

Download full size

Fig. 9. Partial error samples on Multi-PIE dataset

Download full size

Fig. 10. Comparison of network loss function changes. (a) TP-GAN discriminant loss; (b) TP-GAN adversarial loss; (c) proposed model discriminant loss; (d) proposed model adversarial loss

Download full size

Fig. 11. Comparison of facial expression images generation effect after model ablation. (a) Profile; (b) without local path; (c) without Wasserstein; (d) without GP; (e) proposed method; (f) real frontal

Download full size

Fig. 12. Comparison of facial expression recognition rate under different ablation results

Download full size

Fig. 13. Comparison of frontalization generation effect. (a) Real profile; (b) proposed method; (c) TP-GAN^[11]; (d) CAPG-GAN^[9]; (e) FI-GAN^[23]; (f) MTDNN^[24]; (g) 3DMM^[25]; (h) real frontal

Download full size

Input	Operator	Channel	SE	Stride	Repeat
128²×3	Conv 3×3	16		2
64²×16	Bottleneck 3×3	16	√	2
32²×16	Bottleneck 3×3	24		2
16²×24	Bottleneck 5×5	40	√	2
8²×40	Bottleneck 5×5	40	√	1	×2
8²×40	Bottleneck 5×5	48	√	1	×2
8²×48	Bottleneck 5×5	96	√	2
4²×96	Bottleneck 3×3	96	√	1	×2
4²×96	Conv 1×1	256	√	1
4²×256	Pooling 4×4			1
1²×256	Conv 1×1	512		1
1²×512	Conv 1×1	7		1

Table 1. Structure of improved MobileNetV3

Angle /°	-90	-45	0	45	90
Accuracy /%	83.88	87.76	93.27	88.37	82.65

Table 2. Multi-angle facial expression recognition rate on KDEF dataset

Angle /°	-90	-60	-30	0	30	60	90
Accuracy /%	81.83	86.00	88.67	92.17	89.17	85.50	81.33

Table 3. Multi-angle facial expression recognition rate on Multi-PIE dataset

Method	30°	60°	90°
Without L_sym （k₃=0）	87.00	81.83	76.17
Without L_ip （k₄=0）	82.67	73.33	62.50
Proposed method	89.17	85.50	81.33

Table 4. Expression recognition rate under different loss functions on Multi-PIE dataset

Method	Accuracy /%	Parameter computation /MB
VGG-19^［20］	82.15	70.45
ResNet-50^［21］	84.94	23.60
Xception^［22］	90.55	20.88
MobileNetV1^［14］	90.24	3.24
Proposed method	92.17	0.95

Table 5. Recognition results of frontal expressions under different models

Method	-90˚	-45˚	0˚	45˚	90˚
Reference［5］	74.50	72.00	70.50	77.40	79.50
Reference［6］		80.39	84.07	77.03
Reference［7］			86.67	83.81	76.67
Proposed method	83.88	87.76	93.27	88.37	82.65

Table 6. Comparison of multi-angle facial expression recognition rate with existing methods on KDEF dataset

Method	-90˚	-60˚	-30˚	0˚	30˚	60˚	90˚
Reference［26］			93.10	92.80	92.20
Reference［27］			92.58	89.65	94.51
Reference［28］	79.00	83.00	80.00	86.00	83.50	82.00	76.00
Reference［29］				88.20	87.30	83.80	78.80
Reference［30］				83.30	85.80	81.60	92.50
Proposed method	81.83	86.00	88.67	92.17	89.17	85.50	81.33

Table 7. Comparison of multi-angle facial expression recognition rate with existing methods on Multi-PIE dataset

Download Citation

Set citation alerts for the article

Tools

Set citation alerts for the article

Save the article for my favorites

Paper Information