Facial Expression Recognition by Merging Multilayer Features of Lightweight Convolutional Networks

Shen Hao; Meng Qinghao; Liu Yinbo

doi:10.3788/LOP202158.0610005

Journals >Laser & Optoelectronics Progress >Volume 58 >Issue 6 >Page 610005 > Article

Laser & Optoelectronics Progress
Vol. 58, Issue 6, 610005 (2021)

Facial Expression Recognition by Merging Multilayer Features of Lightweight Convolutional Networks

Shen Hao^1、2、3, Meng Qinghao^1、2、3, and Liu Yinbo^{1、2、3、*}

Author Affiliations

¹School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China

²Institute of Robotics and Autonomous Systems, Tianjin University, Tianjin 300072, China

³Tianjin Key Laboratory of Process Detection and Control, Tianjin 300072, China

show less

DOI: 10.3788/LOP202158.0610005 Cite this Article Set citation alerts

Shen Hao, Meng Qinghao, Liu Yinbo. Facial Expression Recognition by Merging Multilayer Features of Lightweight Convolutional Networks[J]. Laser & Optoelectronics Progress, 2021, 58(6): 610005 Copy Citation Text

show less

Fig. 1. Structure of ms_model_v1 model

Download full size

Fig. 2. Bottleneck_M structure

Download full size

Fig. 3. Process of feature selection module

Download full size

Fig. 4. Confusion matrix of RAF-DB dataset using the proposed method

Download full size

Fig. 5. Confusion matrix of AffectNet dataset using the proposed method

Download full size

Layer name	c	s	t	Output size
Conv2D 1 (3×3)	16	2		56×56×16
Bottleneck_M1	16	1	1	56×56×16
Bottleneck_M2	24	2	5	28×28×24
Bottleneck_M3	24	1	5	28×28×24
Bottleneck_M3_1	32	1	5	28×28×32
Bottleneck_M3_2	32	1	5	28×28×32
Feature selection module 1				32
Bottleneck_M4	32	2	5	14×14×32
Bottleneck_M5	32	1	5	14×14×32
Feature selection module 2				32
Bottleneck_M6	40	1	5	14×14×40
Bottleneck_M7	40	1	5	14×14×40
Feature selection module 3				40
Bottleneck_M8	40	1	5	14×14×40
Bottleneck_M9	48	2	5	7×7×48
Bottleneck_M10	64	1	5	7×7×64
Conv2D 2 (1×1)	64	1		7×7×64
Global average pooling				64
Concat				168
Reshape 1				1×1×168
Dropout				1×1×168
Conv2D 3 (1×1)	k			1×1×k
Softmax				1×1×k
Reshape 2				k

Table 1. Parameters of the CNN

Class	Neutral	Happy	Sad	Surprise	Fear	Disgust	Angry	Contempt	Total
Train set	5978	5979	5966	5963	6378	3803	5979	3750	43796
Test set	500	500	500	500	500	500	500	500	4000

Table 2. Number of categories in AffectNet dataset after random screening

Model name	Number of parameters	Model size/Mbit	Acc in RAF-DB/%	Acc in AffectNet/%
base_model_R	195159	2.9	83.64	56.93
base_model_M	195159	2.9	84.23	57.30
ms_model_fully	3299303	40.2	85.45	57.37
ms_model_v1_R	193927	3.0	84.81	57.43
ms_model_v1_M	193927	3.0	85.49	57.70

Table 3. Performance of different models

Method	Acc in RAF-DB/%	Acc in AffectNet/%
Boosting-POOF^[24]	73.19
VGG16^[25]	80.96	51.11
DLP-CNN^[22]	84.13
pACNN^[26]	83.05	55.33
gACNN^[26]	85.07	58.78
E2-CapsNet^[27]	85.24
ms_model_v1_M	85.49	57.70

Table 4. Accuracy of different methods in RAF-DB and AffectNet datasets

Shen Hao, Meng Qinghao, Liu Yinbo. Facial Expression Recognition by Merging Multilayer Features of Lightweight Convolutional Networks[J]. Laser & Optoelectronics Progress, 2021, 58(6): 610005

Download Citation

Set citation alerts for the article

Tools

Set citation alerts for the article

Save the article for my favorites

Paper Information