Multi-Feature Fusion Real-Time Action Recognition Based on 2D to 3D Skeleton

Guoyin Ren; Xiaoqi Lü; Yuhao Li

doi:10.3788/LOP202158.2410010

Journals >Laser & Optoelectronics Progress >Volume 58 >Issue 24 >Page 2410010 > Article

Laser & Optoelectronics Progress
Vol. 58, Issue 24, 2410010 (2021)

Multi-Feature Fusion Real-Time Action Recognition Based on 2D to 3D Skeleton

Guoyin Ren^1、2, Xiaoqi Lü^{1、2、3、*}, and Yuhao Li²

Author Affiliations

¹School of Mechanical Engineering, Inner Mongolia University of Science & Technology, Baotou, Inner Mongolia 0 14010, China;

²School of Information Engineering, Inner Mongolia University of Science & Technology, Baotou, Inner Mongolia 0 14010, China;

³Inner Mongolia University of Technology, Huhhot, Inner Mongolia 0 10051, China

show less

DOI: 10.3788/LOP202158.2410010 Cite this Article Set citation alerts

Guoyin Ren, Xiaoqi Lü, Yuhao Li. Multi-Feature Fusion Real-Time Action Recognition Based on 2D to 3D Skeleton[J]. Laser & Optoelectronics Progress, 2021, 58(24): 2410010 Copy Citation Text

show less

Fig. 1. Key points and numbers of the 2D human skeleton collected by OpenPose

Download full size

Fig. 2. Network structure for training 3D skeleton estimator

Download full size

Fig. 3. 3D skeleton action recognition network with difficult input samples

Download full size

Fig. 4. Local real 3D model and prediction model. (a) 2D skeleton collected by OpenPose; (b) local real 3D skeleton; (c) estimated 3D skeleton

Download full size

Method	Direct	Disc	Eat	Greet	Phone	Photo	Pose	Purch.	Sit	SitD	Smoke	Wait	Avgrage
Ref.[12]	52.8	54.2	54.3	61.8	53.1	53.6	71.7	86.7	61.5	53.4	67.2	54.8	60.4
Ref.[11]	49.2	51.6	47.6	50.5	51.8	48.5	51.7	61.5	70.9	53.7	60.3	48.9	53.9
Ref.[9]	37.7	44.4	40.3	42.1	48.2	54.9	44.4	42.1	54.6	58.0	45.1	46.4	47.3
Ref.[16]	43.2	38.1	40.8	44.4	51.8	43.7	38.4	50.8	52.0	42.1	42.2	44.0	44.3
Ours	44.6	39.5	39.7	41.2	51.1	42.9	40.8	42.9	50.6	40.8	44.6	42.9	43.4

Table 1. Action estimation errors of different methods on the Human3.6M data set unit: mm

Expansion ratio	Without treatment	20%	40%	60%	80%	100%
Accuracy /%	82.2	85.1	85.9	86.5	87.9	88.2

Table 2. Accuracy of the network after enhancing the data set

Method	Year	CS	CV
RA^[28]	2018	85.9	93.5
AS^[29]	2019	86.8	94.2
2s-AGCN^[30]	2019	88.5	95.1
Two-stream TL-GCN^[31]	2020	89.2	95.4
2s-SGCN^[32]	2021	90.1	96.2
Ours	2021	88.2	95.6

Table 3. Accuracy of different methods on the NTU-RGB+D 60 verification data set unit: %

CS	After preprocessing of motion estimation network	Using NTU RGB + D 60
With 2D feature fusion	88.2	86.2
Without 2D feature fusion	82.5	81.1

Table 4. Effect of motion estimation and multi feature fusion on recognition accuracy on NTU RGB + D 60 data set unit: %

Guoyin Ren, Xiaoqi Lü, Yuhao Li. Multi-Feature Fusion Real-Time Action Recognition Based on 2D to 3D Skeleton[J]. Laser & Optoelectronics Progress, 2021, 58(24): 2410010

Download Citation

Set citation alerts for the article

Tools

Set citation alerts for the article

Save the article for my favorites

Paper Information