• Journal of Electronic Science and Technology
  • Vol. 22, Issue 3, 100260 (2024)
Jiang Wu1,2, Yi Shi2, Shun Yan3, and Hong-Mei Yan1,2,*
Author Affiliations
  • 1Yangtze Delta Region Institute (Huzhou), University of Electronic Science and Technology of China, Huzhou, 313001, China
  • 2MOE Key Laboratory for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 611731, China
  • 3College of Engineering, University of California at Santa Barbara, Santa Barbara, 93106, USA
  • show less
    DOI: 10.1016/j.jnlest.2024.100260 Cite this Article
    Jiang Wu, Yi Shi, Shun Yan, Hong-Mei Yan. Global-local combined features to detect pain intensity from facial expression images with attention mechanism[J]. Journal of Electronic Science and Technology, 2024, 22(3): 100260 Copy Citation Text show less
    References

    [1] Lynch M.. Pain as the fifth vital sign. Journal of Intravenous Nursing, 24, 85-94(2001).

    [2] D. LopezMartinez, K. Peng, S.C. Steele, A.J. Lee, D. Bsook, R. Picard, Multitask multiple kernel machines f personalized pain recognition from functional nearinfrared spectroscopy brain signals, in: Proc. of the 24th Intl. Conf. on Pattern Recognition (ICPR), Beijing, China, 2018, pp. 2320–2325.

    [3] Wager T.D., Atlas L.Y., Lindquist M.A., Roy M., Woo C.W., Kross E.. An fMRI-based neurologic signature of physical pain. New Engl. J. Med., 368, 1388-1397(2013).

    [4] Zwakhalen S.M.G., Van Dongen K.A.J., Hamers J.P.H., Abu-Saad H.H.. Pain assessment in intellectually disabled people: Non-verbal indicators. J. Adv. Nurs., 45, 236-245(2004).

    [5] Kunz M., Scharmann S., Hemmeter U., Schepelmann K., Lautenbacher S.. The facial expression of pain in patients with dementia. PAIN®, 133, 221-228(2007).

    [6] Lesage F.X., Berjot S., Deschamps F.. Clinical stress assessment using a visual analogue scale. Occup. Med.-Oxford, 62, 600-605(2012).

    [7] Liu H., Zhang C., Deng Y.-J., Liu T.-T., Zhang Z.-L., Li Y.-F.. Orientation cues-aware facial relationship representation for head pose estimation via transformer. IEEE T. Image Process., 32, 6289-6302(2023).

    [8] H. Liu, T.T. Liu, Y. Chen, Z.L. Zhang, Y.F. Li, EHPE: Skeleton cuesbased Gaussian codinate encoding f efficient human pose estimation, IEEE T. Multimedia (Aug. 2022) 112. DOI: 10.1109TMM.2022.3197364.

    [9] Liu H., Liu T.-T., Zhang Z.-L., Sangaiah A.K., Yang B., Li Y.-F.. ARHPE: Asymmetric relation-aware representation learning for head pose estimation in industrial human-computer interaction. IEEE T. Ind. Inform., 18, 7107-7117(2022).

    [10] Bargshady G., Zhou X.-J., Deo R.C., Soar J., Whittaker F., Wang H.. Ensemble neural network approach detecting pain intensity from facial expressions. Artif. Intell. Med., 109, 101954:1-12(2020).

    [11] N. Ahmed, T. Natarajan, K.R. Rao, Discrete cosine transfm, IEEE T. Comput. C23 (1) (Jan. 1974) 90–93.

    [12] T. Ahonen, A. Hadid, M. Pietikäinen, Face recognition with local binary patterns, in: Proc. of the 8th European Conf. on Computer Vision, Prague, Czech Republic, 2004, pp. 469–481.

    [13] N. Dalal, B. Triggs, Histograms of iented gradients f human detection, in: Proc. of the IEEE Computer Society Conf. on Computer Vision Pattern Recognition, San Diego, USA, 2005, pp. 886–893.

    [14] LeCun Y., Bengio Y., Hinton G.. Deep learning. Nature, 521, 436-444(2015).

    [15] Kamilaris A., Prenafeta-Boldú F.X.. Deep learning in agriculture: A survey. Comput. Electron. Agr., 147, 70-90(2018).

    [16] Xin X.-W., Lin X.-Y., Yang S.-F., Zheng X.. Pain intensity estimation based on a spatial transformation and attention CNN. PLoS One, 15, e0232412:1-15(2020).

    [17] A. Graves, Long shtterm memy, in: A. Graves (Ed.), Supervised Sequence Labelling with Recurrent Neural wks, Springer, Berlin, Heidelberg, Germany, 2012, pp. 37–45.

    [18] Z. Hammal, J.F. Cohn, Automatic detection of pain intensity, in: Proc. of the 14th ACM Intl. Conf. on Multimodal Interaction, Santa Monica, USA, 2012, pp. 47–52.

    [19] J. Zhou, X.P. Hong, F. Su, G.Y. Zhao, Recurrent convolutional neural wk regression f continuous pain intensity estimation in video, in: Proc. of the IEEE Conf. on Computer Vision Pattern Recognition Wkshops, Las Vegas, USA, 2016, pp. 1535–1543.

    [20] E. Friesen, P. Ekman, Facial action coding system: A technique f the measurement of facial movement, Palo Alto 3(2) (1978) 5.

    [21] Prkachin K.M., Solomon P.E.. The structure. reliability and validity of pain expression: Evidence from patients with shoulder pain, Pain, 139, 267-274(2008).

    [22] P. Lucey, J.F. Cohn, K.M. Prkachin, P.E. Solomon, I. Matthews, Painful data: The UNBCMcMaster shoulder pain expression archive database, in: Proc. of the IEEE Intl. Conf. on Automatic Face & Gesture Recognition (FG), Santa Barbara, USA, 2011, pp. 57–64.

    [23] R.J. Yang, X.P. Hong, J.Y. Peng, X.Y. Feng, G.Y. Zhao, Incpating highlevel lowlevel cues f pain intensity estimation, in: Proc. of the 24th Intl. Conf. on Pattern Recognition (ICPR), Beijing, China, 2018, pp. 3495–3500.

    [24] A.B. Ashraf, S. Lucey, J.F. Cohn, et al., The painful face: Pain expression recognition using active appearance models, in: Proc. of the 9th Intl. Conf. on Multimodal Interfaces, Nagoya, Japan, 2007, pp. 9–14.

    [25] S. Kaltwang, O. Rudovic, M. Pantic, Continuous pain intensity estimation from facial expressions, in: Proc. of the 8th Intl. Symposium on Visual Computing, Rethymnon, Crete, Greece, 2012, pp. 368–377.

    [26] Rodriguez P., Cucurull G., Gonzàlez J. et al. Deep pain: Exploiting long short-term memory networks for facial expression classification. IEEE T. Cybernetics, 52, 3314-3324(2022).

    [27] Zakaria N., M. M. Hassim Y.. A review study of the visual geometry group approaches for image classification. J. of Appl. Sci., Tech. and Comput., 1, 14-28(2024).

    [28] Tavakolian M., Hadid A.. A spatiotemporal convolutional neural network for automatic pain intensity estimation from facial dynamics. Int. J. Comput. Vision, 127, 1413-1425(2019).

    [29] Itti L., Koch C.. Computational modelling of visual attention. Nat. Rev. Neurosci., 2, 194-203(2001).

    [30] Liu H., Zheng C., Li D.-T.-C. et al. EDMF: Efficient deep matrix factorization with review feature learning for industrial recommender system. IEEE T. Ind. Inform., 18, 4361-4371(2022).

    [31] Liu H., Zheng C., Li D.-T.-C. et al. Multi-perspective social recommendation method with graph representation learning. Neurocomputing, 468, 469-481(2022).

    [32] H. Liu, C. Zhang, Y.J. Deng, B.C. Xie, T.T. Liu, Y.F. Li, TransIFC: Invariant cuesaware feature concentration learning f efficient finegrained bird image classification, IEEE T. Multimedia (2023). doi: 10.1109TMM.2023.3238548.

    [33] Liu H., Fang S., Zhang Z.-L., Li D.-T.-CH., Lin K., Wang J.-ZH., Collaborative poses perception and matrix fisher distribution for head pose estimation MFDNet:. IEEE T. on Multimedia. 2021,, 24, 2449-2460(2021).

    [34] Li Y., Zeng J.-B., Shan S.-G., Chen X.-L.. Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE T. Image Process., 28, 2439-2450(2019).

    [35] H. Zhang , W. Su , Z. Wang, Weakly supervised localglobal attention wk f facial expression recognition, IEEE Access 8 (Feb. 2020) 3797637987.

    [36] C. Liu, K. Hirota , J. Ma, et al., Facial expression recognition using hybrid features of pixel geometry, IEEE Access 9 (Jan. 2021) 1887618889.

    [37] Liu X., Li J., Duan T.-T., Li J.-T., Wang Y.. DSMA: Reference-based image super-resolution method based on dual-view supervised learning and multi-attention mechanism. IEEE Access, 10, 54649-54659(2022).

    [38] M. Jaderberg, K. Simonyan, A. Zisserman, K. Kavukcuoglu, Spatial transfmer wks, in: Proc. of the 28th Intl. Conf. on Neural Infmation Processing Systems, Montreal, Canada, 2015, pp. 2017–2025.

    [39] Zhang K.-P., Zhang Z.-P., Li Z.-F., Qian Y.. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Proc. Let., 23, 1499-1503(2016).

    [40] Benarroch E.E.. Pain-autonomic interactions: A selective review. Clin. Auton. Res., 11, 343-349(2001).

    [41] Bargshady G., Zhou X.-J., Deo R.C., Soar J., Whittaker F., Wang H.. Enhanced deep learning algorithm development to detect pain intensity from facial expression images. Expert Syst. Appl., 149, 113305:1-10(2020).

    [42] S. Walter, S. Gruss, H. Ehleiter, et al., The biovid heat pain database data f the advancement systematic validation of an automated pain recognition system, in: Proc. of the IEEE Intl. Conf. on Cyberics (CYBCO), Lausanne, Switzerl, 2013, pp. 128–131.

    [43] R.J. Yang, S.J. Tong, M. Bdallo, et al., On pain assessment from facial videos using spatiotempal local des, in: Proc. of the Sixth Intl. Conf. on Image Processing They, Tools Applications (IPTA), Oulu, Finl, 2016, pp. 1–6.

    [44] Huang D., Xia Z.-Q., Li L., Wang K.-W., Feng X.-Y.. Pain-awareness multistream convolutional neural network for pain estimation. J. Electron. Imaging, 28, 043008:1-10(2019).

    [45] L. Pham, T.H. Vu, T.A. Tran, Facial expression recognition using residual masking wk, in: Proc. of the 25th Intl. Conf. on Pattern Recognition (ICPR), Milan, Italy, 2021, pp. 4513–4519.

    [46] Zhao Z.-Q., Liu Q.-S., Wang S.-M.. Learning deep global multi-scale and local attention features for facial expression recognition in the wild. IEEE T. Image Process., 30, 6544-6556(2021).

    [47] Z. Liu, J. Ning, Y. Cao, et al., Video swin transfmer, in: Proc. of the IEEECVF Conf. on Computer Vision Pattern Recognition, New leans, USA, 2022, pp. 3192–3201.

    [48] J.W. Mao, R. Xu, X.S. Yin, Y.Q. Chang, B.L. Nie, A.B. Huang, POSTER++: A simpler stronger facial expression recognition wk [Online]. Available, https:arxiv.gabs2301.12149, February 2023.

    [49] R.R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, GradCAM: Visual explanations from deep wks via gradientbased localization, in: Proc. of the IEEE Intl. Conf. on Computer Vision, Venice, Italy, 2017, pp. 618–626.

    Jiang Wu, Yi Shi, Shun Yan, Hong-Mei Yan. Global-local combined features to detect pain intensity from facial expression images with attention mechanism[J]. Journal of Electronic Science and Technology, 2024, 22(3): 100260
    Download Citation