Skip to main content
SpringerLink
Log in

Spatio-temporal Pain Recognition in CNN-Based Super-Resolved Facial Images

  • Conference paper
  • First Online:
Video Analytics. Face and Facial Expression Recognition and Audience Measurement (VAAM 2016, FFER 2016)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10165))

Abstract

Automatic pain detection is a long expected solution to a prevalent medical problem of pain management. This is more relevant when the subject of pain is young children or patients with limited ability to communicate about their pain experience. Computer vision-based analysis of facial pain expression provides a way of efficient pain detection. When deep machine learning methods came into the scene, automatic pain detection exhibited even better performance. In this paper, we figured out three important factors to exploit in automatic pain detection: spatial information available regarding to pain in each of the facial video frames, temporal axis information regarding to pain expression pattern in a subject video sequence, and variation of face resolution. We employed a combination of convolutional neural network and recurrent neural network to setup a deep hybrid pain detection framework that is able to exploit both spatial and temporal pain information from facial video. In order to analyze the effect of different facial resolutions, we introduce a super-resolution algorithm to generate facial video frames with different resolution setups. We investigated the performance on the publicly available UNBC-McMaster Shoulder Pain database. As a contribution, the paper provides novel and important information regarding to the performance of a hybrid deep learning framework for pain detection in facial images of different resolution.

The original version of this chapter was revised: The spelling of the sixth author’s name was corrected. The erratum to this chapter is available at DOI: 10.1007/978-3-319-56687-0_14

An erratum to this chapter can be found at http://dx.doi.org/10.1007/978-3-319-56687-0_14

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Capel, D., Zisserman, A.: Super-resolution enhancement of text image sequences. In: 2000 Proceedings of the 15th International Conference on Pattern Recognition, vol. 1, pp. 600–605. IEEE (2000)

    Google Scholar 

  2. Craig, K.D., Prkachin, K.M., Grunau, R.E.: The facial expression of pain. In: Handbook of Pain Assessment. Guilford Press (2011)

    Google Scholar 

  3. Craig, K.D., Hyde, S.A., Patrick, C.J.: Genuine, suppressed and faked facial behavior during exacerbation of chronic low back pain. Pain 46(2), 161–171 (1991)

    Article  Google Scholar 

  4. Cristani, M., Cheng, D.S., Murino, V., Pannullo, D.: Distilling information with super-resolution for video surveillance. In: Proceedings of the ACM 2nd International Workshop on Video Surveillance & Sensor Networks, pp. 2–11. ACM (2004)

    Google Scholar 

  5. Debono, D.J., Hoeksema, L.J., Hobbs, R.D.: Caring for patients with chronic pain: pearls and pitfalls. J. Am. Osteopath. Assoc. 113(8), 620–627 (2013). doi:10.7556/jaoa.2013.023

    Article  Google Scholar 

  6. Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  7. Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Graphics Appl. 22(2), 56–65 (2002)

    Article  Google Scholar 

  8. Gers, F.A., Schmidhuber, J.A., Cummins, F.A.: Learning to forget: continual prediction with LSTM. Neural Comput. 12(10), 2451–2471 (2000)

    Article  Google Scholar 

  9. Gunturk, B.K., Batur, A.U., Altunbasak, Y., Hayes, M.H., Mersereau, R.M.: Eigenface-domain super-resolution for face recognition. IEEE Trans. Image Process. 12(5), 597–606 (2003)

    Article  Google Scholar 

  10. Hadjistavropoulos, T., LaChapelle, D.L., MacLeod, F.K., Snider, B., Craig, K.D.: Measuring movement-exacerbated pain in cognitively impaired frail elders. 16, 54–63 (2000)

    Google Scholar 

  11. Haque, M.A., Nasrollahi, K., Moeslund, T.B.: Real-time acquisition of high quality face sequences from an active pan-tilt-zoom camera. In: 2013 10th IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 443–448, August 2013

    Google Scholar 

  12. Haque, M.A., Nasrollahi, K., Moeslund, T.B.: Constructing facial expression log from video sequences using face quality assessment. In: 2014 International Conference on Computer Vision Theory and Applications (VISAPP), vol. 2, pp. 517–525, January 2014

    Google Scholar 

  13. Haque, M.A., Nasrollahi, K., Moeslund, T.B.: Quality-aware estimation of facial landmarks in video sequences. In: 2015 IEEE Winter Conference on Applications of Computer Vision, pp. 678–685, January 2015

    Google Scholar 

  14. Haque, M.A., Nasrollahi, K., Moeslund, T.B.: Pain expression as a biometric: why patients’ self-reported pain doesn’t match with the objectively measured pain? In: 2017 IEEE International Conference on Identity, Security and Behavior Analysis (ISBA), February 2017 (submitted)

    Google Scholar 

  15. Hennings-Yeomans, P.H., Baker, S., Kumar, B.V.: Recognition of low-resolution faces using multiple still images and multiple cameras. In: 2008 2nd IEEE International Conference on Biometrics: Theory, Applications and Systems, BTAS 2008, pp. 1–6. IEEE (2008)

    Google Scholar 

  16. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  17. Huang, T.S., Tsay, R.Y.: Multiple frame image restoration and registration. In: Advances in Computer Vision and Image Processing, pp. 317–339 (1984)

    Google Scholar 

  18. Huang, Z., Wang, R., Shan, S., Chen, X.: Face recognition on large-scale video in the wild with hybrid Euclidean-and-Riemannian metric learning. Pattern Recogn. 48(10), 3113–3124 (2015)

    Article  Google Scholar 

  19. Irani, R., Nasrollahi, K., Moeslund, T.B.: Pain recognition using spatiotemporal oriented energy of facial muscles. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 80–87, June 2015

    Google Scholar 

  20. Irani, R., Nasrollahi, K., Simon, M.O., Corneanu, C.A., Escalera, S., Bahnsen, C., Lundtoft, D.H., Moeslund, T.B., Pedersen, T.L., Klitgaard, M.L., Petrini, L.: Spatiotemporal analysis of RGB-D-T facial images for multimodal pain level recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2015

    Google Scholar 

  21. Kahou, S.E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-Lewandowski, N., Chandias Ferrari, R., Mirza, M., Warde-Farley, D., Courville, A., Vincent, P., Memisevic, R., Pal, C., Bengio, Y.: Emonets: multimodal deep learning approaches for emotion recognition in video. J. Multimodal User Interfaces 10(2), 99–111 (2016)

    Article  Google Scholar 

  22. Kennedy, J.A., Israel, O., Frenkel, A., Bar-Shalom, R., Azhari, H.: Super-resolution in pet imaging. IEEE Trans. Med. Imaging 25(2), 137–147 (2006)

    Article  Google Scholar 

  23. Khorrami, P., Paine, T.L., Brady, K., Dagli, C., Huang, T.S.: How deep neural networks can improve emotion recognition on video data. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 619–623, September 2016

    Google Scholar 

  24. Kim, B.K., Roh, J., Dong, S.Y., Lee, S.Y.: Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J. Multimodal User Interfaces 10(2), 173–189 (2016)

    Article  Google Scholar 

  25. Kim, K.I., Kim, D., Kim, J.H.: Example-based learning for image super-resolution (2004)

    Google Scholar 

  26. Kim, K.I., Kwon, Y.: Example-based learning for single-image super-resolution. In: Rigoll, G. (ed.) DAGM 2008. LNCS, vol. 5096, pp. 456–465. Springer, Heidelberg (2008). doi:10.1007/978-3-540-69321-5_46

    Chapter  Google Scholar 

  27. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  28. Kunz, M., Gruber, A., Lautenbacher, S.: Sex differences in facial encoding of pain. J. Pain 7(12), 915–928 (2006)

    Article  Google Scholar 

  29. Kunz, M., Mylius, V., Schepelmann, K., Lautenbacher, S.: On the relationship between self-report and facial expression of pain. J. Pain 5(7), 368–376 (2004)

    Article  Google Scholar 

  30. Kunz, M., Prkachin, K., Lautenbacher, S.: Smiling in pain: explorations of its social motives. Pain Res. Treat. 2013, e128093 (2013)

    Google Scholar 

  31. Kunz, M., Scharmann, S., Hemmeter, U., Schepelmann, K., Lautenbacher, S.: The facial expression of pain in patients with dementia. Pain 133(1–3), 221–228 (2007)

    Article  Google Scholar 

  32. Lautenbacher, S., Niewelt, B.G., Kunz, M.: Decoding pain from the facial display of patients with dementia: a comparison of professional and nonprofessional observers. Pain Med. 14(4), 469–477 (2013). http://painmedicine.oxfordjournals.org/content/14/4/469

    Article  Google Scholar 

  33. Léonard, N., Waghmare, S., Wang, Y.: Rnn: Recurrent library for torch. arXiv preprint arXiv:1511.07889 (2015)

  34. Li, F., Jia, X., Fraser, D.: Universal HMT based super resolution for remote sensing images. In: 2008 15th IEEE International Conference on Image Processing, ICIP 2008, pp. 333–336. IEEE (2008)

    Google Scholar 

  35. Li, H., Hua, G.: Hierarchical-PEP model for real-world face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, pp. 4055–4064 (2015)

    Google Scholar 

  36. Lin, F.C., Fookes, C.B., Chandran, V., Sridharan, S.: Investigation into optical flow super-resolution for surveillance applications (2005)

    Google Scholar 

  37. Lucey, P., Cohn, J.F., Matthews, I., Lucey, S., Sridharan, S., Howlett, J., Prkachin, K.M.: Automatically detecting pain in video through facial action units. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 41(3), 664–674 (2011)

    Article  Google Scholar 

  38. Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., Matthews, I.: Painful data: the UNBC-McMaster shoulder pain expression archive database. In: 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG 2011), pp. 57–64, March 2011

    Google Scholar 

  39. Maintz, J.A., Viergever, M.A.: A survey of medical image registration. Med. Image Anal. 2(1), 1–36 (1998)

    Article  Google Scholar 

  40. Malczewski, K., Stasinski, R.: Toeplitz-based iterative image fusion scheme for MRI. In: 2008 15th IEEE International Conference on Image Processing, ICIP 2008, pp. 341–344. IEEE (2008)

    Google Scholar 

  41. Milanfar, P.: Super-Resolution Imaging. CRC Press, Boca Raton (2010)

    Google Scholar 

  42. Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25(6), 1423–1468 (2014)

    Article  Google Scholar 

  43. Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: British Machine Vision Conference, vol. 1, p. 6 (2015)

    Google Scholar 

  44. Peled, S., Yeshurun, Y.: Superresolution in MRI: application to human white matter fiber tract visualization by diffusion tensor imaging. Magn. Reson. Med. 45(1), 29–35 (2001)

    Article  Google Scholar 

  45. Prkachin, K.M.: The consistency of facial expressions of pain: a comparison across modalities. 51, 297–306 (1992)

    Google Scholar 

  46. Prkachin, K.M., Berzins, S., Mercer, S.R.: Encoding and decoding of pain expressions: a judgement study. Pain 58(2), 253–259 (1994)

    Article  Google Scholar 

  47. Prkachin, K.M., Solomon, P.E.: The structure, reliability and validity of pain expression: Evidence from patients with shoulder pain. 139, 267–274 (2008)

    Google Scholar 

  48. Prkachin, K., Schultz, I., Berkowitz, J., Hughes, E., Hunt, D.: Assessing pain behaviour of low-back pain patients in real time: concurrent validity and examiner sensitivity. Behav. Res. Ther. 40(5), 595–607 (2002)

    Article  Google Scholar 

  49. Ranganathan, H., Chakraborty, S., Panchanathan, S.: Multimodal Emotion Recognition Using Deep Learning Architectures. Institute of Electrical and Electronics Engineers Inc., United States (2016)

    Book  Google Scholar 

  50. Sezer, O.G., Altunbasak, Y., Ercil, A.: Face recognition with independent component-based super-resolution. In: Electronic Imaging 2006, pp. 607705–607705. International Society for Optics and Photonics (2006)

    Google Scholar 

  51. Sikdar, A., Behera, S.K., Dogra, D.P.: Computer vision guided human pulse rate estimation a review. IEEE Rev. Biomed. Eng. PP(99), 1 (2016)

    Google Scholar 

  52. Sikka, K., Ahmed, A.A., Diaz, D., Goodwin, M.S., Craig, K.D., Bartlett, M.S., Huang, J.S.: Automated assessment of children’s postoperative pain using computer vision. Pediatrics 136(1), 124–131 (2015)

    Article  Google Scholar 

  53. Vallerand, A.H., Polomano, R.C.: The relationship of gender to pain. Pain Manage. Nurs. 1(3, Supplement 1), 8–15 (2000)

    Article  Google Scholar 

  54. Yang, J., Huang, T.: Image super-resolution: historical overview and future challenges. In: Super-Resolution Imaging, pp. 20–34 (2010)

    Google Scholar 

  55. Yang, J., Ren, P., Chen, D., Wen, F., Li, H., Hua, G.: Neural aggregation network for video face recognition. arXiv preprint arXiv:1603.05474 (2016)

  56. Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI 2015, pp. 435–442. ACM, New York (2015)

    Google Scholar 

  57. Zhou, J., Hong, X., Su, F., Zhao, G.: Recurrent convolutional neural network regression for continuous pain intensity estimation in video. arXiv preprint arXiv:1605.00894 (2016)

Download references

Author information

Authors and Affiliations

  1. Computer Vision Center (UAB), University of Barcelona, Barcelona, Spain

    Marco Bellantonio, Pau Rodriguez, Sergio Escalera & Jordi Gonzalez

  2. Visual Analysis of People (VAP) Laboratory, Aalborg University, Aalborg, Denmark

    Mohammad A. Haque, Kamal Nasrollahi & Thomas B. Moeslund

  3. iCV Research Group, Institute of Technology, University of Tartu, Tartu, Estonia

    Taisi Telve, Pejman Rasti & Gholamreza Anbarjafari

Authors
  1. Marco Bellantonio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad A. Haque
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pau Rodriguez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kamal Nasrollahi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Taisi Telve
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sergio Escalera
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jordi Gonzalez
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Thomas B. Moeslund
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Pejman Rasti
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Gholamreza Anbarjafari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marco Bellantonio .

Editor information

Editors and Affiliations

  1. Aalborg University, Aalborg, Denmark

    Kamal Nasrollahi

  2. Institute of Applied Sciences and Intelligent Systems, Lecce, Italy

    Cosimo Distante

  3. Stevens Institute of Technology, Hoboken, New Jersey, USA

    Gang Hua

  4. Queen Mary University of London, London, United Kingdom

    Andrea Cavallaro

  5. Aalborg University, Aalborg, Denmark

    Thomas B. Moeslund

  6. Università di Catania, Catania, Italy

    Sebastiano Battiato

  7. Rensselaer Polytechnic Institute, Troy, New York, USA

    Qiang Ji

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Bellantonio, M. et al. (2017). Spatio-temporal Pain Recognition in CNN-Based Super-Resolved Facial Images. In: Nasrollahi, K., et al. Video Analytics. Face and Facial Expression Recognition and Audience Measurement. VAAM FFER 2016 2016. Lecture Notes in Computer Science(), vol 10165. Springer, Cham. https://doi.org/10.1007/978-3-319-56687-0_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-56687-0_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-56686-3

  • Online ISBN: 978-3-319-56687-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation