Skip to main content
SpringerLink
Log in

Local Binary Patterns Based Facial Expression Recognition for Efficient Smart Applications

  • Chapter
  • First Online:
Security in Smart Cities: Models, Applications, and Challenges

Abstract

Facial expressions are direct means of communication of human’s emotional state. Hence facial expression recognition (FER) has always been a topic of great interest of researchers specially for smart applications. Numerous approaches have been proposed for FER using Local binary patterns (LBP). This chapter presents an analysis of LBP feature descriptor for FER. State of the art approaches have been discussed that use LBP as their main component. Here, basic LBP operator along with several variants and their main properties are described that have been proved useful for FER. A general framework for FER is described which includes four consecutive modules. These modules are preprocessing, feature extraction, dimensionality reduction and classification. LBP based FER results have been reported on three benchmark datasets JAFFE, CK+ and Yale. Experimentation demonstrates usefulness of LBP in FER.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bousmalis K, Mehu M, Pantic M (2013) Towards the automatic detection of spontaneous agreement and disagreement based on nonverbal behaviour: a survey of related cues, databases, and tools. Image Vis Comput 31(2):203–221

    Google Scholar 

  2. Bartlett MS, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing facial expression: machine learning and application to spontaneous behavior. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol 2, pp 568–573

    Google Scholar 

  3. Tian YL, Kanade T, Cohn JF (2005) Facial expression analysis. Handbook of face recognition. Springer, New York, pp 247–275

    Google Scholar 

  4. Rudovic O, Pantic M, Patras I (2013) Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans Pattern Anal Mach Intell 35(6):1357–1369

    Google Scholar 

  5. Pantic M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22(12):1424–1445

    Google Scholar 

  6. Bettadapura V (2012) Face expression recognition and analysis: the state of the art. arXiv preprint arXiv:1203.6722, http://arxiv.org/pdf/1203.6722

  7. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36(1):259–275

    MATH  Google Scholar 

  8. Ekman P, Friesen WV (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, USA

    Google Scholar 

  9. Zhang Z (1999) Feature based facial expression recognition: sensitivity analysis and experiment with a multi-layer perceptron. Int J Pattern Recognit Artif Intell 13(6):893–911

    Google Scholar 

  10. Guo G, Dyer CR (2003) Simultaneous feature selection and classifier training via linear programming: a case study for face expression recognition. In: 2003 IEEE computer society conference on computer vision and pattern recognition (CVPR’03), vol 1, pp 346–352

    Google Scholar 

  11. Valstar M, Pantic M (2006) Fully automatic facial action unit detection and temporal analysis. In: IEEE computer society computer vision and pattern recognition workshop (CVPRW’06), pp 149–149

    Google Scholar 

  12. Valstar MF, Patras I, Pantic M (2005) Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: IEEE computer society computer vision and pattern recognition workshop (CVPRW’05), vol 3, p 76

    Google Scholar 

  13. Franco L, Treves A (2001) A neural network facial expression recognition system using unsupervised local processing. In: 2nd international symposium on image and signal processing and analysis (ISPA’01), pp 628–632

    Google Scholar 

  14. Uddin MZ, Lee JJ, Kim TS (2009) An enhanced independent component-based human facial expression recognition from video. IEEE Trans Consum Electron 55(4):2216–2224

    Google Scholar 

  15. Lekshmi VP, Sasikumar M (2009) Analysis of facial expression using Gabor and SVM. Int J Recent Trends Eng 1(2):47–50

    Google Scholar 

  16. Owusu E, Zhan Y, Mao QR (2014) A neural-AdaBoost based facial expression recognition system. Expert Syst Appl 41(7):3383–3390

    Google Scholar 

  17. Kumbhar M, Jadhav A, Patil M (2012) Facial expression recognition based on image feature. Int J Comput Commun Eng 1(2):117–119

    Google Scholar 

  18. Shinohara Y, Otsu N (2004) Facial expression recognition using fisher weight maps. In: Sixth IEEE international conference on automatic face and gesture recognition, pp 499–504

    Google Scholar 

  19. Ahmed F, Bari H, Hossain E (2014) Person-independent facial expression recognition based on compound local binary pattern (CLBP). Int Arab J Inf Technol 11(2):195–203

    Google Scholar 

  20. Nigam S, Khare A (2015) Multiscale local binary patterns for facial expression-based human emotion recognition. In: Computational vision and robotics. Springer, New Delhi, pp 71–77

    Google Scholar 

  21. Darwish A, Hassanien AE, Elhoseny M, Sangaiah AK, Muhammad K (2017) The impact of the hybrid platform of internet of things and cloud computing on healthcare systems: opportunities, challenges, and open problems. J Ambient Intell Humanized Comput. First Online: 29 Dec 2017. https://doi.org/10.1007/s12652-017-0659-1

    Google Scholar 

  22. Elhoseny M, Abdelaziz A, Salama A, Riad AM, Sangaiah AK, Muhammad K (2018) A hybrid model of internet of things and cloud computing to manage big data in health services applications. Future Gen Comput Syst. Available online 15 Mar 2018 (In Press). https://doi.org/10.1016/j.future.2018.03.005

    Google Scholar 

  23. Abdelaziza A, Elhoseny M, Salama AS, Riad AM (2018) A machine learning model for improving healthcare services on cloud computing environment. Measurement 119:117–128. https://doi.org/10.1016/j.measurement.2018.01.022

    Article  Google Scholar 

  24. Zhang J, Tan T (2002) Brief review of invariant texture analysis methods. Pattern Recogn 35(3):735–747

    MATH  Google Scholar 

  25. Pietikäinen M (2005) Image analysis with local binary patterns. In: Image analysis. Springer, Berlin, pp 115–118

    Google Scholar 

  26. Mäenpää T, Pietikäinen M (2005) Texture analysis with local binary patterns. In: Handbook of pattern recognition and computer vision, vol 3, pp 197–216

    Google Scholar 

  27. Mäenpää T (2003) The Local binary pattern approach to texture analysis: extensions and applications. Doctoral dissertation, University of Oulu

    Google Scholar 

  28. Turtinen M (2007) Learning and recognizing texture characteristics using local binary patterns. Doctoral dissertation, University of Oulu

    Google Scholar 

  29. Ilea DE, Whelan PF (2011) Image segmentation based on the integration of color–texture descriptors—a review. Pattern Recogn 44(10):2479–2501

    MATH  Google Scholar 

  30. Nigam S, Khare A (2016) Integration of moment invariants and uniform local binary patterns for human activity recognition in video sequences. Multimedia Tools Appl 75(24):17303–17332

    Google Scholar 

  31. Wang A, Wang S, Lucieer A (2010) Segmentation of multispectral high-resolution satellite imagery based on integrated feature distributions. Int J Remote Sens 31(6):1471–1483

    Google Scholar 

  32. Burçin K, Vasif NV (2011) Down syndrome recognition using local binary patterns and statistical evaluation of the system. Expert Syst Appl 38(7):8690–8695

    Google Scholar 

  33. Nigam S, Khare A (2015) Multiresolution approach for multiple human detection using moments and local binary patterns. Multimedia Tools Appl 74(17):7037–7062

    Google Scholar 

  34. Mattivi R, Shao L (2010) Spatio-temporal dynamic texture descriptors for human motion recognition. In: Intelligent video event analysis and understanding. Springer, Berlin, pp 69–91

    Google Scholar 

  35. Nanni L, Brahnam S, Lumini A (2011) Combining different local binary pattern variants to boost performance. Expert Syst Appl 38(5):6209–6216

    Google Scholar 

  36. Nanni L, Lumini A, Brahnam S (2012) Survey on LBP based texture descriptors for image classification. Expert Syst Appl 39(3):3634–3641

    Google Scholar 

  37. Pietikäinen M, Ojala T, Xu Z (2000) Rotation-invariant texture classification using feature distributions. Pattern Recogn 33(1):43–52

    Google Scholar 

  38. Ojala T, Pietikäinen M, Mäenpää T (2000) Gray scale and rotation invariant texture classification with local binary patterns. In: European conference on computer vision (ECCV’2000). Springer, Berlin, pp 404–420

    Google Scholar 

  39. Zhao G, Ahonen T, Matas J, Pietikainen M (2012) Rotation-invariant image and video description with local binary pattern features. IEEE Trans Image Process 21(4):1465–1477

    MathSciNet  MATH  Google Scholar 

  40. Guo Z, Zhang L, Zhang D (2010) Rotation invariant texture classification using LBP variance (LBPV) with global matching. Pattern Recogn 43(3):706–719

    MATH  Google Scholar 

  41. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    MATH  Google Scholar 

  42. Lahdenoja O, Poikonen J, Laiho M (2013) Towards understanding the formation of uniform local binary patterns, vol 2013, Article ID 429347, 20 p. https://doi.org/10.1155/2013/429347

    Google Scholar 

  43. Chan CH, Kittler J, Messer K (2007) Multi-scale local binary pattern histograms for face recognition. Springer, Berlin, pp 809–818

    Google Scholar 

  44. Liao S, Zhu X, Lei Z, Zhang L, Li SZ (2007) Learning multi-scale block local binary patterns for face recognition. In: Advances in biometrics. Springer, Berlin, pp 828–837

    Google Scholar 

  45. Guo Z, Zhang D (2010) A completed modeling of local binary pattern operator for texture classification. IEEE Trans Image Process 19(6):1657–1663

    MathSciNet  MATH  Google Scholar 

  46. Heikkilä M, Pietikäinen M, Schmid C (2009) Description of interest regions with local binary patterns. Pattern Recogn 42(3):425–436

    MATH  Google Scholar 

  47. Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650

    MathSciNet  MATH  Google Scholar 

  48. Wang L, He DC (1990) Texture classification using texture spectrum. Pattern Recogn 23(8):905–910

    Google Scholar 

  49. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vision 57(2):137–154

    Google Scholar 

  50. Elhoseny M, Ramírez-González G, Abu-Elnasr OM, Shawkat SA, Arunkumar N, Farouk A (2018) Secure medical data transmission model for IoT-based healthcare systems. IEEE Access 6:20596–20608. https://doi.org/10.1109/access.2018.2817615

    Article  Google Scholar 

  51. Shehab A, Elhoseny M, Muhammad K, Sangaiah AK, Yang P, Huang H, Hou G (2018) Secure and robust fragile watermarking scheme for medical images. IEEE Access 6:10269–10278. https://doi.org/10.1109/access.2018.2799240

    Article  Google Scholar 

  52. Jolliffe I (2005) Principal component analysis. Wiley, New York

    MATH  Google Scholar 

  53. Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828

    Google Scholar 

  54. Gang L, Xiao-hua L, Ji-Liu Z, Xiao-gang G (2009) Geometric feature based facial expression recognition using multiclass support vector machines. In: 2009 IEEE international conference on granular computing (GRC’09), pp 318–321

    Google Scholar 

  55. Burges CJ (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 2(2):121–167

    Google Scholar 

  56. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  57. Dailey MN, Joyce C, Lyons MJ, Kamachi M, Ishi H, Gyoba J, Cottrell GW (2010) Evidence and a computational explanation of cultural differences in facial expression recognition. Emotion 10(6):874

    Google Scholar 

  58. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings of the fourth IEEE international conference on automatic face and gesture recognition, 2000. IEEE, pp 46–53

    Google Scholar 

  59. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 94–101

    Google Scholar 

  60. Yale Facial Expression Database. http://vision.ucsd.edu/content/yale-face-database

  61. Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: ICME 2005. IEEE international conference on multimedia and expo, 2005. IEEE, 5 pp

    Google Scholar 

Download references

Acknowledgements

This work is supported by Science and Engineering Research Board, Department of Science and Technology, Government of India under grant number PDF/2016/003644.

Author information

Authors and Affiliations

  1. Computer Science and Engineering Department, S. P. Memorial Institute of Technology, Kaushambi, Uttar Pradesh, India

    Swati Nigam & A. K. Misra

  2. Department of Computer Science, Banasthali Vidyapith, Banasthali, Rajasthan, India

    Rajiv Singh

Authors
  1. Swati Nigam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rajiv Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. K. Misra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rajiv Singh .

Editor information

Editors and Affiliations

  1. Faculty of Computers and Information, Cairo University, Giza, Egypt

    Aboul Ella Hassanien

  2. Faculty of Computers and Information, Mansoura University, Dakahlia, Egypt

    Mohamed Elhoseny

  3. Department of Computer Science, Georgia Southern University, Statesboro Campus, Statesboro, GA, USA

    Syed Hassan Ahmed

  4. Department of Computer Science and Engineering, National Institute of Technology Patna, Patna University Campus, Patna, Bihar, India

    Amit Kumar Singh

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Nigam, S., Singh, R., Misra, A.K. (2019). Local Binary Patterns Based Facial Expression Recognition for Efficient Smart Applications. In: Hassanien, A., Elhoseny, M., Ahmed, S., Singh, A. (eds) Security in Smart Cities: Models, Applications, and Challenges. Lecture Notes in Intelligent Transportation and Infrastructure. Springer, Cham. https://doi.org/10.1007/978-3-030-01560-2_13

Download citation

Publish with us

Policies and ethics

Search

Navigation