Extracting Local Binary Patterns from Image Key Points: Application to Automatic Facial Expression Recognition

  • Xiaoyi Feng
  • Yangming Lai
  • Xiaofei Mao
  • Jinye Peng
  • Xiaoyue Jiang
  • Abdenour Hadid
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7944)


Facial expression recognition has widely been investigated in the literature. The need of accurate facial alignment has however limited the deployment of facial expression systems in real-world applications. In this paper, a novel feature extraction method is proposed. It is based on extracting local binary patterns (LBP) from image key points. The face region is first segmented into six facial components (left eye, right eye, left eyebrow, right eyebrow, nose, and mouth). Then, local binary patterns are extracted only from the edge points of each facial component. Finally, the local binary pattern features are collected into a histogram and fed to an SVM classifier for facial expression recognition. Compared to the traditional LBP methodology extracting the features from all image pixels, our proposed approach extracts LBP features only from a set of points of face components, yielding in more compact and discriminative representations. Furthermore, our proposed approach does not require face alignment. Extensive experimental analysis on the commonly used JAFFE facial expression benchmark database showed very promising results, outperforming those of the traditional local binary pattern approach.


Local Binary Patterns Facial expression Recognition Key Points 


  1. 1.
    Zhao, G., Pietikäinen, M.: Dynamic Texture Recognition Using Local Binary Patterns with An Application to Facial Expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)CrossRefGoogle Scholar
  2. 2.
    Huang, X., Zhao, G., Zheng, W., Pietikäinen, M.: Spatiotemporal Local Monogenic Binary Patterns for Facial Expression Recognition. IEEE Signal Processing Letters 19(5), 243–246 (2012)CrossRefGoogle Scholar
  3. 3.
    Feng, X., Pietikäinen, M., Hadid, A.: Facial Expression Recognition with Local Binary Patterns and Linear Programming. Pattern Recognition and Image Analysis 15(2), 546–548 (2005)Google Scholar
  4. 4.
    He, L., Zou, C., Zhao, L., Hu, D.: An enhanced LBP feature based on facial expression recognition. In: Proc. Ann. Int. Conf. Eng. Med. Biol. Soc., pp. 3300–3303 (2005)Google Scholar
  5. 5.
    Liao, S., Fan, W., Chung, A.C.S., Yeung, D.Y.: Facial expression recognition using advanced local binary patterns, tsallis entropies and global appearance features. In: Proc. IEEE Int. Conf. Image Process., pp. 665–668 (October 2006)Google Scholar
  6. 6.
    Feng, X., Lv, B., Li, Z., Zhang, J.: A novel feature extraction method for facial expression recognition. In: Proc. Joint Conf. Inform. Sci. Issue Adv. Intell. Syst. Res., Kaohsiung, Taiwan, pp. 371–375 (2006)Google Scholar
  7. 7.
    Cao, J., Tong, C.: Facial expression recognition based on LBP-EHMM. In: Proc. Congr. Image Signal Process. (2008)Google Scholar
  8. 8.
    Fu, X., Wei, W.: Centralized binary patterns embedded with image Euclidean distance for facial expression recognition. In: Proc. Int. Conf. Neural Comput., pp. IV: 115–119 (October 2008)Google Scholar
  9. 9.
    Ahmed, F., Hossain, E., Bari, A., Shihavuddin, A.: Compound Local Binary Pattern (CLBP) for Robust Facial Expression Recognition. In: International Conference of Soft Computing and Pattern Recognition (SoCPaR), pp. 391–395 (2011)Google Scholar
  10. 10.
    Lyons, M., Akamastu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with Gabor wavelets. In: Proc. Third IEEE Conf. Face and Gesture Recognition, Nara, Japan, pp. 200–205 (April 1998)Google Scholar
  11. 11.
    Nilsson, M., Nordberg, J., Claesson, I.: Face Detection Using Local SMQT Features and Split up SNOW Classifiers. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP (2007)Google Scholar
  12. 12.
    Valstar, M.F., Pantic, M.: Biologically vs. logic inspired encoding of facial actions and emotions. In: Proc. of IEEE Intl. Conf. on Multimedia and Expo (ICME), pp. 325–328 (2006)Google Scholar
  13. 13.
    Bartlett, M.S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., Movellan, J.: Recognizing facial expression: machine learning and application to spontaneous behavior. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2005)Google Scholar
  14. 14.
    Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on Local Binary Patterns: A comprehensive study. Image and Vision Computing 27, 27803–27816 (2009)CrossRefGoogle Scholar
  15. 15.
    David, D.G.: Lowe, Distinctive Image Features from Scale-Invariant Points. International Journal of Computer Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  16. 16.
    Ojala, T., Pietikäinen, M., Mäenpää, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7), 971–987 (2002)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Xiaoyi Feng
    • 1
  • Yangming Lai
    • 1
  • Xiaofei Mao
    • 1
  • Jinye Peng
    • 1
  • Xiaoyue Jiang
    • 1
  • Abdenour Hadid
    • 2
  1. 1.College of Electronics and InformationNorthwestern Polytechnic UniversityXianChina
  2. 2.Center for Machine Vision Research, Department of Computer Science and EngineeringUniversity of OuluFinland

Personalised recommendations