Abstract
In this paper, a face emotion is considered as the result of the composition of multiple concurrent signals, each corresponding to the movements of a specific facial muscle. These concurrent signals are represented by means of a set of multi-scale appearance features that might be correlated with one or more concurrent signals. The extraction of these appearance features from a sequence of face images yields to a set of time series. This paper proposes to use the dynamics regulating each appearance feature time series to recognize among different face emotions. To this purpose, an ensemble of Hankel matrices corresponding to the extracted time series is used for emotion classification within a framework that combines nearest neighbor and a majority vote schema. Experimental results on a public available dataset show that the adopted representation is promising and yields state-of-the-art accuracy in emotion classification.
Chapter PDF
Similar content being viewed by others
References
Chew, S.W., Lucey, P., Lucey, S., Saragih, J., Cohn, J.F., Sridharan, S.: Person-independent facial expression detection using constrained local models. In: Conf. and Workshop on Automatic Face & Gesture Recognition (FG), pp. 915–920. IEEE (2011)
Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI) 23(6), 681–685 (2001)
Cuturi, M.: Fast global alignment kernels. In: Int. Conf. on Machine Learning (ICML), pp. 929–936 (2011)
Cuturi, M., Vert, J., Birkenes, O., Matsui, T.: A kernel for time series based on global alignments. In: Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), vol. 2, pp. II-413. IEEE (2007)
Dibeklioğlu, H.: Enabling dynamics in face analysis. Ph.D. thesis, University of Amsterdam (2014)
Fasel, B., Luettin, J.: Automatic facial expression analysis: a survey. Pattern Recognition 36(1), 259–275 (2003)
Hare, S., Saffari, A., Torr, P.H.: Struck: Structured output tracking with kernels. In: Int. Conf. on Computer Vision (ICCV), pp. 263–270. IEEE (2011)
Huang, C., Ai, H., Li, Y., Lao, S.: High-performance rotation invariant multiview face detection. IEEE Trans. on Pattern Analysis and Machine Intelligence 29(4), 671–686 (2007)
Jeni, L.A., Girard, J.M., Cohn, J.F., De La Torre, F.: Continuous AU intensity estimation using localized, sparse facial feature space. In: Conf. on Automatic Face & Gesture Recognition (FG), pp. 1–7. IEEE (2013)
Lacava, P.G., Golan, O., Baron-Cohen, S., Myles, B.S.: Using assistive technology to teach emotion recognition to students with asperger syndrome a pilot study. Remedial and Special Education 28(3), 174–181 (2007)
Lee, H.Y., Lee, W.H.: A study on interactive media art to apply emotion recognition. International Journal of Multimedia & Ubiquitous Engineering 9(12) (2014)
Li, B., Camps, O.I., Sznaier, M.: Cross-view activity recognition using Hankelets. In: Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1362–1369. IEEE (2012)
Lienhart, R., Maydt, J.: An extended set of haar-like features for rapid object detection. In: Int. Conf. on Image Processing, vol. 1, pp. I-900. IEEE (2002)
Lo Presti, L., La Cascia, M.: An on-line learning method for face association in personal photo collection. Image and Vision Computing (2012)
Lo Presti, L., La Cascia, M.: Tracking your detector performance: how to grow an effective training set in tracking-by-detection methods. In: Int. Conf. on Computer Vision Theory and Applications (VISAPP), pp. 1–8 (2015)
Lo Presti, L., La Cascia, M.: Using Hankel matrices for Dynamics-based Facial Emotion Recognition and Pain Detection. In: Int. Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1–8 (2015)
Lo Presti, L., La Cascia, M., Sclaroff, S., Camps, O.: Gesture modeling by hanklet-based hidden markov model. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9005, pp. 529–546. Springer, Heidelberg (2015)
Lo Presti, L., Sclaroff, S., Rozga, A.: Joint alignment and modeling of correlated behavior streams. In: Int. Conf. on Computer Vision-Workshops (ICCVW), pp. 730–737 (2013)
Lorincz, A., Jeni, L.A., Szabó, Z., Cohn, J.F., Kanade, T.: Emotional expression classification using time-series kernels. In: Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 889–895. IEEE (2013)
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The Extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101. IEEE (2010)
Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., Matthews, I.: Painful data: the UNBC-McMaster shoulder pain expression archive database. In: Conf. and W. on Automatic Face & Gesture Recognition (FG), pp. 57–64. IEEE (2011)
Nie, S., Wang, Z., Ji, Q.: A generative restricted Boltzmann machine based method for high-dimensional motion data modeling. Computer Vision and Image Understanding (2015)
Rabbitt, S.M., Kazdin, A.E., Scassellati, B.: Integrating socially assistive robotics into mental healthcare interventions: Applications and recommendations for expanded use. Clinical Psychology Review 35, 35–46 (2015)
Ramirez Rivera, A., Castillo, R., Chae, O.: Local directional number pattern for face analysis: Face and expression recognition. IEEE Transactions on Image Processing (TIP) 22(5), 1740–1752 (2013)
Rehg, J.M., et al.: Decoding children’s social behavior. In: Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 3414–3421. IEEE (2013)
Sankaranarayanan, A.C., Turaga, P.K., Baraniuk, R.G., Chellappa, R.: Compressive acquisition of dynamic scenes. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part I. LNCS, vol. 6311, pp. 129–142. Springer, Heidelberg (2010)
Sariyanidi, E., Gunes, H., Cavallaro, A.: Automatic analysis of facial affect: A survey of registration, representation and recognition. EEE Trans. on Pattern Analysis and Machine Intelligence (PAMI) (2014)
Shan, C., Gong, S., McOwan, P.W.: Dynamic facial expression recognition using a Bayesian temporal manifold model. In: BMVC, pp. 297–306 (2006)
Slama, R., Wannous, H., Daoudi, M., Srivastava, A.: Accurate 3D action recognition using learning on the Grassmann manifold. Pattern Recognition (PR) 48(2), 556–567 (2015)
Viberg, M.: Subspace-based methods for the identification of linear time-invariant systems. Automatica 31(12), 1835–1851 (1995)
Viola, P., Jones, M.J.: Robust real-time face detection. International Journal of Computer Vision 57(2), 137–154 (2004)
Wang, Z., Wang, S., Ji, Q.: Capturing complex spatio-temporal relations among facial muscles for facial expression recognition. In: Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 3422–3429. IEEE (2013)
Yang, P., Liu, Q., Metaxas, D.: Similarity features for facial event analysis. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 685–696. Springer, Heidelberg (2008)
Yang, P., Liu, Q., Metaxas, D.N.: Boosting coded dynamic features for facial action units and facial expression recognition. In: Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1–6. IEEE (2007)
Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI) 31(1), 39–58 (2009)
Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI) 29(6), 915–928 (2007)
Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A.: Face recognition: A literature survey. ACM Computing Surveys (CSUR) 35(4), 399–458 (2003)
Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2879–2886. IEEE (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Lo Presti, L., La Cascia, M. (2015). Ensemble of Hankel Matrices for Face Emotion Recognition. In: Murino, V., Puppo, E. (eds) Image Analysis and Processing — ICIAP 2015. ICIAP 2015. Lecture Notes in Computer Science(), vol 9280. Springer, Cham. https://doi.org/10.1007/978-3-319-23234-8_54
Download citation
DOI: https://doi.org/10.1007/978-3-319-23234-8_54
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-23233-1
Online ISBN: 978-3-319-23234-8
eBook Packages: Computer ScienceComputer Science (R0)