Abstract
We investigate the personalization of deep convolutional neural networks for facial expression analysis from still images. While prior work has focused on population-based (“one-size-fits-all”) approaches, we formulate and construct personalized models via a mixture of experts and supervised domain adaptation approach, showing that it improves greatly upon non-personalized models. Our experiments demonstrate the ability of the model personalization to quickly and effectively adapt to limited amounts of target data. We also provide a novel training methodology and architecture for creating personalized machine learning models for more effective analysis of emotion state.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
For notational simplicity, we drop here the dependence on the source/target subjects.
- 2.
For instance, only the ResNet features of target subjects need be provided as input to the adapted model, as original face images cannot be reconstructed from those features.
- 3.
Note, however, that the Resnet used to extract the features for these models was fine-tuned using the labeled source data.
References
Peterson, K., Rudovic, O., Guerrero, R., Picard, R.W.: Personalized Gaussian processes for future prediction of Alzheimer’s disease progression. In: NIPS Workshop on Machine Learning for Healthcare (2017)
Jaques, N., Rudovic, O., Taylor, S., Sano, A., Picard, R.: Predicting tomorrow’s mood, health, and stress level using personalized multitask learning and domain adaptation. In: IJCAI Workshop (2017)
Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE TPAMI 31, 39–58 (2009)
Rudovic, O., Lee, J., Dai, M., Schuller, B., Picard, R.: Personalized machine learning for robot perception of affect and engagement in autism therapy. arXiv preprint arXiv:1802.01186 (2018)
Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE TAC (2017)
Martinez, D.L., Rudovic, O., Picard, R.: Personalized automatic estimation of self-reported pain intensity from facial expressions. In: IEEE CVPR’W (2017)
Csurka, G.: Domain adaptation for visual applications: a comprehensive survey. CoRR, abs/1702.05374 (2017). http://arxiv.org/abs/1702.05374
Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive mixtures of local experts. Neural Comput. 3(1), 79–87 (1991)
Ringeval, F., Sonderegger, A., Sauer, J., Lalanne, D.: Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions. In: IEEE FG (Workshops) (2013)
Valstar, M., Gratch, J., Schuller, B., Ringeval, F., Lalanne, D., Torres Torres, M., Scherer, S., Stratou, G., Cowie, R., Pantic, M.: Avec 2016: depression, mood, and emotion recognition workshop and challenge. In: Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge. ACM (2016)
Collobert, R., Bengio, S., Bengio, Y.: A parallel mixture of SVMS for very large scale problems. In: NIPS (2002)
Shahbaba, B., Neal, R.: Nonlinear models using Dirichlet process mixtures. JMLR 10, 1829–1850 (2009)
Theis, L., Bethge, M.: Generative image modeling using spatial LSTMS. In: NIPS (2015)
Yao, B., Walther, D., Beck, D., Fei-Fei, L.: Hierarchical mixture of classification experts uncovers interactions between brain regions. In: NIPS (2009)
Rasmussen, C.E., Ghahramani, Z.: Infinite mixtures of Gaussian process experts. In: NIPS (2002)
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In: ICLR (2017)
Jiang, H., Learned-Miller, E.: Face detection with the faster R-CNN. In: IEEE FG (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE CVPR (2016)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)
Jiang, J.: A literature survey on domain adaptation of statistical classifiers. 3 (2008). http://sifaka.cs.uiuc.edu/jiang4/domainadaptation/survey
Chollet, F., et al.: Keras (2015)
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.: Tensorflow: a system for large-scale machine learning. In: OSDI (2016)
Tzirakis, P., Zafeiriou, S., Schuller, B.W.: End2you-the imperial toolkit for multimodal profiling by end-to-end learning. arXiv preprint arXiv:1802.01115 (2018)
Acknowledgments
The work of O. Rudovic has been funded by the European Union H2020, Marie Curie Action - Individual Fellowship no. 701236 (EngageMe).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Feffer, M., Rudovic, O.(., Picard, R.W. (2018). A Mixture of Personalized Experts for Human Affect Estimation. In: Perner, P. (eds) Machine Learning and Data Mining in Pattern Recognition. MLDM 2018. Lecture Notes in Computer Science(), vol 10935. Springer, Cham. https://doi.org/10.1007/978-3-319-96133-0_24
Download citation
DOI: https://doi.org/10.1007/978-3-319-96133-0_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-96132-3
Online ISBN: 978-3-319-96133-0
eBook Packages: Computer ScienceComputer Science (R0)