Skip to main content
Log in

The emotion–facial expression link: evidence from human and automatic expression recognition

  • Original Article
  • Published:
Psychological Research Aims and scope Submit manuscript

Abstract

While it has been taken for granted in the development of several automatic facial expression recognition tools, the question of the coherence between subjective feelings and facial expressions is still a subject of debate. On one hand, the “Basic Emotion View” conceives emotions as genetically hardwired and, therefore, being genuinely displayed through facial expressions. Consequently, emotion recognition is perceiver independent. On the other hand, the constructivist approach conceives emotions as socially constructed, the emotional meaning of a facial expression being inferred by the perceiver. Hence, emotion recognition is perceiver dependent. In order (1) to evaluate the coherence between the subjective feeling of emotions and their spontaneous facial displays, and (2) to compare the recognition of such displays by human perceivers and by an automatic facial expression classifier, 232 videos of expressers recruited to carry out an emotion elicitation task were annotated by 1383 human perceivers as well as by Affdex, an automatic classifier. Results show a weak consistency between self-reported emotional states by expressers and their facial emotional displays. They also show low accuracy both of human perceivers and of the automatic classifier to infer the subjective feeling from the spontaneous facial expressions displayed by expressers. However, the results are more in favor of a perceiver-dependent view. Based on these results, the hypothesis of genetically hardwired emotion genuinely displayed is difficult to support, whereas the idea of emotion and facial expression as being socially constructed appears to be more likely. Accordingly, automatic emotion recognition tools based on facial expressions should be questioned.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

The R code and relevant data for statistical computing are available on the “The emotion–facial expression link (data and code)” repository from the Center for Open Science at https://osf.io/erua5/ (Dupré & Tcherkassof, 2020). The data that support the findings of this study are available from the DynEmo database but restrictions apply to the availability of these data. DynEmo is devoted to academic research to the exclusion of any other purpose, and so the data are not publicly available. Data are however available from the authors upon reasonable request and with permission of DynEmo.

References

Download references

Acknowledgements

The authors would like to thank the developers of the following R packages used to process, analyze, display and report data: R (Version 4.0.2; R Core Team, 2019) and the R-packages caret (Version 6.0.86; Jed Wing et al., 2019), here (Version 0.1; Müller, 2017), janitor (Version 2.0.1; Firke, 2019), kableExtra (Version 1.2.1; Zhu, 2019), papaja (Version 0.1.0.9997; Aust & Barth, 2018), and tidyverse (Version 1.3.0; Wickham, 2017).

Author information

Authors and Affiliations

Authors

Contributions

The authors made the following contributions. AT: Conceptualization, Project administration, Validation, Writing—Original Draft Preparation, Writing—Review and Editing; DD: Investigation, Data curation, Formal Analysis, Software, Visualization, Writing—Original Draft Preparation, Writing—Review and Editing.

Corresponding author

Correspondence to Anna Tcherkassof.

Ethics declarations

Conflicts of interest

No potential conflicts of interests regarding the research reported in the manuscript exist.

Ethical approval

This is an observational study. The CERGA Research Ethics Committee has confirmed that no ethical approval is required.

Informed consent

Written informed consent was obtained from all individual participants included in the original study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tcherkassof, A., Dupré, D. The emotion–facial expression link: evidence from human and automatic expression recognition. Psychological Research 85, 2954–2969 (2021). https://doi.org/10.1007/s00426-020-01448-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00426-020-01448-4

Navigation