Advertisement

Nonverbal Synchrony of Facial Movements and Expressions Predict Therapeutic Alliance During a Structured Psychotherapeutic Interview

  • Kenji YokotaniEmail author
  • Gen Takagi
  • Kobun Wakashima
Original Paper

Abstract

Nonverbal synchrony (NVS) of a patient’s and therapist’s body parts during a therapy session has been linked with therapeutic alliance. However, the link between NVS of face parts with therapeutic alliance remains unclear. The clarification of this link is important in understanding NVS. Accordingly, we used a video imaging technique to provide quantitative evidence of this link. The 55 participants in this study were the same as in a previous study. Both the participants’ and the therapist’s faces were video recorded during structured psychotherapeutic interviews. Our machine quantified 500,500 participants’ faces and 500,500 therapists’ faces from the perspectives of facial movements and expressions. Results show that absolute synchrony of happy and scared expressions were positively related to therapeutic alliance. However, symmetrical synchrony of left eye movements negatively predicted therapeutic alliance, although participants’ sex, age, volume of facial movements, and volume of facial expressions were controlled. Absolute synchrony of facial expressions was regarded as emotional interaction within 2 s delay, whereas symmetrical synchrony of left eye movements was regarded as a blocker of emotional interaction.

Keywords

Nonverbal synchrony Facial movement Facial expression Video imaging technique Structured psychotherapeutic interview Symmetrical communication pattern 

Notes

Acknowledgements

We appreciate Dr. Kurosawa, Tai for his insightful feedbacks on our early draft.

Funding

The present study was funded by a Grant from the Foundation for the Fusion of Science and Technology (Heisei27-10) and from the Japan Society for the Promotion of Science Grants-in-Aid for Scientific Research(18K02141).

References

  1. Aas, I. M. (2011). Guidelines for rating global assessment of functioning (GAF). Annals of General Psychiatry, 10(1), 2.  https://doi.org/10.1186/1744-859X-10-2.CrossRefPubMedPubMedCentralGoogle Scholar
  2. Acosta, J. C., & Ward, N. G. (2011). Achieving rapport with turn-by-turn, user-responsive emotional coloring. Speech Communication, 53(9), 1137–1148.  https://doi.org/10.1016/j.specom.2010.11.006.CrossRefGoogle Scholar
  3. Arriaga, O., Valdenegro-Toro, M., & Plöger, P. (2017). Real-time convolutional neural networks for emotion and gender classification. arXiv:1710.07557 [Cs]. Retrieved from http://arxiv.org/abs/1710.07557.
  4. Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., & Plumb, I. (2001). The “reading the mind in the eyes” test, revised version: A study with normal adults, and adults with Asperger syndrome or high-functioning autism. Journal of Child Psychology and Psychiatry, 42(2), 241–251.  https://doi.org/10.1111/1469-7610.00715.CrossRefPubMedGoogle Scholar
  5. Bernieri, F. J. (1988). Coordinated movement and rapport in teacher–student interactions. Journal of Nonverbal Behavior, 12(2), 120–138.  https://doi.org/10.1007/BF00986930.CrossRefGoogle Scholar
  6. Bernieri, F. J., Davis, J. M., Rosenthal, R., & Knee, C. R. (1994). Interactional synchrony and rapport: Measuring synchrony in displays devoid of sound and facial affect. Personality and Social Psychology Bulletin, 20(3), 303–311.  https://doi.org/10.1177/0146167294203008.CrossRefGoogle Scholar
  7. Boker, S. M., Xu, M., Rotondo, J. L., & King, K. (2002). Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series. Psychological Methods, 7(3), 338–355.  https://doi.org/10.1037/1082-989X.7.3.338.CrossRefPubMedGoogle Scholar
  8. Bradski, G., & Kaehler, A. (2008). Learning OpenCV: Computer vision with the OpenCV library. O’Reilly Media, Inc.Google Scholar
  9. Carrier, P.-L., Courville, A., Goodfellow, I. J., Mirza, M., & Bengio, Y. (2013). FER-2013 face database. Universit de Montral.Google Scholar
  10. Chartrand, T. L., & Bargh, J. A. (1999). The chameleon effect: The perception–behavior link and social interaction. Journal of Personality and Social Psychology, 76(6), 893–910.  https://doi.org/10.1037/0022-3514.76.6.893.CrossRefPubMedGoogle Scholar
  11. Chartrand, T. L., & Lakin, J. L. (2013). The antecedents and consequences of human behavioral mimicry. Annual Review of Psychology, 64(1), 285–308.  https://doi.org/10.1146/annurev-psych-113011-143754.CrossRefPubMedGoogle Scholar
  12. Condon, W. S., & Ogston, W. D. (1966). Sound film analysis of normal and pathological behavior patterns. Journal of Nervous and Mental Disease, 143(4), 338–347.  https://doi.org/10.1097/00005053-196610000-00005.CrossRefPubMedGoogle Scholar
  13. D’Mello, S., Picard, R. W., & Graesser, A. (2007). Toward an affect-sensitive autotutor. IEEE Intelligent Systems, 22(4), 53–61.  https://doi.org/10.1109/MIS.2007.79.CrossRefGoogle Scholar
  14. de la Peña, C. M., Friedlander, M. L., Escudero, V., & Heatherington, L. (2012). How do therapists ally with adolescents in family therapy? An examination of relational control communication in early sessions. Journal of Counseling Psychology, 59(3), 339–351.  https://doi.org/10.1037/a0028063.CrossRefPubMedGoogle Scholar
  15. Ekman, P. (1993). Facial expression and emotion. American Psychologist, 48(4), 384–392.  https://doi.org/10.1037/0003-066X.48.4.384.CrossRefPubMedGoogle Scholar
  16. Ekman, P. (2003). Darwin, deception, and facial expression. Annals of the New York Academy of Sciences, 1000(1), 205–221.CrossRefGoogle Scholar
  17. Ekman, P., & Friesen, W. V. (1976). Measuring facial movement. Environmental Psychology and Nonverbal Behavior, 1(1), 56–75.  https://doi.org/10.1007/BF01115465.CrossRefGoogle Scholar
  18. Elvins, R., & Green, J. (2008). The conceptualization and measurement of therapeutic alliance: An empirical review. Clinical Psychology Review, 28(7), 1167–1187.  https://doi.org/10.1016/j.cpr.2008.04.002.CrossRefPubMedGoogle Scholar
  19. Erchul, W. P., Sheridan, S. M., Ryan, D. A., Grissom, P. F., Killough, C. E., & Mettler, D. W. (1999). Patterns of relational communication in conjoint behavioral consultation. School Psychology Quarterly, 14(2), 121–147.CrossRefGoogle Scholar
  20. Escudero, V., Rogers, L. E., & Gutierrez, E. (1997). Patterns of relational control and nonverbal affect in clinic and nonclinic couples. Journal of Social and Personal Relationships, 14(1), 5–29.  https://doi.org/10.1177/0265407597141001.CrossRefGoogle Scholar
  21. First, M. B., Spitzer, R. L., Gibbon, M., & Williams, J. B. W. (1997). Structured clinical interview for DSM-IV axis I disorders. Washington, DC: American Psychiatric Publishing Inc.Google Scholar
  22. First, M. B., Spitzer, R. L., Gibbon, M., & Williams, J. B. W. (2010). Structured clinical interview for DSM-IV axis I disorders (S. Takahashi, T. Kitamura, T. Okano, & A. Kikuchi, Trans.). Tokyo: Nippon Hyoron Sha co., Ltd. (Original work published 1997)Google Scholar
  23. Fraser, S., Vachon, M., Hassan, G., & Parent, V. (2016). Communicating power and resistance: Exploring interactions between aboriginal youth and non-aboriginal staff members in a residential child welfare facility. Qualitative Research in Psychology, 13(1), 67–91.  https://doi.org/10.1080/14780887.2015.1106629.CrossRefGoogle Scholar
  24. Gatewood, J. B., & Rosenwein, R. (1981). Interactional synchrony: Genuine or spurious? A critique of recent research. Journal of Nonverbal Behavior, 6(1), 12–29.  https://doi.org/10.1007/BF00987933.CrossRefGoogle Scholar
  25. Graham, M. H. (2003). Confronting multicollinearity in ecological multiple regression. Ecology, 84(11), 2809–2815.  https://doi.org/10.1890/02-3114.CrossRefGoogle Scholar
  26. Heatherington, L., & Friedlander, M. L. (1990). Complementarity and symmetry in family therapy communication. Journal of Counseling Psychology, 37(3), 261–268.  https://doi.org/10.1037/0022-0167.37.3.261.CrossRefGoogle Scholar
  27. Hughes, S. M., & Aung, T. (2018). Symmetry in motion: Perception of attractiveness changes with facial movement. Journal of Nonverbal Behavior, 42(3), 267–283.  https://doi.org/10.1007/s10919-018-0277-4.CrossRefGoogle Scholar
  28. Kakii, T. (1997). Characteristics of multimedia counseling: A study of an interactive TV system. The Japanese Journal of Psychology, 68(1), 9–16.  https://doi.org/10.4992/jjpsy.68.9.CrossRefPubMedGoogle Scholar
  29. Kimura, M., & Daibo, I. (2006). Interactional synchrony in conversations about emotional episodes: A measurement by “the between-participants pseudosynchrony experimental paradigm”. Journal of Nonverbal Behavior, 30(3), 115–126.  https://doi.org/10.1007/s10919-006-0011-5.CrossRefGoogle Scholar
  30. King, D. E. (2009). Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10, 1755–1758.Google Scholar
  31. Künecke, J., Wilhelm, O., & Sommer, W. (2017). Emotion recognition in nonverbal face-to-face communication. Journal of Nonverbal Behavior, 41(3), 221–238.  https://doi.org/10.1007/s10919-017-0255-2.CrossRefGoogle Scholar
  32. Kupper, Z., Ramseyer, F., Hoffmann, H., & Tschacher, W. (2015). Nonverbal synchrony in social interactions of patients with schizophrenia indicates socio-communicative deficits. PLoS ONE, 10(12), e0145882.  https://doi.org/10.1371/journal.pone.0145882.CrossRefPubMedPubMedCentralGoogle Scholar
  33. Lakin, J. L., & Chartrand, T. L. (2003). Using nonconscious behavioral mimicry to create affiliation and rapport. Psychological Science, 14(4), 334–339.  https://doi.org/10.1111/1467-9280.14481.CrossRefPubMedPubMedCentralGoogle Scholar
  34. Langlois, J. H., & Roggman, L. A. (1990). Attractive faces are only average. Psychological Science, 1(2), 115–121.  https://doi.org/10.1111/j.1467-9280.1990.tb00079.x.CrossRefGoogle Scholar
  35. Lavelle, M., Healey, P. G. T., & McCabe, R. (2013). Is nonverbal communication disrupted in interactions involving patients with schizophrenia? Schizophrenia Bulletin, 39(5), 1150–1158.  https://doi.org/10.1093/schbul/sbs091.CrossRefPubMedGoogle Scholar
  36. Lee, D. H., Mirza, R., Flanagan, J. G., & Anderson, A. K. (2014). Optical origins of opposing facial expression actions. Psychological Science, 25(3), 745–752.  https://doi.org/10.1177/0956797613514451.CrossRefPubMedGoogle Scholar
  37. Levi, G., & Hassner, T. (2015). Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In Proceedings of the 2015 ACM on international conference on multimodal interaction (pp. 503–510).  https://doi.org/10.1145/2818346.2830587.
  38. Lozza, N., Spoerri, C., Ehlert, U., Kesselring, M., Hubmann, P., Tschacher, W., et al. (2018). Nonverbal synchrony and complementarity in unacquainted same-sex dyads: A comparison in a competitive context. Journal of Nonverbal Behavior, 42(2), 179–197.  https://doi.org/10.1007/s10919-018-0273-8.CrossRefGoogle Scholar
  39. Martin, D. J., Garske, J. P., & Katherine, M. (2000). Relation of the therapeutic alliance with outcome and other variables: A meta-analytic review. Journal of Consulting and Clinical Psychology, 68(3), 438–450.  https://doi.org/10.1037/0022-006X.68.3.438.CrossRefPubMedGoogle Scholar
  40. Matsugu, M., Mori, K., Mitari, Y., & Kaneda, Y. (2003). Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Networks, 16(5), 555–559.  https://doi.org/10.1016/S0893-6080(03)00115-1.CrossRefPubMedGoogle Scholar
  41. McFarland, D. H. (2001). Respiratory markers of conversational interaction. Journal of Speech, Language, and Hearing Research, 44(1), 128–143.  https://doi.org/10.1044/1092-4388(2001/012).CrossRefPubMedGoogle Scholar
  42. Miles, L. K., Lumsden, J., Richardson, M. J., & Neil Macrae, C. (2011). Do birds of a feather move together? Group membership and behavioral synchrony. Experimental Brain Research, 211(3), 495–503.  https://doi.org/10.1007/s00221-011-2641-z.CrossRefPubMedGoogle Scholar
  43. Paladino, M.-P., Mazzurega, M., Pavani, F., & Schubert, T. W. (2010). Synchronous multisensory stimulation blurs self-other boundaries. Psychological Science, 21(9), 1202–1207.  https://doi.org/10.1177/0956797610379234.CrossRefPubMedGoogle Scholar
  44. Paulick, J., Deisenhofer, A.-K., Ramseyer, F., Tschacher, W., Boyle, K., Rubel, J., et al. (2017). Nonverbal synchrony: A new approach to better understand psychotherapeutic processes and drop-out. Journal of Psychotherapy Integration, No Pagination Specified-No Pagination Specified..  https://doi.org/10.1037/int0000099.CrossRefGoogle Scholar
  45. Paulick, J., Rubel, J. A., Deisenhofer, A.-K., Schwartz, B., Thielemann, D., Altmann, U., et al. (2018). Diagnostic features of nonverbal synchrony in psychotherapy: Comparing depression and anxiety. Cognitive Therapy and Research, 42(5), 539–551.  https://doi.org/10.1007/s10608-018-9914-9.CrossRefGoogle Scholar
  46. Ramseyer, F., & Tschacher, W. (2011). Nonverbal synchrony in psychotherapy: Coordinated body movement reflects relationship quality and outcome. Journal of Consulting and Clinical Psychology, 79(3), 284–295.  https://doi.org/10.1037/a0023419.CrossRefPubMedGoogle Scholar
  47. Ramseyer, F., & Tschacher, W. (2014). Nonverbal synchrony of head- and body-movement in psychotherapy: Different signals have different associations with outcome. Frontiers in Psychology.  https://doi.org/10.3389/fpsyg.2014.00979.CrossRefPubMedPubMedCentralGoogle Scholar
  48. Repp, B. H., & Su, Y.-H. (2013). Sensorimotor synchronization: A review of recent research (2006–2012). Psychonomic Bulletin & Review, 20(3), 403–452.  https://doi.org/10.3758/s13423-012-0371-2.CrossRefGoogle Scholar
  49. Riehle, M., Kempkensteffen, J., & Lincoln, T. M. (2017). Quantifying facial expression synchrony in face-to-face dyadic interactions: Temporal dynamics of simultaneously recorded facial EMG signals. Journal of Nonverbal Behavior, 41(2), 85–102.  https://doi.org/10.1007/s10919-016-0246-8.CrossRefGoogle Scholar
  50. Riehle, M., & Lincoln, T. M. (2018). Investigating the social costs of schizophrenia: Facial expressions in dyadic interactions of people with and without schizophrenia. Journal of Abnormal Psychology, 127(2), 202–215.  https://doi.org/10.1037/abn0000319.CrossRefPubMedGoogle Scholar
  51. Rogers, L. E., & Farace, R. V. (1975). Analysis of relational communication in dyads: New measurement procedures. Human Communication Research, 1(3), 222–239.  https://doi.org/10.1111/j.1468-2958.1975.tb00270.x.CrossRefGoogle Scholar
  52. Schmidt, R. C., Morr, S., Fitzpatrick, P., & Richardson, M. J. (2012). Measuring the dynamics of interactional synchrony. Journal of Nonverbal Behavior, 36(4), 263–279.  https://doi.org/10.1007/s10919-012-0138-5.CrossRefGoogle Scholar
  53. Semin, G. R., & Cacioppo, J. T. (2008). Grounding social cognition: Synchronization, coordination, and co-regulation. In Embodied grounding: Social, cognitive, affective, and neuroscientific approaches (pp. 119–147).  https://doi.org/10.1017/CBO9780511805837.006.
  54. Shockley, K., Santana, M.-V., & Fowler, C. A. (2003). Mutual interpersonal postural constraints are involved in cooperative conversation. Journal of Experimental Psychology: Human Perception and Performance, 29(2), 326–332.  https://doi.org/10.1037/0096-1523.29.2.326.CrossRefPubMedGoogle Scholar
  55. Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., et al. (2009). Multiple imputation for missing data in epidemiological and clinical research: Potential and pitfalls. BMJ, 338, b2393.  https://doi.org/10.1136/bmj.b2393.CrossRefPubMedPubMedCentralGoogle Scholar
  56. Stratou, G., Hoegen, R., Lucas, G., & Gratch, J. (2017). Investigating gender differences in temporal dynamics during an iterated social dilemma: An automatic analysis using networks. In 2017 Seventh international conference on affective computing and intelligent interaction (ACII) (pp. 531–536).  https://doi.org/10.1109/ACII.2017.8273650.
  57. Tschacher, W., & Pfammatter, M. (2016). Embodiment in psychotherapy—A necessary complement to the canon of common factors? European Psychotherapy, 2016(2017), 5–21.  https://doi.org/10.7892/boris.93002.CrossRefGoogle Scholar
  58. Tschacher, W., Rees, G. M., & Ramseyer, F. (2014). Nonverbal synchrony and affect in dyadic interactions. Frontiers in Psychology.  https://doi.org/10.3389/fpsyg.2014.01323.CrossRefPubMedPubMedCentralGoogle Scholar
  59. Vacharkulksemsuk, T., & Fredrickson, B. L. (2012). Strangers in sync: Achieving embodied rapport through shared movements. Journal of Experimental Social Psychology, 48(1), 399–402.  https://doi.org/10.1016/j.jesp.2011.07.015.CrossRefPubMedPubMedCentralGoogle Scholar
  60. Vicaria, I. M., & Dickens, L. (2016). Meta-analyses of the intra- and interpersonal outcomes of interpersonal coordination. Journal of Nonverbal Behavior, 40(4), 335–361.  https://doi.org/10.1007/s10919-016-0238-8.CrossRefGoogle Scholar
  61. Watzlawick, P., Bavelas, J. B., & Jackson, D. D. (2011). Pragmatics of human communication: A study of interactional patterns, pathologies and paradoxes. New York: W. W. Norton & Company.Google Scholar
  62. Won, A. S., Bailenson, J. N., Stathatos, S. C., & Dai, W. (2014). Automatically detected nonverbal behavior predicts creativity in collaborating dyads. Journal of Nonverbal Behavior, 38(3), 389–408.  https://doi.org/10.1007/s10919-014-0186-0.CrossRefGoogle Scholar
  63. Yokotani, K., Takagi, G., & Wakashima, K. (2018). Advantages of virtual agents over clinical psychologists during comprehensive mental health interviews using a mixed methods design. Computers in Human Behavior, 85, 135–145.  https://doi.org/10.1016/j.chb.2018.03.045.CrossRefGoogle Scholar
  64. Yokotani, K., & Tamura, K. (2015). Effects of personalized feedback interventions on drug-related reoffending: A pilot study. Prevention Science, 16(8), 1169–1176.  https://doi.org/10.1007/s11121-015-0571-x.CrossRefPubMedGoogle Scholar
  65. Yokotani, K., & Tamura, K. (2016). The effect of a social reintegration (parole) program on drug-related prison inmates in Japan: A 4-year prospective study. Asian Journal of Criminology.  https://doi.org/10.1007/s11417-016-9235-4.CrossRefGoogle Scholar
  66. Yoo, J.-C., & Han, T. H. (2009). Fast normalized cross-correlation. Circuits, Systems and Signal Processing, 28(6), 819.  https://doi.org/10.1007/s00034-009-9130-7.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Graduate School of Technology, Industrial and Social SciencesTokushima UniversityTokushima-shiJapan
  2. 2.Graduate School of EducationTohoku UniversitySendaiJapan

Personalised recommendations