Abstract
Conversations contain a wide spectrum of multimodal information that gives us hints about the emotions and moods of the speaker. In this paper, we developed a system that supports humans to analyze conversations. Our main contribution is the identification of appropriate multimodal features and the integration of such features into verbatim conversation transcripts. We demonstrate the ability of our system to take in a wide range of multimodal information and automatically generated a prediction score for the depression state of the individual. Our experiments showed that this approach yielded better performance than the baseline model. Furthermore, the multimodal narrative approach makes it easy to integrate learnings from other disciplines, such as conversational analysis and psychology. Lastly, this interdisciplinary and automated approach is a step towards emulating how practitioners record the course of treatment as well as emulating how conversational analysts have been analyzing conversations by hand.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
American Psychiatric Association, et al.: Diagnostic and statistical manual of mental disorders (DSM-5®). American Psychiatric Publishing (2013)
Jouvent, R., Widlöcher, D., et al.: Speech pause time and the retardation rating scale for depression (ERD): towards a reciprocal validation. J. Affect. Disord. 6, 123–127 (1984)
Stassen, H.H., Kuny, S., Hell, D.: The speech analysis approach to determining onset of improvement under antidepressants. Eur. Neuropsychopharmacol. 8, 303–310 (1998)
Jefferson, G.: Glossary of transcript symbols with an introduction. Pragmat. Beyond New Ser. 125, 13–34 (2004)
Antol, S., et al.: VQA: visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2425–2433 (2015)
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., Carlos Niebles, J.: Dense-captioning events in videos. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 706–715 (2017)
Gratch, J., et al.: The distress analysis interview corpus of human and computer interviews. In: LREC, pp. 3123–3128 (2014)
Ringeval, F., et al.: AVEC 2019 workshop and challenge: state-of-mind, detecting depression with AI, and cross-cultural affect recognition. arXiv prepr. arXiv:1907.11510 (2019)
Buyukdura, J.S., McClintock, S.M., Croarkin, P.E.: Psychomotor retardation in depression: biological underpinnings, measurement, and treatment. Prog. Neuro-Psychopharmacol. Biol. Psychiatry 35, 395–409 (2011)
Kotov, R., Gamez, W., Schmidt, F., Watson, D.: Linking big personality traits to anxiety, depressive, and substance use disorders: a meta-analysis. Psychol. Bull. 136, 768 (2010)
Du, L., Bakish, D., Ravindran, A.V., Hrdina, P.D.: Does fluoxetine influence major depression by modifying five-factor personality traits? J. Affect. Disord. 71, 235–241 (2002)
Angst, J., Gamma, A., Gastpar, M., Lépine, J.-P., Mendlewicz, J., Tylee, A.: Gender differences in depression. Eur. Arch. Psychiatry Clin. Neurosci. 252, 201–209 (2002)
Nolen-Hoeksema, S., Larson, J., Grayson, C.: Explaining the gender difference in depressive symptoms. J. Pers. Soc. Psychol. 77, 1061 (1999)
Fonzi, L., Matteucci, G., Bersani, G.: Laughter and depression: hypothesis of pathogenic and therapeutic correlation. Riv. Psichiatr. 45, 1–6 (2010)
Berenbaum, H., Oltmanns, T.F.: Emotional experience and expression in schizophrenia and depression. J. Abnorm. Psychol. 101, 37 (1992)
Ryokai, K., Durán López, E., Howell, N., Gillick, J., Bamman, D.: Capturing, representing, and interacting with laughter. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, p. 358 (2018)
Yang, L., Jiang, D., He, L., Pei, E., Oveneke, M.C., Sahli, H.: Decision tree based depression classification from audio video and language information. In: Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, pp. 89–96 (2016)
Ekman, P., Friesen, W.V: Manual for the facial action coding system. Consulting Psychologists Press (1978)
Cannizzaro, M., Harel, B., Reilly, N., Chappell, P., Snyder, P.J.: Voice acoustical measurement of the severity of major depression. Brain Cogn. 56, 30–35 (2004)
Gardner, M., et al.: AllenNLP: a deep semantic natural language processing platform (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Kim, J.Y., Kim, G.Y., Yacef, K. (2019). Detecting Depression in Dyadic Conversations with Multimodal Narratives and Visualizations. In: Liu, J., Bailey, J. (eds) AI 2019: Advances in Artificial Intelligence. AI 2019. Lecture Notes in Computer Science(), vol 11919. Springer, Cham. https://doi.org/10.1007/978-3-030-35288-2_25
Download citation
DOI: https://doi.org/10.1007/978-3-030-35288-2_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-35287-5
Online ISBN: 978-3-030-35288-2
eBook Packages: Computer ScienceComputer Science (R0)