Passive Sensing of Affective and Cognitive Functioning in Mood Disorders by Analyzing Keystroke Kinematics and Speech Dynamics

  • Faraz Hussain
  • Jonathan P. StangeEmail author
  • Scott A. Langenecker
  • Melvin G. McInnis
  • John Zulueta
  • Andrea Piscitello
  • Bokai Cao
  • He Huang
  • Philip S. Yu
  • Peter Nelson
  • Olusola A. Ajilore
  • Alex Leow
Part of the Studies in Neuroscience, Psychology and Behavioral Economics book series (SNPBE)


Mood disorders can be difficult to diagnose, evaluate, and treat. They involve affective and cognitive components, both of which need to be closely monitored over the course of the illness. Current methods like interviews and rating scales can be cumbersome, sometimes ineffective, and oftentimes infrequently administered. Even ecological momentary assessments, when used alone, are susceptible to many of the same limitations and still require active participation from the subject. Passive, continuous, frictionless, and ubiquitous means of recording and analyzing mood and cognition obviate the need for more frequent and lengthier doctor’s visits, can help identify misdiagnoses, and would potentially serve as an early warning system to better manage medication adherence and prevent hospitalizations. Activity trackers and smartwatches have long provided exactly such a tool for evaluating physical fitness. What if smartphones, voice assistants, and eventually Internet of Things devices and ambient computing systems could similarly serve as fitness trackers for the brain, without imposing any additional burden on the user? In this chapter, we explore two such early approaches—an in-depth analytical technique based on examining meta-features of virtual keyboard usage and corresponding typing kinematics, and another method which analyzes the acoustic features of recorded speech—to passively and unobtrusively understand mood and cognition in people with bipolar disorder. We review innovative studies that have used these methods to build mathematical models and machine learning frameworks that can provide deep insights into users’ mood and cognitive states. We then outline future research considerations and close by discussing the opportunities and challenges afforded by these modes of researching mood disorders and passive sensing approaches in general.



We are grateful to the Robert Wood Johnson Foundation, the Prechter Bipolar Research Fund, Apple, Luminary Labs, and Sage Bionetworks, all of whom have helped enable much of the research discussed in this chapter. Jonathan P. Stange was supported by grant K23MH112769 from NIMH.


  1. Ajilore O (2018) A voice-enabled diabetes self-management program that addresses mood—the DiaBetty experience. In: American Diabetes Association’s 78th Scientific Sessions, Orlando, FL, USAGoogle Scholar
  2. Ajilore O, Vizueta N, Walshaw P et al (2015) Connectome signatures of neurocognitive abnormalities in euthymic bipolar I disorder. J Psychiatr Res 68:37–44CrossRefGoogle Scholar
  3. American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders (DSM-5®). American Psychiatric Publishing, Arlington, VA, USACrossRefGoogle Scholar
  4. Anderson K, Burford O, Emmerton L (2016) Mobile health apps to facilitate self-care: a qualitative study of user experiences. PLoS ONE 11(5):e0156164CrossRefGoogle Scholar
  5. Andreassen O, Houenou J, Duchesnay E et al (2018) 121. Biological insight from large-scale studies of bipolar disorder with multi-modal imaging and genomics. Biol Psychiat 83(9):S49–S50Google Scholar
  6. Asselbergs J, Ruwaard J, Ejdys M et al (2016) Mobile phone-based unobtrusive ecological momentary assessment of day-to-day mood: an explorative study. J Med Internet Res 18(3):e72CrossRefGoogle Scholar
  7. Avunjian N (2018) ‘Westworld’ cognition cowboy hats are a step up from a real science tool (inverse). USC Leonard Davis School of Gerontology.
  8. Balthazar P, Harri P, Prater A et al (2018) Protecting your patients’ interests in the era of big data, artificial intelligence, and predictive analytics. J Am Coll Radiol 15(3, Part B):580–586Google Scholar
  9. Banks IM (2002) Look to windward. Simon and SchusterGoogle Scholar
  10. Banks IM (2010) Surface detail. OrbitGoogle Scholar
  11. Bourne C, Aydemir Ö, Balanzá-Martínez V et al (2013) Neuropsychological testing of cognitive impairment in euthymic bipolar disorder: an individual patient data meta-analysis. Acta Psychiat Scand 128(3):149–162CrossRefGoogle Scholar
  12. Canhoto AI, Arp S (2017) Exploring the factors that support adoption and sustained use of health and fitness wearables. J Mark Manag 33(1–2):32–60CrossRefGoogle Scholar
  13. Cao B, Zheng L, Zhang C et al (2017) Deepmood: modeling mobile phone typing dynamics for mood detection. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 747–755Google Scholar
  14. Cho K, van Merrienboer B, Gulcehre C et al (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. eprint arXiv:1406.1078
  15. Chung JE, Joo HR, Fan JL et al (2018) High-density, long-lasting, and multi-region electrophysiological recordings using polymer electrode arrays. bioRxiv:242693Google Scholar
  16. Clifford C (2017) This former Google[X] exec is building a high-tech hat that she says will make telepathy possible in 8 years. This former Google[X] exec is building a high-tech hat that she says will make telepathy possible in 8 years.
  17. Cummings N, Schuller BW (2019) Advances in computational speech analysis for mobile sensing. In: Baumeister H, Montag C (eds) Mobile sensing and psychoinformatics. Berlin, Springer, pp 141–159Google Scholar
  18. Dixon-Román E (2016) Algo-Ritmo: more-than-human performative acts and the racializing assemblages of algorithmic architectures. Cult Studies? Crit Methodol 16(5):482–490Google Scholar
  19. Durstewitz D, Koppe G, Meyer-Lindenberg A (2019) Deep neural networks in psychiatry. Mol PsychiatryGoogle Scholar
  20. Ebner-Priemer UW, Eid M, Kleindienst N et al (2009) Analytic strategies for understanding affective (in)stability and other dynamic processes in psychopathology. J Abnorm Psychol 118(1):195–202CrossRefGoogle Scholar
  21. Ebner-Priemer UW, Trull TJ (2009) Ecological momentary assessment of mood disorders and mood dysregulation. Psychol Assess 21(4):463–475CrossRefGoogle Scholar
  22. Feng CH (2018) How a smartwatch literally saved this man’s life and why he wants more people to wear one. South China Morning Post.
  23. Fu T-M, Hong G, Zhou T et al (2016) Stable long-term chronic brain mapping at the single-neuron level. Nat Methods 13:875CrossRefGoogle Scholar
  24. Gideon J, Provost EM, McInnis M (2016) Mood state prediction from speech of varying acoustic quality for individuals with bipolar disorder. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), 20–25 March 2016, pp 2359–2363Google Scholar
  25. Global Burden of Disease Collaborative Network (2017) Global Burden of Disease study 2016 (GBD 2016) results. Institute for Health Metrics and Evaluation (IHME) Seattle, United StatesGoogle Scholar
  26. Hou L, Bergen SE, Akula N et al (2016) Genome-wide association study of 40,000 individuals identifies two novel loci associated with bipolar disorder. Hum Mol Genet 25(15):3383–3394CrossRefGoogle Scholar
  27. Huang H, Cao B, Yu PS et al (2018) dpMood: exploiting local and periodic typing dynamics for personalized mood prediction. Paper presented at the IEEE International Conference on Data MiningGoogle Scholar
  28. Ikeda M, Takahashi A, Kamatani Y et al (2017) A genome-wide association study identifies two novel susceptibility loci and trans population polygenicity associated with bipolar disorder. Mol Psychiatr 23:639CrossRefGoogle Scholar
  29. Jepsen ML (2017) Open Water Internet Inc. Optical imaging of diffuse medium. U.S. Patent No. 9,730,649,Google Scholar
  30. Karam ZN, Provost EM, Singh S et al (2014) Ecologically valid long-term mood monitoring of individuals with bipolar disorder using speech. In: 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), 4–9 May 2014, pp 4858–4862Google Scholar
  31. Khorram S, Gideon J, McInnis MG et al (2016) Recognition of depression in bipolar disorder: leveraging cohort and person-specific knowledge. In: INTERSPEECHGoogle Scholar
  32. Khorram S, Jaiswal M, Gideon J et al (2018) The PRIORI emotion dataset: linking mood to emotion detected in-the-wild. ArXiv e-printsGoogle Scholar
  33. Kubiak T, Smyth JM (2019) Connecting domains—ecological momentary assessment in a mobile sensing framework. In: Baumeister H, Montag C (eds) Mobile sensing and psychoinformatics. Berlin, Springer, pp 201–207Google Scholar
  34. Leow A, Ajilore O, Zhan L et al (2013) Impaired inter-hemispheric integration in bipolar disorder revealed with brain network analyses. Biol Psychiat 73(2):183–193CrossRefGoogle Scholar
  35. Lovatt M, Holmes J (2017) Digital phenotyping and sociological perspectives in a Brave New World. Addiction (Abingdon, England) 112(7):1286–1289CrossRefGoogle Scholar
  36. Martinez-Martin N, Kreitmair K (2018) Ethical issues for direct-to-consumer digital psychotherapy apps: addressing accountability, data protection, and consent. JMIR Ment Health 5(2):e32–e32CrossRefGoogle Scholar
  37. McInnis M, Gideon J, Mower Provost E (2017) Digital Phenotyping in bipolar disorder. Eur Neuropsychopharm 27:S440CrossRefGoogle Scholar
  38. Messner E-M, Probst T, O’Rourke T et al (2019) mHealth applications: potentials, limitations, current quality and future directions. In Baumeister H, Montag C (eds) Mobile sensing and psychoinformatics. Berlin, SpringerGoogle Scholar
  39. Montag C, Markowetz A, Blaszkiewicz K et al (2017) Facebook usage on smartphones and gray matter volume of the nucleus accumbens. Behav Brain Res 329:221–228Google Scholar
  40. Muthukrishna M, Henrich J (2019) A problem in theory. Nat Hum BehavGoogle Scholar
  41. National Collaborating Centre for Mental Health (2018) Bipolar disorder: the NICE guideline on the assessment and management of bipolar disorder in adults, children and young people in primary and secondary care. In: British Psychological Society, pp 39–40Google Scholar
  42. Perlow J (2018) How Apple watch saved my life. ZDNet.
  43. Phillips ML, Kupfer DJ (2013) Bipolar disorder diagnosis: challenges and future directions. Lancet 381(9878):1663–1671CrossRefGoogle Scholar
  44. Phillips ML, Ladouceur CD, Drevets WC (2008) A neural model of voluntary and automatic emotion regulation: implications for understanding the pathophysiology and neurodevelopment of bipolar disorder. Mol Psychiatr 13:833CrossRefGoogle Scholar
  45. Rabbi M, Klasnja P, Choudhury T et al (2019) Optimizing mHealth interventions with a bandit. In: Baumeister H, Montag C (eds) Mobile sensing and psychoinformatics. Berlin, Springer, pp 277–291Google Scholar
  46. Samzelius J (2016) Neurametrix Inc. System and method for continuous monitoring of central nervous system diseases. U.S. Patent No. 15,166,064,Google Scholar
  47. Sanford K (2018) Will this “neural lace” brain implant help us compete with AI?
  48. Sariyska R, Rathner E-M, Baumeister H et al (2018) Feasibility of linking molecular genetic markers to real-world social network size tracked on smartphones. Front Neurosci 12(945)Google Scholar
  49. Shropshire C (2015) Americans prefer texting to talking, report says. Chicago Tribune.
  50. Stange JP, Zulueta J, Langenecker SA et al (2018) Let your fingers do the talking: passive typing instability predicts future mood outcomes. Bipolar Disord 20(3):285–288CrossRefGoogle Scholar
  51. Steel Z, Marnane C, Iranpour C et al (2014) The global prevalence of common mental disorders: a systematic review and meta-analysis 1980–2013. 43(2):476–493Google Scholar
  52. Sun L, Wang Y, Cao B et al (2017) Sequential keystroke behavioral biometrics for mobile user identification via multi-view deep learning. Paper presented at the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 01 November 2017Google Scholar
  53. Turakhia MP (2018) Moving from big data to deep learning—the case of atrial fibrillation. JAMA Cardiol 3(5):371–372CrossRefGoogle Scholar
  54. Turakhia MP, Desai M, Hedlin H et al (2019) Rationale and design of a large-scale, app-based study to identify cardiac arrhythmias using a smartwatch: the Apple heart study. Am Heart J 207:66–75CrossRefGoogle Scholar
  55. Wolkenstein L, Bruchmuller K, Schmid P et al (2011) Misdiagnosing bipolar disorder—do clinicians show heuristic biases? J Affect Disorders 130(3):405–412CrossRefGoogle Scholar
  56. Zulueta J, Piscitello A, Rasic M et al (2018) Predicting mood disturbance severity with mobile phone keystroke metadata: a biaffect digital phenotyping study. J Med Internet Res 20(7):e241CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Faraz Hussain
    • 1
  • Jonathan P. Stange
    • 2
    Email author
  • Scott A. Langenecker
    • 3
  • Melvin G. McInnis
    • 4
  • John Zulueta
    • 1
  • Andrea Piscitello
    • 5
  • Bokai Cao
    • 6
  • He Huang
    • 7
  • Philip S. Yu
    • 7
  • Peter Nelson
    • 8
  • Olusola A. Ajilore
    • 1
  • Alex Leow
    • 1
  1. 1.Collaborative Neuroimaging Environment for ConnectomicsUniversity of IllinoisChicagoUSA
  2. 2.Cognition and Affect Regulation LabUniversity of IllinoisChicagoUSA
  3. 3.University Neuropsychiatric InstituteUniversity of UtahSalt Lake CityUSA
  4. 4.Heinz C. Prechter Bipolar Research ProgramUniversity of MichiganAnn ArborUSA
  5. 5.Department of Electronics, Information and BioengineeringPolitecnico di MilanoMilanItaly
  6. 6.Video Understanding TeamApplied Machine Learning, FacebookMenlo ParkUSA
  7. 7.Department of Computer ScienceUniversity of IllinoisChicagoUSA
  8. 8.College of Engineering, University of IllinoisChicagoUSA

Personalised recommendations