Effects of Age-Related Cognitive Decline on Elderly User Interactions with Voice-Based Dialogue Systems

  • Masatomo KobayashiEmail author
  • Akihiro Kosugi
  • Hironobu Takagi
  • Miyuki Nemoto
  • Kiyotaka Nemoto
  • Tetsuaki Arai
  • Yasunori Yamada
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11749)


Cognitive functioning that affects user behaviors is an important factor to consider when designing interactive systems for the elderly, including emerging voice-based dialogue systems such as smart speakers and voice assistants. Previous studies have investigated the interaction behaviors of dementia patients with voice-based dialogue systems, but the extent to which age-related cognitive decline in the non-demented elderly influences the user experiences of modern voice-based dialogue systems remains uninvestigated. In this work, we conducted an empirical study in which 40 healthy elderly participants performed tasks on a voice-based dialogue system. Analysis showed that cognitive scores assessed by neuropsychological tests were significantly related to vocal characteristics, such as pauses and hesitations, as well as to behavioral differences in error-handing situations, such as when the system failed to recognize the user’s intent. On the basis of the results, we discuss design implications towards the tailored design of voice-based dialogue systems for ordinary older adults with age-related cognitive decline.


Voice-based interactions Smart speakers Voice assistants Aging Age-related cognitive decline 



We thank all of the participants in the experiment.


  1. 1.
    López, G., Quesada, L., Guerrero, L.A.: Alexa vs Siri vs Cortana vs Google Assistant: a comparison of speech-based natural user interfaces. In: Nunes, I. (ed.) International Conference on Applied Human Factors and Ergonomics, pp. 241–250. Springer, Heidelberg (2017). Scholar
  2. 2.
    Ma, M., Skubic, M., Ai, K., Hubbard, J.: July. Angel-echo: a personalized Health care application. In: Proceedings of the Second IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies, pp. 258–259. IEEE Press (2017)Google Scholar
  3. 3.
    Reis, A., Paulino, D., Paredes, H., Barroso, J.: Using intelligent personal assistants to strengthen the elderlies’ social bonds. In: Antona, M., Stephanidis, C. (eds.) UAHCI 2017. LNCS, vol. 10279, pp. 593–602. Springer, Cham (2017). Scholar
  4. 4.
    Portet, F., Vacher, M., Golanski, C., Roux, C., Meillon, B.: Design and evaluation of a smart home voice interface for the elderly: acceptability and objection aspects. Pers. Ubiquit. Comput. 17(1), 127–144 (2013)CrossRefGoogle Scholar
  5. 5.
    Wolters, M.K., Kelly, F., Kilgour, J.: Designing a spoken dialogue interface to an intelligent cognitive assistant for people with dementia. Health Inform. J. 22(4), 854–866 (2016)CrossRefGoogle Scholar
  6. 6.
    Russo, A., et al.: Dialogue Systems and Conversational Agents for Patients with Dementia: the human-robot interaction. Rejuvenation Res. 22, 109–120 (2018)CrossRefGoogle Scholar
  7. 7.
    Smith, A.L., Chaparro, B.S.: Smartphone text input method performance, usability, and preference with younger and older adults. Hum. Factors 57(6), 1015–1028 (2015)CrossRefGoogle Scholar
  8. 8.
    Bragg, D., Bennett, C., Reinecke, K., Ladner, R.: A large inclusive study of human listening rates. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, p. 444. ACM (2018)Google Scholar
  9. 9.
    Watson, C.M.: An analysis of trouble and repair in the natural conversations of people with dementia of the Alzheimer’s type. Aphasiology 13(3), 195–218 (1999)CrossRefGoogle Scholar
  10. 10.
    Rudzicz, F., Wang, R., Begum, M., Mihailidis, A.: Speech interaction with personal assistive robots supporting aging at home for individuals with Alzheimer’s disease. ACM Trans. Access. Comput. (TACCESS) 7(2), 6 (2015)Google Scholar
  11. 11.
    Bucks, R.S., Singh, S., Cuerden, J.M., Wilcock, G.K.: Analysis of spontaneous, conversational speech in dementia of Alzheimer type: evaluation of an objective technique for analysing lexical performance. Aphasiology 14(1), 71–91 (2000)CrossRefGoogle Scholar
  12. 12.
    Khodabakhsh, A., Yesil, F., Guner, E., Demiroglu, C.: Evaluation of linguistic and prosodic features for detection of Alzheimer’s disease in Turkish conversational speech. EURASIP J. Audio Speech Music. Process. 2015(1), 9 (2015)CrossRefGoogle Scholar
  13. 13.
    Hoffmann, I., Nemeth, D., Dye, C.D., Pákáski, M., Irinyi, T., Kálmán, J.: Temporal parameters of spontaneous speech in Alzheimer’s disease. Int. J. Speech-Lang. Pathol. 12(1), 29–34 (2010)CrossRefGoogle Scholar
  14. 14.
    König, A., et al.: Automatic speech analysis for the assessment of patients with predementia and Alzheimer’s disease. Alzheimer’s Dement.: Diagn., Assess. Dis. Monit. 1(1), 112–124 (2015)Google Scholar
  15. 15.
    Georgila, K., Wolters, M., Moore, J.D., Logie, R.H.: The MATCH corpus: a corpus of older and younger users’ interactions with spoken dialogue systems. Lang. Resour. Eval. 44(3), 221–261 (2010)CrossRefGoogle Scholar
  16. 16.
    Salber, D., Coutaz, J.: A wizard of Oz platform for the study of multimodal systems. In: INTERACT 1993 and CHI 1993 Conference Companion on Human Factors in Computing Systems, pp. 95–96. ACM, April 1993Google Scholar
  17. 17.
    Motti, L.G., Vigouroux, N., Gorce, P.: Interaction techniques for older adults using touchscreen devices: a literature review. In: Proceedings of the 25th Conference on l’Interaction Homme-Machine, p. 125. ACM, November 2013Google Scholar
  18. 18.
    Kobayashi, M., Hiyama, A., Miura, T., Asakawa, C., Hirose, M., Ifukube, T.: Elderly user evaluation of mobile touchscreen interactions. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011. LNCS, vol. 6946, pp. 83–99. Springer, Heidelberg (2011). Scholar
  19. 19.
    Wacharamanotham, C., Hurtmanns, J., Mertens, A., Kronenbuerger, M., Schlick, C., Borchers, J.: Evaluating swabbing: a touchscreen input method for elderly users with tremor. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 623–626, ACM 2011Google Scholar
  20. 20.
    Nicolau, H., Jorge, J.: Elderly text-entry performance on touchscreens. In: Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 127–134. ACM, October 2012Google Scholar
  21. 21.
    Sato, D., Morimura, T., Katsuki, T., Toyota, Y., Kato, T., Takagi, H.: Automated help system for novice older users from touchscreen gestures. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 3073–3078. IEEE, December 2016Google Scholar
  22. 22.
    Wobbrock, J.O., Kane, S.K., Gajos, K.Z., Harada, S., Froehlich, J.: Ability-based design: concept, principles and examples. ACM Trans. Access. Comput. (TACCESS) 3(3), 9 (2011)Google Scholar
  23. 23.
    Gajos, K.Z., Weld, D.S., Wobbrock, J.O.: Automatically generating personalized user interfaces with Supple (2010)CrossRefGoogle Scholar
  24. 24.
    Trewin, S., Keates, S., Moffatt, K.: Developing steady clicks:: a method of cursor assistance for people with motor impairments. In: Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 26–33. ACM (2006)Google Scholar
  25. 25.
    Wobbrock, J.O., Fogarty, J., Liu, S.Y.S., Kimuro, S., Harada, S.: The angle mouse: target-agnostic dynamic gain adjustment based on angular deviation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1401–1410. ACM, April 2009Google Scholar
  26. 26.
    Sato, D., Kobayashi, M., Takagi, H., Asakawa, C., Tanaka, J.: How voice augmentation supports elderly web users. In: The proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 155–162. ACM, October 2011Google Scholar
  27. 27.
    Weiner, J., Engelbart, M., Schultz, T.: Manual and automatic transcriptions in dementia detection from speech. Proc. Interspeech 2017, 3117–3121 (2017)CrossRefGoogle Scholar
  28. 28.
    Rudzicz, F., Wang, R., Begum, M., Mihailidis, A.: Speech recognition in Alzheimer’s disease with personal assistive robots. In: Proceedings of the 5th Workshop on Speech and Language Processing for Assistive Technologies, pp. 20–28 (2014)Google Scholar
  29. 29.
    Zajicek, M.: Aspects of HCI research for older people. Univ. Access Inf. Soc. 5(3), 279–286 (2006)CrossRefGoogle Scholar
  30. 30.
    Ienca, M., Fabrice, J., Elger, B., Caon, M., Pappagallo, A.S., Kressig, R.W., Wangmo, T.: Intelligent assistive technology for Alzheimer’s disease and other dementias: a systematic review. J. Alzheimers Dis. 56(4), 1301–1340 (2017)CrossRefGoogle Scholar
  31. 31.
    Granata, C., Chetouani, M., Tapus, A., Bidaud, P., Dupourqué, V.: September. Voice and graphical-based interfaces for interaction with a robot dedicated to elderly and people with cognitive disorders. In: 2010 IEEE RO-MAN, pp. 785–790. IEEE (2010)Google Scholar
  32. 32.
    Ziman, R., Walsh, G.: Factors affecting seniors’ perceptions of voice-enabled user interfaces. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, CHI EA 2018, 6 p. (2018)Google Scholar
  33. 33.
    Vipperla, R., Wolters, M., Georgila, K., Renals, S.: Speech input from older users in smart environments: challenges and perspectives. In: Stephanidis, C. (ed.) UAHCI 2009. LNCS, vol. 5615, pp. 117–126. Springer, Heidelberg (2009). Scholar
  34. 34.
    Bost, J., Moore, J.D.: An analysis of older users’ interactions with spoken dialogue systems. In: LREC, pp. 1176–1181 (2014)Google Scholar
  35. 35.
    Cucchiarini, C., Hamme, H.V., Herwijnen, O.V., Smits, F.: Jasmin-CGN: extension of the spoken Dutch corpus with speech of elderly people, children and non-natives in the human-machine interaction modality (2006)Google Scholar
  36. 36.
    Rösner, D.F., Frommer, J., Friesen, R., Haase, M., Lange, J., Otto, M.: LAST MINUTE: a multimodal corpus of speech-based user-companion interactions. In: LREC, pp. 2559–2566, May 2012Google Scholar
  37. 37.
    Rösner, D., et al.: The LAST MINUTE Corpus as a Research Resource: From Signal Processing to Behavioral Analyses in User-Companion Interactions. In Companion Technology (pp. 277–299). Springer, Cham 2017Google Scholar
  38. 38.
    Wolters, M.K., Kilgour, J., MacPherson, S.E., Dzikovska, M., Moore, J.D.: The CADENCE corpus: a new resource for inclusive voice interface design. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 3963–3966. ACM, April 2015Google Scholar
  39. 39.
    Rudzicz, F., Chan Currie, L., Danks, A., Mehta, T., Zhao, S.: Automatically identifying trouble-indicating speech behaviors in Alzheimer’s disease. In: Proceedings of the 16th International ACM SIGACCESS Conference on Computers & Accessibility, pp. 241–242. ACM, October 2014Google Scholar
  40. 40.
    Chinaei, H., Currie, L.C., Danks, A., Lin, H., Mehta, T., Rudzicz, F.: Identifying and avoiding confusion in dialogue with people with Alzheimer’s disease. Comput. Linguist. 43(2), 377–406 (2017)CrossRefGoogle Scholar
  41. 41.
    Kirshner, H.S.: Primary progressive aphasia and Alzheimer’s disease: brief history, recent evidence. Curr. Neurol. Neurosci. Rep. 12(6), 709–714 (2012)CrossRefGoogle Scholar
  42. 42.
    MacKay, D.G., James, L.E., Hadley, C.B.: Amnesic HM’s performance on the language competence test: parallel deficits in memory and sentence production. J. Clin. Exp. Neuropsychol. 30(3), 280–300 (2008)CrossRefGoogle Scholar
  43. 43.
    Van Velzen, M., Garrard, P.: From hindsight to insight–retrospective analysis of language written by a renowned Alzheimer’s patient. Interdisc. Sci. Rev. 33(4), 278–286 (2008)CrossRefGoogle Scholar
  44. 44.
    Oulhaj, A., Wilcock, G.K., Smith, A.D., de Jager, C.A.: Predicting the time of conversion to MCI in the elderly role of verbal expression and learning. Neurology 73(18), 1436–1442 (2009)CrossRefGoogle Scholar
  45. 45.
    Petersen, R.C., Caracciolo, B., Brayne, C., Gauthier, S., Jelic, V., Fratiglioni, L.: Mild cognitive impairment: a concept in evolution. J. Intern. Med. 275(3), 214–228 (2014)CrossRefGoogle Scholar
  46. 46.
    Mueller, K.D., et al.: Verbal fluency and early memory decline: results from the Wisconsin registry for Alzheimer’s prevention. Arch. Clin. Neuropsychol. 30(5), 448–457 (2015)CrossRefGoogle Scholar
  47. 47.
    Bertola, L., et al.: Graph analysis of verbal fluency test discriminate between patients with Alzheimer’s disease, mild cognitive impairment and normal elderly controls. Front. Aging Neurosci. 6, 185 (2014)Google Scholar
  48. 48.
    Lundholm, K.F., Fraser, K., Kokkinakis, D.: Automated syntactic analysis of language abilities in persons with mild and subjective cognitive impairment. Stud. Health Technol. Inform. 247, 705–709 (2018)Google Scholar
  49. 49.
    Henry, J.D., Crawford, J.R., Phillips, L.H.: Verbal fluency performance in dementia of the Alzheimer’s type: a meta-analysis. Neuropsychologia 42(9), 1212–1222 (2004)CrossRefGoogle Scholar
  50. 50.
    Kavé, G., Goral, M.: Word retrieval in connected speech in Alzheimer’s disease: a review with meta-analyses. Aphasiology 32(1), 4–26 (2018)CrossRefGoogle Scholar
  51. 51.
    Lunsford, R., Heeman, P.A.: Using linguistic indicators of difficulty to identify mild cognitive impairment. In: Sixteenth Annual Conference of the International Speech Communication Association (2015)Google Scholar
  52. 52.
    Toth, L., et al.: A speech recognition-based solution for the automatic detection of mild cognitive impairment from spontaneous speech. Curr. Alzheimer Res. 15(2), 130–138 (2018)MathSciNetCrossRefGoogle Scholar
  53. 53.
    Ahmed, S., de Jager, C.A., Haigh, A.M., Garrard, P.: Semantic processing in connected speech at a uniformly early stage of autopsy-confirmed Alzheimer’s disease. Neuropsychology 27(1), 79 (2013)CrossRefGoogle Scholar
  54. 54.
    Fraser, K.C., Meltzer, J.A., Rudzicz, F.: Linguistic features identify Alzheimer’s disease in narrative speech. J. Alzheimers Dis. 49(2), 407–422 (2016)CrossRefGoogle Scholar
  55. 55.
    Sajjadi, S.A., Patterson, K., Tomek, M., Nestor, P.J.: Abnormalities of connected speech in semantic dementia vs Alzheimer’s disease. Aphasiology 26(6), 847–866 (2012)CrossRefGoogle Scholar
  56. 56.
    Croisile, B., Ska, B., Brabant, M.J., Duchene, A., Lepage, Y., Aimard, G., Trillet, M.: Comparative study of oral and written picture description in patients with Alzheimer’s disease. Brain Lang. 53(1), 1–19 (1996)CrossRefGoogle Scholar
  57. 57.
    Ahmed, S., Haigh, A.M.F., de Jager, C.A., Garrard, P.: Connected speech as a marker of disease progression in autopsy-proven Alzheimer’s disease. Brain 136(12), 3727–3737 (2013)CrossRefGoogle Scholar
  58. 58.
    Natale, M., Entin, E., Jaffe, J.: Vocal interruptions in dyadic communication as a function of speech and social anxiety. J. Pers. Soc. Psychol. 37(6), 865 (1979)CrossRefGoogle Scholar
  59. 59.
    Brewer, R., Garcia, R.C., Schwaba, T., Gergle, D., Piper, A.M.: Exploring traditional phones as an e-mail interface for older adults. TACCESS 8(2), 6 (2016)CrossRefGoogle Scholar
  60. 60.
    Barthel Activities of Daily Living (ADL) Index: Occasional paper (Royal College of General Practitioners) (59), 24 (1993)Google Scholar
  61. 61.
    Folstein, M.F., Folstein, S.E., McHugh, P.R.: “Mini-mental state”: a practical method for grading the cognitive state of patients for the clinician. J. Psychiatr. Res. 12(3), 189–198 (1975)CrossRefGoogle Scholar
  62. 62.
    Dubois, B., Slachevsky, A., Litvan, I., Pillon, B.F.A.B.: The FAB: a frontal assessment battery at bedside. Neurology 55(11), 1621–1626 (2000)CrossRefGoogle Scholar
  63. 63.
    Wechsler, D.: A standardized memory scale for clinical use. J. Psychol. 19(1), 87–95 (1945)CrossRefGoogle Scholar
  64. 64.
    Wechsler, D.: WMS-R: Wechsler memory scale-revised: manual. Psychological Corporation (1984)Google Scholar
  65. 65.
    Reitan, R.M.: Validity of the trail making test as an indicator of organic brain damage. Percept. Mot. Skills 8(3), 271–276 (1958)CrossRefGoogle Scholar
  66. 66.
    Stuss, D.T., Levine, B.: Adult clinical neuropsychology: lessons from studies of the frontal lobes. Annu. Rev. Psychol. 53(1), 401–433 (2002)CrossRefGoogle Scholar
  67. 67.
    Alexa Design Guide. Accessed 25 Jan 2019
  68. 68.
  69. 69.
  70. 70.
    Raux, A., Eskenazi, M.: Optimizing endpointing thresholds using dialogue features in a spoken dialogue system. In: Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue (SIGdial 2008), pp. 1–10 (2008)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  • Masatomo Kobayashi
    • 1
    Email author
  • Akihiro Kosugi
    • 1
  • Hironobu Takagi
    • 1
  • Miyuki Nemoto
    • 2
  • Kiyotaka Nemoto
    • 2
  • Tetsuaki Arai
    • 2
  • Yasunori Yamada
    • 1
  1. 1.IBM ResearchTokyoJapan
  2. 2.University of TsukubaTsukubaJapan

Personalised recommendations