Skip to main content
SpringerLink
Log in

A New Approach to Measure User Experience with Voice-Controlled Intelligent Assistants: A Pilot Study

  • Conference paper
  • First Online:
Human-Computer Interaction. Multimodal and Natural Interaction (HCII 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12182))

Included in the following conference series:

Abstract

Voice-controlled intelligent assistants use a conversational user interface (CUI), a system that relies on natural language processing and artificial intelligence to have verbal interactions with end-users. In this research, we propose a multi-method approach to assess user experience with a smart voice assistant through triangulation of psychometric and psychophysiological measures. The approach aims to develop a richer understanding of what the users experience during the interaction, which could provide new insights to researchers and developers in the field of voice assistant. We apply this new approach in a pilot study, and we show that each method captures a part of emotional variance during the interaction. Results suggest that emotional valence is better captured with psychometric measures, whereas arousal is better detected with psychophysiological measures.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Agourram, H., Alvarez, J., Sénécal, S., Lachize, S., Gagné, J., Léger, P.-M.: The relationship between technology self-efficacy beliefs and user satisfaction – user experience perspective. In: Kurosu, M. (ed.) HCII 2019. LNCS, vol. 11568, pp. 389–397. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22636-7_29

    Chapter  Google Scholar 

  • Beauchesne, A., et al.: User-centered gestures for mobile phones: exploring a method to evaluate user gestures for UX designers. In: Marcus, A., Wang, W. (eds.) HCII 2019. LNCS, vol. 11584, pp. 121–133. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23541-3_10

    Chapter  Google Scholar 

  • Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25(1), 49–59 (1994)

    Article  Google Scholar 

  • Braithwaite, J.J., Watson, D.G., Jones, R., Rowe, M.: A guide for analyzing electrodermal activity (EDA) & skin conductance responses (SCRs) for psychological experiments. Psychophysiology 49(1), 1017–1034 (2013)

    Google Scholar 

  • Brocke, J.V., Riedl, R., Léger, P.M.: Application strategies for neuroscience in information systems design science research. J. Comput. Inf. Syst. 53(3), 1–13 (2013)

    Google Scholar 

  • Brown, J.D.: What issues affect Likert-scale questionnaire formats. Shiken JALT Test. Eval. SIG Newslett. 4(1), 27–30 (2000)

    MathSciNet  Google Scholar 

  • Burns, N., Grove, S.K.: The Practice of Nursing Research: Conduct, Critique and Utilization. Elsevier, Amsterdam (2005)

    Google Scholar 

  • Bethel, C.L., Salomon, K., Murphy, R.R., Burke, J.L.: Survey of psychophysiology measurements applied to human-robot interaction. In: 16th IEEE International Symposium on Robot & Human Interactive Communication (2007)

    Google Scholar 

  • Nass, C.: Wired for Speech: How Voice Activates and Advances the Human-Computer Relationship. MIT Press, Cambridge (2005)

    Google Scholar 

  • Cohn, J.F., Kanade, T.: Use of automated facial image analysis for measurement of emotion expression. In: Handbook of Emotion Elicitation and Assessment, pp. 222–238 (2007)

    Google Scholar 

  • Courtemanche, F., Léger, P.-M., Dufresne, A., Fredette, M., Labonté-LeMoyne, É., Sénécal, S.: Physiological heatmaps: a tool for visualizing users’ emotional reactions. Multimedia Tools Appl. 77(9), 11547–11574 (2018)

    Article  Google Scholar 

  • Danner, L., Sidorkina, L., Joechl, M., Duerrschmid, K.: Make a face! Implicit and explicit measurement of facial expressions elicited by orange juices using face reading technology. Food Qual. Prefer. 32, 167–172 (2014)

    Article  Google Scholar 

  • De Guinea, A.O., Titah, R., Leger, P.M.: Explicit and implicit antecedents of users’ behavioral beliefs in information systems: a neuropsychological investigation. J. Manag. Inf. Syst. 30(4), 179–210 (2014)

    Article  Google Scholar 

  • Guinea, D., Ortiz, A., Titah, R., Léger, P.-M.: Measure for measure: a two study multi-trait multi-method investigation of construct validity in IS research. Comput. Hum. Behav. 29(3), 833–844 (2013)

    Article  Google Scholar 

  • De Singly, F.: Le questionnaire, 4th edn. Armand Colin, Paris (2016)

    Google Scholar 

  • Dirican, A.C., Göktürk, M.: Psychophysiological measures of human cognitive states applied in human-computer interaction. Procedia Comput. Sci. 3, 1361–1367 (2011)

    Article  Google Scholar 

  • Easwara Moorthy, A., Vu, K.-P.L.: Privacy concerns for use of voice activated personal assistant in the public space. Int. J. Hum.-Comput. Interact. 31(4), 307–335 (2015)

    Article  Google Scholar 

  • Ekman, P.: Facial expression and emotion. Am. Psychol. 48(4), 384 (1993)

    Article  Google Scholar 

  • Ekman, P., Keltner, D.: Universal facial expressions of emotion. In: Segerstrale, U., Molnar, P. (eds.) Nonverbal Communication: Where Nature Meets Culture, pp. 27–46. Lawrence Erlbaum Associates, Inc., Mahwah (1997)

    Google Scholar 

  • Fonteyn, M.E., Kuipers, B., Grobe, S.J.: A description of think aloud method and protocol analysis. Qual. Health Res. 3(4), 430–441 (1993)

    Article  Google Scholar 

  • Georges, V., Courtemanche, F., Sénécal, S., Léger, P.M., Nacke, L., Pourchon, R.: The adoption of psychophysiological measures as an evaluation tool in UX. In: International Conference on HCI in Business, Government, and Organizations, pp. 90–98. Springer, Cham, July 2017

    Google Scholar 

  • Giroux-Huppé, C., Sénécal, S., Fredette, M., Chen, S.L., Demolin, B., Léger, P.-M.: Identifying psychophysiological pain points in the online user journey: the case of online grocery. In: Marcus, A., Wang, W. (eds.) HCII 2019. LNCS, vol. 11586, pp. 459–473. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23535-2_34

    Chapter  Google Scholar 

  • Jiang, J., et al.: Automatic online evaluation of intelligent assistants. In: Proceedings of the 24th International Conference on World Wide Web - WWW 2015. Presented at the 24th International Conference (2015). https://doi.org/10.1145/2736277.2741669

  • Lamontagne, C., et al.: User test: how many users are needed to find the psychophysiological pain points in a journey map? In: Ahram, T., Taiar, R., Colson, S., Choplin, A. (eds.) IHIET 2019. AISC, vol. 1018, pp. 136–142. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-25629-6_22

    Chapter  Google Scholar 

  • Lane, R.D., Chua, P.M., Dolan, R.J.: Common effects of emotional valence, arousal and attention on neural activation during visual processing of pictures. Neuropsychologia 37(9), 989–997 (1999)

    Article  Google Scholar 

  • Lau, J., Zimmerman, B., Schaub, F.: Alexa, are you listening? Privacy perceptions, concerns and privacy-seeking behaviors with smart speakers. In: Proceedings of the ACM on Human-Computer Interaction, vol. 2, no. CSCW, pp. 1–311 (2018)

    Google Scholar 

  • Léger, P.-M., Courtemanche, F., Fredette, M., Sénécal, S.: A cloud-based lab management and analytics software for triangulated human-centered research. In: Davis, F.D., Riedl, R., vom Brocke, J., Léger, P.-M., Randolph, A.B. (eds.) Information Systems and Neuroscience. LNISO, vol. 29, pp. 93–99. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-01087-4_11

    Chapter  Google Scholar 

  • Léger, P.M.,Davis, F.D., Cronan, T.P., Perret, J.:Neuropsychophysiological correlates of cognitive absorption in an enactive training context. Comput. Hum. Behav. 34, 273–283 (2014)

    Google Scholar 

  • Léger, P.-M., Charland, P., Sénécal, S., Cyr, S.: Predicting properties of cognitive pupillometry in human–computer interaction: a preliminary investigation. In: Davis, F.D., Riedl, R., vom Brocke, J., Léger, P.-M., Randolph, A.B. (eds.) Information Systems and Neuroscience. LNISO, vol. 25, pp. 121–127. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-67431-5_14

    Chapter  Google Scholar 

  • Lopatovska, I., Oropeza, H.: User interactions with “Alexa” in public academic space. Proc. Assoc. Inf. Sci. Technol. 55(1), 309–318 (2018). https://doi.org/10.1002/pra2.2018.14505501034

    Article  Google Scholar 

  • Lopatovska, I., Williams, H.: Personification of the Amazon Alexa: BFF or a mindless companion. In: Proceedings of the 2018 Conference on Human Information Interaction & Retrieval, pp. 265–268. ACM, March 2018

    Google Scholar 

  • Lourties, S., Léger, P.-M., Sénécal, S., Fredette, M., Chen, S.L.: Testing the convergent validity of continuous self-perceived measurement systems: an exploratory study. In: Nah, F.F.-H., Xiao, B.S. (eds.) HCIBGO 2018. LNCS, vol. 10923, pp. 132–144. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91716-0_11

    Chapter  Google Scholar 

  • Maunier, B., et al.: Keep calm and read the instructions: factors for successful user equipment setup. In: Nah, F.F.-H., Xiao, B.S. (eds.) HCIBGO 2018. LNCS, vol. 10923, pp. 372–381. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91716-0_29

    Chapter  Google Scholar 

  • Dawson, M.E., Schell, A.M., Filion, D.L.: The electrodermal system. In: Cacioppo, J.T., Tassinary, L.G., Berntson, G.G. (eds.) Handbook of Psychophysiology, vol. 2. Cambridge University Press, Cambridge (2000)

    Google Scholar 

  • Nicholl, H.: Diaries as a method of data collection in research. Paediatr. Care 22(7), 16–20 (2010). https://doi.org/10.7748/paed2010.09.22.7.16.c7948

    Article  Google Scholar 

  • Noldus FaceReader methodology. https://info.noldus.com/free-white-paper-on-facereader-methodology

  • Petrock, V.: Voice Assistant Use Reaches Critical Mass. Retrieved from e-Marketer database, 15 August 2019

    Google Scholar 

  • Piedmont, R.L.: Social desirability bias. In: Encyclopedia of Quality of Life and Well-Being Research, pp. 6036–6037 (2014). https://doi.org/10.1007/978-94-007-0753-5_2746

  • Riedl, R., Fischer, T., Léger, P.-M., Davis, F.: A decade of NeuroIS research: progress, challenges, and future directions. Data Base Adv. Inf. Syst. 51 (2020, in press)

    Google Scholar 

  • Rowe, D.W., Sibert, J., Irwin, D.: Heart rate variability: indicator of user state as an aid to human-computer interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 480–487. ACM Press/Addison-Wesley Publishing Co, January 1998

    Google Scholar 

  • Sciuto, A., Saini, A., Forlizzi, J., Hong, J.I.: Hey Alexa, what’s up? In: Proceedings of the 2018 on Designing Interactive Systems Conference 2018 - DIS 2018. Presented at the 2018 (2018). https://doi.org/10.1145/3196709.3196772

  • Sirois, S., Brisson, J.: Pupillometry. Wiley Interdisc. Rev. Cogn. Sci. 5(6), 679–692 (2014)

    Article  Google Scholar 

  • Tung, L.: Amazon: We sold tens of millions of Echo devices in 2018, and Alexa has now 70 000 skills. Retrieved from ZDnet database, 20 December 2018

    Google Scholar 

  • Vom Brocke, J., Hevner, A., Léger, P.M., Walla, P., Riedl, R.: Advancing a NeuroIS research agenda with four areas of societal contributions. Eur. J. Inf. Syst. 29, 9–24 (2020)

    Google Scholar 

  • Xiong, J., Zuo, M.: What does existing NeuroIS research focus on? Inf. Syst. 89, 101462 (2020). https://doi.org/10.1016/j.is.2019.101462

    Article  Google Scholar 

  • Zaman, B., Shrimpton-Smith, T.: The FaceReader: measuring instant fun of use. In: Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles, Chicago, pp. 457–460. ACM, October 2006

    Google Scholar 

  • D'Mello, S., Calvo, R.A.: Beyond the basic emotions: what should affective computing compute? In: CHI 2013 Extended Abstracts on Human Factors in Computing Systems, pp. 2287–2294 (2013)

    Google Scholar 

  • Heiden, M., Lyskov, E., Djupsjöbacka, M., Hellström, F., Crenshaw, A.G.: Effects of time pressure and precision demands during computer mouse work on muscle oxygenation and position sense. Eur. J. Appl. Physiol. 94(1–2), 97–106 (2005)

    Article  Google Scholar 

  • Leite, I., Henriques, R., Martinho, C., Paiva, A.: Sensors in the wild: exploring electrodermal activity in child-robot interaction. In: 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 41–48. IEEE, March 2013

    Google Scholar 

  • Boucsein, W.: Electrodermal Activity, 2nd edn. Springer, Boston (2012). https://doi.org/10.1007/978-1-4614-1126-0

  • Russell, J.A.: Core affect and the psychological construction of emotion. Psychol. Rev. 110(1), 145 (2003)

    Google Scholar 

  • Deng, L., Poole, M.S.: Affect in web interfaces: a study of the impacts of web page visual complexity and order. MIS Q, 711–730 (2010)

    Google Scholar 

  • Dijksterhuis, A., Smith, P.K., Van Baaren, R.B., Wigboldus, D.H.: The unconscious consumer: effects of environment on consumer behavior. J. Consum. Psychol. 15(3), 193–202 (2005)

    Google Scholar 

  • Garg, R., Moreno, C.: Exploring everyday sharing practices of smart speakers. In: IUI Workshops, January 2019

    Google Scholar 

  • Porcheron, M., Fischer, J.E., Reeves, S., Sharples, S.: Voice interfaces in everyday life. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–12 April 2018

    Google Scholar 

  • Riedl, R., Léger, P.M.: Fundamentals of NeuroIS. Studies in Neuroscience, Psychology and Behavioral Economics. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-662-45091-8

Download references

Author information

Authors and Affiliations

  1. HEC Montréal, Montréal, Canada

    Félix Le Pailleur, Bo Huang, Pierre-Majorique Léger & Sylvain Sénécal

Authors
  1. Félix Le Pailleur
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pierre-Majorique Léger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sylvain Sénécal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Félix Le Pailleur .

Editor information

Editors and Affiliations

  1. The Open University of Japan, Chiba, Japan

    Masaaki Kurosu

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Le Pailleur, F., Huang, B., Léger, PM., Sénécal, S. (2020). A New Approach to Measure User Experience with Voice-Controlled Intelligent Assistants: A Pilot Study. In: Kurosu, M. (eds) Human-Computer Interaction. Multimodal and Natural Interaction. HCII 2020. Lecture Notes in Computer Science(), vol 12182. Springer, Cham. https://doi.org/10.1007/978-3-030-49062-1_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-49062-1_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-49061-4

  • Online ISBN: 978-3-030-49062-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation