Skip to main content
SpringerLink
Log in

Evaluating the User Experience of Human–Robot Interaction

  • Chapter
  • First Online:
Human-Robot Interaction

Part of the book series: Springer Series on Bio- and Neurosystems ((SSBN,volume 12))

Abstract

For social robots, like in all other digitally interactive systems, products, services, and devices, positive user experience (UX) is necessary in order to achieve the intended benefits and societal relevance of human–robot interaction (HRI). The experiences that humans have when interacting with robots have the power to enable, or disable, the robots’ acceptance rate and utilization in society. For a commercial robot product, it is the achieved UX in the natural context when fulfilling its intended purpose that will determine its success. The increased number of socially interactive robots in human environments and their level of participation in everyday activities obviously highlights the importance of systematically evaluating the quality of the interaction from a human-centered perspective. There is also a need for robot developers to acquire knowledge about proper UX evaluation, both in theory and in practice. In this chapter we are asking: What is UX evaluation? Why should UX evaluation be performed? When is it appropriate to conduct a UX evaluation? How could a UX evaluation be carried out? Where could UX evaluation take place? Who should perform the UX evaluation and for whom? The aim is to briefly answer these questions in the context of doing UX evaluation in HRI, highlighting evaluation processes and methods that have methodological validity and reliability as well as practical applicability. We argue that each specific HRI project needs to take the UX perspective into account during the whole development process. We suggest that a more diverse use of methods in HRI will benefit the field, and the future users of social robots will benefit even more.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Dautenhahn, K.: Some brief thoughts on the past and future of human–robot interaction. ACM Trans. Hum. Robot. Interact. 7(1, Article 4), 3 (2018)

    Google Scholar 

  2. Dautenhahn, K.: Socially intelligent robots: dimensions of human–robot interaction. Phil. Trans. R. Soc. B 362(1480), 679–704 (2007)

    Article  Google Scholar 

  3. Dautenhahn, K.: Methodology & themes of human–robot interaction: a growing research field. Int. J. Adv. Robot. Syst. 4(1), 103–108 (2007)

    Google Scholar 

  4. Hartson, H.R., Pyla, P.S.: The UX book: Agile UX design for quality user experience. Morgan Kaufmann, Amsterdam (2018)

    Google Scholar 

  5. Hassenzahl, M.: User experience and experience design. In: Soegaard, M., Dam, R.F. (eds.) The Encyclopedia of Human–Computer Interaction, 2nd edn. The Interaction Design Foundation, Aarhus, Denmark (2013). Accessed from: http://www.interaction-design.org/encyclopedia/user_experience_and_experience_design.html

  6. Hassenzahl, M., Tractinsky, N.: User experience—a research agenda. Behav. Inf. Technol. 25(2), 91–97 (2006)

    Article  Google Scholar 

  7. Weiss, A., Bernhaupt, R., Yoshida, E.: Addressing user experience and societal impact in a user study with a humanoid robot. In: Proceedings of the Symposium on New Frontiers in Human–Robot Interaction, AISB2009, pp. 150–157 (2009)

    Google Scholar 

  8. Anderson, J., McRee, J., Wilson, R., The Effective UI Team: Effective UI. O’Reilly, Sebastopol, CA (2010)

    Google Scholar 

  9. ISO DIS 9241–210: Ergonomics of human system interaction—part 210: human-centred design for interactive systems. International Organization for Standardization, Switzerland (2019). Accessed from: https://www.iso.org/obp/ui/#iso:std:iso:9241:-210:ed-2:v1:en

  10. Gould, J.D., Lewis, C.: Designing for usability: key principles and what designers think. Commun. ACM 28(3), 300–311 (1985)

    Article  Google Scholar 

  11. Powers, A.: What robotics can learn from HCI. Interactions 15(2), 67–69 (2008)

    Article  Google Scholar 

  12. About YuMi at ABB. Accessed from: http://www.abb.se/cawp/seitp202/f1347b3f51420722c1257ec2003dd739.aspx?_ga=2.214128350.817155711.1528981398-1202336802.1528981398

  13. Alenljung, B., Lindblom, J.: User experience of socially interactive robots: its role and relevance. In: Vallverdú, J. (ed.) Synthesizing Human Emotion in Intelligent Systems and Robotics, pp. 352–364. IGI Global, Hershey, Pennsylvania, USA (2015)

    Chapter  Google Scholar 

  14. Alenljung, B., Lindblom, J., Andreasson, R., Ziemke, T.: User experience in social human–robot interaction. Int. J. Ambient Comput. Intell. 8(1), 13–32 (2017)

    Google Scholar 

  15. Lindblom, J., Andreasson, R.: Current challenges for UX evaluation of human–robot interaction. In: Schlick, C., Trzcieliński, S. (eds.) Advances in Ergonomics of Manufacturing: Managing the Enterprise of the Future. Advances in Intelligent Systems and Computing, vol. 490, pp. 267–278. Springer International Publishing, Switzerland (2016)

    Google Scholar 

  16. Bartneck, C., Kulić, D., Croft, E., Zoghbi, S.: Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int. J. Soc. Rob. 1(1), 71–81 (2009)

    Article  Google Scholar 

  17. Boden, M., Bryson, J., Caldwell, D., Dautenhahn, K., Edwards, L., Kember, S., Newman, P., Parry, V., Pegman, G., Rodden, T., Sorrell, T., Wallis, M., Whitby, B., Winfield, A.: Principles of robotics: regulating robots in the real world. Connection Sci. 29(2), 124–129 (2017)

    Article  Google Scholar 

  18. Goodrich, M.A., Schultz, A.C.: Human–robot interaction: a survey. Found. Trends Hum. Comput. Interact. 1(3), 203–275 (2007)

    Article  Google Scholar 

  19. Dautenhahn, K.: 2013. Human–Robot Interaction. In: Soegaard, M., Dam, R.F. (eds.) The Encyclopedia of Human–Computer Interaction, 2nd edn. The Interaction Design Foundation, Aarhus, Denmark. Accessed from: http://www.interaction-design.org/encyclopedia/human-robot_interaction.html

  20. Thrun, S.: Toward a framework for human–robot interaction. Hum. Comput. Interact. 19(1), 9–24 (2004)

    Article  Google Scholar 

  21. Yanco, H.A., Drury, J.: Classifying human–robot interaction: an updated taxonomy. In: IEEE International Conference on Systems, Man and Cybernetics 2004, vol. 3, pp. 2841–2846 (2004)

    Google Scholar 

  22. Alenljung, B., Andreasson, R., Billing, E.A., Lindblom, J., Lowe, R.: User experience of conveying emotions by touch. In: Proceedings of the 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 1240–1247, Lisbon, Portugal (2017)

    Google Scholar 

  23. Hassenzahl, M., Roto, V.: Being and doing: a perspective on user experience and its measurement. Interfaces 72, 10–12 (2007)

    Google Scholar 

  24. Keizer, S., Kastoris, P., Foster, M.E., Deshmukh, A.A., Lemon, O.: Evaluating a social multi-user interaction model using a Nao robot. In: RO-MAN: The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pp. 318–322, Edinburgh, Scotland, UK, 25–29 Aug 2014

    Google Scholar 

  25. Xu, Q., Ng, J., Tan, O., Huang, Z., Tay, B., Park, T.: Methodological issues in scenario-based evaluation of human–robot interaction. Int. J. Soc. Robot. 7(2), 279–291 (2015)

    Article  Google Scholar 

  26. Weiss, A., Bernhaupt, R., Tscheligi, M.: The USUS evaluation framework for user-centered HRI. In: Dautenhahn, K., Saunders, J. (eds.) New Frontiers in Human–Robot Interaction, pp. 89–110. John Benjamins Publishing, Amsterdam (2011)

    Chapter  Google Scholar 

  27. Dumas, J.S., Redish, J.: A Practical Guide to Usability Testing. Ablex Publishing Corporation, Norwood, NJ (1999)

    Google Scholar 

  28. Sim, D.Y.Y., Loo, C.K.: Extensive assessment and evaluation methodologies on assistive social robots for modelling human–robot interaction–a review. Inf. Sci. 301, 305–344 (2015)

    Article  Google Scholar 

  29. Lincoln, Y.S., Guba, E.G.: Naturalistic inquiry. Sage, Newbury Park (1985)

    Book  Google Scholar 

  30. Rohrer, C.: 2014. When to use which user-experience research methods. Nielsen Norman Group. https://www.nngroup.com/articles/which-ux-research-methods/. Accessed 2 Sept 2019

  31. Patton, M.Q.: Qualitative Research and Evaluation Methods, 3rd edn. Sage, London (2002)

    Google Scholar 

  32. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3(2), 77–101 (2006)

    Article  Google Scholar 

  33. Young, J.E., Sung, J.Y., Voida, A., Sharlin, E., Igarashi, T., Cristensen, H.I., Grinter, R.E.: Evaluating human–robot interaction: focusing on the holistic interaction experience. Int. J. Soc. Robot. 3, 53–67 (2011)

    Article  Google Scholar 

  34. Schackel, B.: Usability—context, framework, definition, design and evaluation. Interact. Comput. 21, 339–346 (2009)

    Article  Google Scholar 

  35. Blandford, A.E., Hyde, J.K., Green, T.R.G., Connell, I.: Scoping analytical usability evaluation methods: a case study. Hum. Comput. Interact. 23, 278–327 (2008)

    Article  Google Scholar 

  36. Benyon, D.: Designing User Experience: A Guide to HCI, UX and Interaction Design, 4th edn. Pearson, Harlow, England (2019)

    Google Scholar 

  37. Nielsen, J., Mack, R.L. (eds.): Usability Inspection Methods. Wiley, New York (1994)

    Google Scholar 

  38. Nielsen, J.: Heuristic evaluation. In: Nielsen, J., Mack, R.L. (eds.) Usability Inspection Methods, pp. 25–62. Wiley, New York (1994)

    Google Scholar 

  39. Clarkson, E., Arkin, R.C.: Applying heuristic evaluation to human–robot interaction systems. In: FLAIRS Conference, pp. 44–49, Key West, Florida, USA (2007)

    Google Scholar 

  40. Weiss, A., Wurhofer, D., Bernhaupt, R., Altmaninger, M., Tscheligi, M.: A methodological adaptation for heuristic evaluation of HRI. In: RO-MAN 2010: Proceedings of the 19th IEEE International Symposium on Robot and Human Interactive Communication, pp. 1–6, Viareggio, Italy (2010)

    Google Scholar 

  41. Lewis, C., Polson, P., Wharton, C., Rieman, J. Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces. In: Proceedings ACM CHI’90 Conference, pp. 235–242, Seattle, WA, USA, 1–5 April 1999 (1990)

    Google Scholar 

  42. Wharton, C., Rieman, J., Lewis, C., Polson, P.: The cognitive walkthrough method: a practitioner’s guide. In: Nielsen, J., Mack, R.L. (eds.) Usability Inspection Methods, pp. 105–140. Wiley, New York (1994)

    Google Scholar 

  43. Rogers, Y.: HCI Theory: Classical, Modern, and Contemporary. Morgan & Claypool Publishers, San Rafael, CA (2012)

    Google Scholar 

  44. Andreasson, R., Alenljung, B., Billing, E., Lowe, R.: Affective touch in human–robot interaction: conveying emotion to the nao robot. Int. J. Soc. Rob. 10(4), 473–491 (2018)

    Google Scholar 

  45. Lowe, R., Andreasson, R., Alenljung, B., Lund, A., Billing, E.: Designing for a wearable affective interface for the NAO robot: a study of emotion conveyance by touch. Multimodal Technol. Interact. 2(1), 2 (2018)

    Article  Google Scholar 

  46. Alenljung, B., Lowe, R., Andreasson, R., Billing, E., Lindblom, J.: Conveying emotions by touch to the Nao robot: a user experience perspective. Multimodal Technol. Inter. 2(4), Article no. 82 (2018)

    Google Scholar 

  47. Thomas, B.: ‘Quick and dirty’ usability tests. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, I.L. (eds.) Usability Evaluation in Industry, pp. 107–114. Taylor & Francis, London (1996)

    Google Scholar 

  48. Vermeeren, A.P.O.S, Law, E.L.-C., Roto, V., Obrist, M., Hoonhout, J., Väänänen-Vainio-Mattila, K.: User experience evaluation methods: current state and development needs. In: Proceedings of the 6th Nordic Conference on Human–Computer Interaction: Extending Boundaries (NordiCHI ‘10), pp. 521–530, Reykjavik, Iceland, 16–20 Oct 2010

    Google Scholar 

  49. Bisio, A., Sciutti, A., Nori, F., Metta, G., Fadiga, L., Sandini, G., Pozzo, T.: Motor contagion during human–human and human–robot interaction. PLoS ONE 9(8) (2014). https://doi.org/10.1371/journal.pone.0106172

  50. Rogers, Y., Marshall, P.: Research in the Wild. Morgan & Claypool Publishers, San Rafael, CA (2017)

    Book  Google Scholar 

  51. Frennert, S., Eftring, H., Östlund, B.: Case report: implications of doing research on socially assistive robots in real homes. Int. J. Soc. Robot. 9(3), 401–415 (2017)

    Article  Google Scholar 

  52. Beagley, N.I.: Field-based prototyping. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, I.L. (eds.) Usability Evaluation in Industry, pp. 95–104. Taylor & Francis, London (1996)

    Google Scholar 

  53. Kujala, S., Roro, V., Väänenen-Vainio-Mattila, K., Karaponos, E., Sinnelä, A.: UX curve: a method for evaluating long-term user experience. Interact. Comput. 23, 473–483 (2011)

    Article  Google Scholar 

  54. Nielsen, J., Lyngbæk, U.: Two field studies of hypermedia usability. In: McAleese, R., Green, C. (eds.) Hypertext: State of the Art, pp. 64–72. Intellect, Oxford, England (1990)

    Google Scholar 

  55. Duh, H.B.-L., Tan, G.C.B., Chen, V.H.: Usability evaluation for mobile device: a comparison of laboratory and field test. In: MobileHCI’06, pp. 181–186. Helsinki, Finland, 12–15 Sept 2006

    Google Scholar 

  56. Kaikkonen, A., Kekäläinen, A., Cankar, M., Kallio, T., Kankainen, A.: Usability testing of mobile applications: a comparison between laboratory and field testing. J. Usability Stud. 1(1), 4–16 (2005)

    Google Scholar 

  57. Brooke, J.: SUS: a quick and dirty usability scale. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, I.L. (eds.) Usability Evaluation in Industry, pp. 189–194. Taylor & Francis, London (1996)

    Google Scholar 

  58. Hassenzahl, M., Burmester, M., Koller, F.: AttrakDiff: Ein fragebogen zur messung wahrgenommener hedonischer und pragmatischer qualität (AttrakDif: a questionnaire for the measurement of perceived hedonic and pragmatic quality). In: Proceedings of the Mensch & Computer 2003, Interaktion in Bewegung, Stuttgart (2003)

    Google Scholar 

  59. Pressman, R.S.: Software Engineering: A Practitioner’s Approach, 5th edn. McGraw Hill, London (2000)

    Google Scholar 

  60. Sutcliffe, A.: User-Centred Requirements Engineering: Theory and Practice. Springer, London (2002)

    Book  Google Scholar 

  61. Zowghi, D., Coulin, C.: Requirements elicitation: a survey of techniques, approaches, and tools. In: Aurum, A., Wohlin, C. (eds.) Engineering and Managing Software Requirements, pp. 21–46. Springer, Berlin, Germany (2005)

    Google Scholar 

  62. de Graaf, M.M.A., Allouch, S.B.: Exploring influencing variables for the acceptance of social robots. Robot. Auton. Syst. 6(12), 1476–1486 (2013)

    Article  Google Scholar 

  63. Whiteside, J.A., Bennett, J., Holtzblatt, K.: Usability engineering: our experience and evolution. In: Helander, M. (ed.) Handbook of Human–Computer Interaction, pp. 791–817. Elsevier Science, Amsterdam, The Netherlands (1988)

    Chapter  Google Scholar 

  64. Lewis, J.R.: Sample sizes for usability studies: additional considerations. Hum. Factors 36(2), 368–378 (1994)

    Article  Google Scholar 

  65. Virzi, R.A.: Refining the test phase of usability evaluation: how many subjects is enough? Hum. Factors 34(4), 457–468 (1992)

    Article  Google Scholar 

  66. Rosson, M.B., Carroll, J.M.: Scenario-based design. In: Jacko, J., Sears, A. (eds.) The Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 1032–1050. Lawrence Erlbaum Associates, Mahwah (2002)

    Google Scholar 

  67. Good, M.D., Whiteside, J.A., Wixon, D.R., Jones, S.J.: Building a user-derived interface. Commun. ACM 27(10), 1032–1043 (1984)

    Article  Google Scholar 

  68. Riek, L.D.: Wizard of Oz studies in HRI: a systematic review and new reporting guidelines. J. Hum. Rob. Interact. 1(1), 119–136 (2012)

    Article  Google Scholar 

  69. Hornbæk, K.: Current practice in measuring usability: challenges to usability studies and research. Int. J. Hum. Comput. Stud. 64(2), 79–102 (2006)

    Article  Google Scholar 

  70. Bevan, N.: What is the difference between the purpose of usability and user experience evaluation methods. In: Proceedings of the Workshop UXEM 2009 (INTERACT 2009), pp. 1–4, Uppsala, Sweden (2009)

    Google Scholar 

  71. Lavery, D., Cockton, G., Atkinson, M.P.: Comparison of evaluation methods using structured usability problem reports. Behav. Inf. Technol. 16(4–5), 246–266 (1997)

    Article  Google Scholar 

  72. Andre, T.S., Hartson, H.R., Belz, S.M., McCreary, F.A.: The user action framework: a reliable foundation for usability engineering support tools. Int. J. Hum. Comput. Stud. 54, 107–136 (2001)

    Article  Google Scholar 

  73. Barnum, C.M.: Usability testing essentials: ready, set… test!. Morgan Kaufmann, Amsterdam (2011)

    Google Scholar 

  74. https://tla.mpi.nl/tools/tla-tools/elan/

  75. Bartneck, C., Kulic, D., Croft, E., Zoghbi, S.: Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int. J. Soc. Rob. 1, 71–81 (2009)

    Article  Google Scholar 

  76. Nomura, T., Kanda, T., Suzuki, T.: Experimental investigation into influence of negative attitudes toward robots on human–robot interaction. AI & Soc. 20(2), 138–150 (2006)

    Article  Google Scholar 

Download references

Acknowledgements

Lindblom and Alenljung especially wish to thank all students that have participated in our usability and UX evaluation courses during the years, and all authors also wish to thank the participants in their conducted HRI studies. This work was supported by the Knowledge Foundation, Stockholm, under SIDUS grant agreement no. 20140220 (AIR, Action and intention recognition in human interaction with autonomous systems).

Author information

Authors and Affiliations

  1. University of Skövde, Box 408, 541 28, Skövde, Sweden

    Jessica Lindblom, Beatrice Alenljung & Erik Billing

Authors
  1. Jessica Lindblom
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Beatrice Alenljung
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Erik Billing
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jessica Lindblom .

Editor information

Editors and Affiliations

  1. University Paris 8, Saint-Denis, France

    Céline Jost

  2. IUT de Vannes—Département STID, University of South Brittany, Vannes, France

    Brigitte Le Pévédic

  3. IDLab—imec—ELIS, Ghent, Belgium

    Tony Belpaeme

  4. Social, Therapeutic, and Robotic Systems Lab, Mississippi State University, Mississippi State, MS, USA

    Cindy Bethel

  5. Robotics and Automation Group, Aalborg University, Aalborg, Denmark

    Dimitrios Chrysostomou

  6. Research and Knowledge Exchange, Faculty of Technology, Design and Environment, Oxford Brookes University, Oxford, UK

    Nigel Crook

  7. Laboratoire Ethologie Animale et Humaine, Université Rennes 1, Paimpont, France

    Marine Grandgeorge

  8. Center for Human-Computer Interaction, University of Salzburg, Salzburg, Austria

    Nicole Mirnig

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Lindblom, J., Alenljung, B., Billing, E. (2020). Evaluating the User Experience of Human–Robot Interaction. In: Jost, C., et al. Human-Robot Interaction. Springer Series on Bio- and Neurosystems, vol 12. Springer, Cham. https://doi.org/10.1007/978-3-030-42307-0_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-42307-0_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-42306-3

  • Online ISBN: 978-3-030-42307-0

  • eBook Packages: Social SciencesSocial Sciences (R0)

Publish with us

Policies and ethics

Search

Navigation