Don’t Leave Me Alone: Retrospective Think Aloud Supported by Real-Time Monitoring of Participant’s Physiology

  • Alexandros LiapisEmail author
  • Christos Katsanos
  • Michalis Xenos
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10901)


Think aloud protocols are widely applied in user experience studies. In this paper, the effect of two different applications of the Retrospective Think Aloud (RTA) protocol on the number of user-reported usability issues is examined. To this end, 30 users were asked to use the National Cadastre and Mapping Agency web application and complete a set of tasks, such as measuring the land area of a square in their hometown. The order of tasks was randomized per participant. Next, participants were involved in RTA sessions. Each participant was involved in two different RTA modes: (a) the strict guidance, in which the facilitator stayed in the background and prompted participants to keep thinking aloud based on his judgement and experience, and (b) the physiology-supported interventions, in which the facilitator intervened based on real-time monitoring of user’s physiological signals. During each session, three participant’s physiological signals were recorded: skin conductance, skin temperature and blood volume pulse. Participants were also asked to provide valence-arousal ratings for each self-reported usability issue. Analysis of the collected data showed that participants in the physiology-supported RTA mode reported significantly more usability issues. No significant effect of the RTA mode was found on the valence-arousal ratings for the reported usability issues. Participants’ physiological signals during the RTA sessions did not also differ significantly between the two modes.


Human-Computer Interaction Physiological signals Usability evaluation Retrospective Think Aloud 


  1. 1.
    Nielsen, J.: Usability Engineering. Morgan Kaufmann Publishers Inc., San Francisco (1993)zbMATHGoogle Scholar
  2. 2.
    Rubin, J., Chisnell, D., Spool, J.: Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley, Indianapolis (2008)Google Scholar
  3. 3.
    Seffah, A., Donyaee, M., Kline, R.B., Padda, H.K.: Usability measurement and metrics: a consolidated model. Softw. Qual. J. 14, 159–178 (2006)CrossRefGoogle Scholar
  4. 4.
    Lazar, D.J., Feng, D.J.H., Hochheiser, D.H.: Research Methods in Human-Computer Interaction. Wiley, London (2010)Google Scholar
  5. 5.
    Tsui, W.-H., Lee, P., Hsiao, T.-C.: The effect of emotion on keystroke: an experimental study using facial feedback hypothesis. In: Conference Proceedings of IEEE Engineering in Medicine and Biology Society (EMBC 2013), pp. 2870–2873 (2013)Google Scholar
  6. 6.
    Liapis, A., Katsanos, C., Sotiropoulos, D., Xenos, M., Karousos, N.: Recognizing emotions in human computer interaction: studying stress using skin conductance. In: Abascal, J., Barbosa, S., Fetter, M., Gross, T., Palanque, P., Winckler, M. (eds.) INTERACT 2015. LNCS, vol. 9296, pp. 255–262. Springer, Cham (2015). Scholar
  7. 7.
    Mandryk, R.L., Atkins, M.S.: A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. Int. J. Hum. Comput. Stud. 65, 329–347 (2007)CrossRefGoogle Scholar
  8. 8.
    Liapis, A., Xenos, M.: The physiological measurements as a critical indicator in users’ experience evaluation. In: Proceedings of the 17th Panhellenic Conference on Informatics, pp. 258–263. ACM, New York (2013)Google Scholar
  9. 9.
    Georges, V., Courtemanche, F., Sénécal, S., Léger, P.-M., Nacke, L., Pourchon, R.: The adoption of physiological measures as an evaluation tool in UX. In: Nah, F.F.-H., Tan, C.-H. (eds.) HCIBGO 2017. LNCS, vol. 10293, pp. 90–98. Springer, Cham (2017). Scholar
  10. 10.
    Hernandez, J., Paredes, P., Roseway, A., Czerwinski, M.: Under pressure: sensing stress of computer users. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 51–60. ACM, New York (2014)Google Scholar
  11. 11.
    Wilson, G.M., Sasse, M.A.: Do users always know what’s good for them? Utilising physiological responses to assess media quality. In: McDonald, S., Waern, Y., Cockton, G. (eds.) People and Computers XIV — Usability or Else!, pp. 327–339. Springer, London (2000)CrossRefGoogle Scholar
  12. 12.
    Anderson, N.B.: Levels of analysis in health science. A framework for integrating sociobehavioral and biomedical research. Ann. N. Y. Acad. Sci. 840, 563–576 (1998)CrossRefGoogle Scholar
  13. 13.
    Diamond, D.M., Campbell, A.M., Park, C.R., Halonen, J., Zoladz, P.R.: The temporal dynamics model of emotional memory processing: a synthesis on the neurobiological basis of stress-induced amnesia, flashbulb and traumatic memories, and the Yerkes-Dodson law. Neural Plast. 2007, 60803 (2007)CrossRefGoogle Scholar
  14. 14.
    Nisbett, R., Wilson, T.: Telling more than we can know: verbal reports on mental processes. Psychol. Rev. 84, 231–259 (1977)CrossRefGoogle Scholar
  15. 15.
    Ericsson, K.A.: Protocol Analysis: Verbal Reports as Data. MIT Press, Cambridge (1993)Google Scholar
  16. 16.
    Griffiths, M.D.: The role of cognitive bias and skill in fruit machine gambling. Br. J. Psychol. 85, 351–369 (1994)CrossRefGoogle Scholar
  17. 17.
    Eger, N., Ball, L.J., Stevens, R., Dodd, J.: Cueing retrospective verbal reports in usability testing through eye-movement replay. In: Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI…But Not As We Know It, vol. 1, pp. 129–137. British Computer Society, Swinton, UK (2007)Google Scholar
  18. 18.
    Teague, R., De Jesus, K., Ueno, M.N.: Concurrent vs. post-task usability test ratings. In: Extended Abstracts on Human Factors in Computing Systems, CHI 2001, pp. 289–290. ACM, New York (2001)Google Scholar
  19. 19.
    Russo, J.E., Johnson, E.J., Stephens, D.L.: The validity of verbal protocols. Mem. Cogn. 17, 759–769 (1989)CrossRefGoogle Scholar
  20. 20.
    Peute, L.W.P., de Keizer, N.F., Jaspers, M.W.M.: The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool. J. Biomed. Inf. 55, 1–10 (2015)CrossRefGoogle Scholar
  21. 21.
    van den Haak, M., De Jong, M., Jan Schellens, P.: Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behav. Inf. Technol. 22, 339–351 (2003)CrossRefGoogle Scholar
  22. 22.
    Liapis, A., Karousos, N., Katsanos, C., Xenos, M.: Evaluating user’s emotional experience in HCI: the PhysiOBS approach. In: Kurosu, M. (ed.) HCI 2014. LNCS, vol. 8511, pp. 758–767. Springer, Cham (2014). Scholar
  23. 23.
    Russell, J.A., Weiss, A., Mendelsohn, G.A.: Affect Grid: A single-item scale of pleasure and arousal. J. Pers. Soc. Psychol. 57, 493–502 (1989)CrossRefGoogle Scholar
  24. 24.
    Field, A.: Discovering Statistics Using IBM SPSS Statistics. Sage, Thousand Oaks (2013)Google Scholar
  25. 25.
    Cohen, J.: A power primer. Psychol. Bull. 112, 155–159 (1992)CrossRefGoogle Scholar
  26. 26.
    Liapis, A., Katsanos, C., Sotiropoulos, D., Xenos, M., Karousos, N.: Subjective assessment of stress in HCI: a study of the valence-arousal scale using skin conductance. In: Proceedings of the 11th Biannual Conference on Italian SIGCHI Chapter, pp. 174–177. ACM, New York (2015)Google Scholar
  27. 27.
    Katsanos, C., Tselios, N., Karousos, N., Xenos, M.: Learning web form design by using the KLM form analyzer: a case study. In: Proceedings of the 19th Panhellenic Conference on Informatics, pp. 44–49. ACM, New York (2015)Google Scholar
  28. 28.
    Tselios, N., Altanopoulou, P., Katsanos, C.: Effectiveness of a framed wiki-based learning activity in the context of HCI education. In: 2011 15th Panhellenic Conference on Informatics, pp. 368–372 (2011)Google Scholar
  29. 29.
    Katsanos, C., Tsakoumis, A., Avouris, N.: Web accessibility: design of an educational system to support guidelines learning. In: Proceedings of the 13th Pan-Hellenic Conference on Informatics with International Participation, pp. 155–164, Corfu (2009)Google Scholar
  30. 30.
    Katsanos, C., Tselios, N., Xenos, M.: Tool-mediated HCI modeling instruction in a campus-based software quality course. In: Proceedings of the 20th International Conference on Human-Computer Interaction, Las Vegas (2018, in press)Google Scholar
  31. 31.
    Katsanos, C., Tselios, N., Tsakoumis, A., Avouris, N.: Learning about web accessibility: a project based tool-mediated approach. Educ. Inf. Technol. 17, 79–94 (2012)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Alexandros Liapis
    • 1
    Email author
  • Christos Katsanos
    • 2
    • 3
  • Michalis Xenos
    • 4
  1. 1.School of Science and TechnologyHellenic Open UniversityPatrasGreece
  2. 2.Department of InformaticsAristotle University of ThessalonikiThessalonikiGreece
  3. 3.HCI Group, Electrical and Computer Engineering DepartmentUniversity of PatrasPatrasGreece
  4. 4.Computer Engineering and Informatics DepartmentUniversity of PatrasPatrasGreece

Personalised recommendations