Guiding Usability Newcomers to Understand the Context of Use: Towards Models of Collaborative Heuristic Evaluation

  • André de Lima SalgadoEmail author
  • Flávia de Souza Santos
  • Renata Pontin de Mattos Fortes
  • Patrick C. K. Hung
Part of the International Series on Computer Entertainment and Media Technology book series (ISCEMT)


Usability inspection methods are liable to the expertise effect among distinct evaluators. Hence, understanding the difficulties faced by evaluators of low expertise (novices and newcomers) is a requirement to move forward in the field. However, the following question remains: Which of the terms that compose usability (User, Goal, Effectiveness, Efficiency, Satisfaction, Context of Use and Task) is the most difficult for newcomers to understand? This exploratory study aims to compare usability newcomers’ difficulties on understanding different terms that compose the usability, based on the definition showed by the ISO/IEC 25066. To achieve this goal, we conducted a survey with 38 usability newcomers. Observations on our survey show the Context of Use may be the most difficult term to be understood by newcomers. Thus, we suggest the adoption of scenarios, storyboards, and domain-specific principles as the basis for newcomers in HEs, when practitioners cannot count on experts for the inspection. In addition, we suggest three (3) different models of Collaborative Heuristic Evaluation aimed to provide newcomers with important insights about Context of Use. Finally, we suggest as future works to study the validity of such models.


Usability Heuristic evaluation Evaluators Novice Newcomer Expertise effect Evaluator effect 



This study was supported by the grants 2017/15239-0 and 2015/24525-0, São Paulo Research Foundation (FAPESP). We also thank to CAPES and University of São Paulo for their important support.


  1. 1.
    Boone, H. N., & Boone, D. A. (2012). Analyzing likert data. Journal of extension, 50(2), pp 1–5.Google Scholar
  2. 2.
    Brajnik, G., Yesilada, Y., & Harper, S. (2011). The expertise effect on web accessibility evaluation methods. Human–Computer Interaction, 26(3), pp 246–283. DOI: CrossRefGoogle Scholar
  3. 3.
    Bruun, A., & Stage, J. (2014). Barefoot usability evaluations. Behaviour & Information Technology, 33(11), pp 1148–1167. DOI: CrossRefGoogle Scholar
  4. 4.
    Bruun, A., & Stage, J. (2015). New approaches to usability evaluation in software development: Barefoot and crowdsourcing. Journal of Systems and Software, 105, pp 40–53. DOI: CrossRefGoogle Scholar
  5. 5.
    Bryman, A. (2015). Social research methods. Oxford university press. ISBN: 978-0-19-968945-3Google Scholar
  6. 6.
    Buykx, L. (2009). Improving heuristic evaluation through collaborative working. Masters Dissertation. The University of York Department of Computer Science.Google Scholar
  7. 7.
    Chambel, T. (2016, November). Interactive and Immersive Media Experiences. In Proceedings of the 22nd Brazilian Symposium on Multimedia and the Web, pp 1–1. ACM. DOI:
  8. 8.
    de Lima Salgado, A., & Fortes, R. P. M. (2016). Heuristic Evaluation for Novice Evaluators. In International Conference of Design, User Experience, and Usability, Springer International Publishing, pp 387–398. DOI: CrossRefGoogle Scholar
  9. 9.
    DeLone, W. H., & McLean, E. R. (1992). Information systems success: The quest for the dependent variable. Information systems research, 3(1), pp 60–95. DOI: CrossRefGoogle Scholar
  10. 10.
    Delone, W. H., & McLean, E. R. (2003). The DeLone and McLean model of information systems success: a ten-year update. Journal of management information systems, 19(4), pp 9–30. DOI: CrossRefGoogle Scholar
  11. 11.
    Demers, R. A. (1981). System design for usability. Communications of the ACM, 24(8), pp 494–501. DOI: CrossRefGoogle Scholar
  12. 12.
    Ebling, M. R., & Want, R. (2017). Pervasive Computing Revisited. IEEE Pervasive Computing, 16(3), pp 17–19. DOI: CrossRefGoogle Scholar
  13. 13.
    Ferreira, D. F. (2008). Estatística multivariada. Editora UFLA. ISBN: 978-85-87692-52-8Google Scholar
  14. 14.
    Georgsson, M., Weir, C., & Staggers, N. (2014). Revisiting heuristic evaluation methods to improve the reliability of findings. In MIE2014. IOS Press. DOI: CrossRefGoogle Scholar
  15. 15.
    Hair, J. F., Anderson, R. E., Babin, B. J. & Black, W. C. (2010). . Multivariate data analysis: A global perspective Pearson Upper Saddle River, NJ. ISBN: 0133792684Google Scholar
  16. 16.
    Hermawati, S., & Lawson, G. (2016). Establishing usability heuristics for heuristics evaluation in a specific domain: Is there a consensus? Applied ergonomics, 56, pp 34–51. DOI: CrossRefGoogle Scholar
  17. 17.
    Hertzum, M. (2017). Commentary: Usability—A Sensitizing Concept. Human–Computer Interaction, pp 1–4. DOI: CrossRefGoogle Scholar
  18. 18.
    Hertzum, M., & Jacobsen, N. E. (2001). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13(4), pp 421–443. DOI: CrossRefGoogle Scholar
  19. 19.
    Hornbæk, K. (2010). Dogmas in the assessment of usability evaluation methods. Behaviour & Information Technology, 29(1), pp 97–111. DOI: CrossRefGoogle Scholar
  20. 20.
    Huang, B. (2012). A Comparison of Remote Collaborative Heuristic Evaluation by Novices and Experts with User-based Evaluation. Masters Dissertation. The University of York Department of Computer Science.Google Scholar
  21. 21.
    Hung P.C.K., Tang J.K.T., & Kanev K. (2017) Introduction. In: Tang J., Hung P. (eds) Computing in Smart Toys. International Series on Computer Entertainment and Media Technology Springer, Cham, pp 1–5. DOI: CrossRefGoogle Scholar
  22. 22.
    ISO 25066 (2016). Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Common Industry Format (CIF) for Usability - Evaluation Report.Google Scholar
  23. 23.
    ISO 9241-210 (2010). Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems.Google Scholar
  24. 24.
    Johannessen, G. H. J., & Hornbæk, K. (2014). Must evaluation methods be about usability? Devising and assessing the utility inspection method. Behaviour & Information Technology, 33(2), pp 195–206. DOI: CrossRefGoogle Scholar
  25. 25.
    Kangas, K., Seitamaa-Hakkarainen, P., & Hakkarainen, K. (2013). Design expert’s participation in elementary students’ collaborative design process. International Journal of Technology and Design Education, 23(2), pp 161–178. DOI: CrossRefGoogle Scholar
  26. 26.
    Lewis, J. R. (2014). Usability: lessons learned… and yet to be learned. International Journal of Human-Computer Interaction, 30(9), pp 663–684. DOI: CrossRefGoogle Scholar
  27. 27.
    MacDonald, C. M., & Atwood, M. E. (2013, April). Changing perspectives on evaluation in HCI: past, present, and future. In CHI’13 Extended Abstracts on Human Factors in Computing Systems, pp 1969–1978. ACM. DOI:
  28. 28.
    MacFarlane, S., & Pasiali, A. (2005). Adapting the heuristic evaluation method for use with children. In Workshop on child computer interaction: methodological research, Interact, pp 28–31.Google Scholar
  29. 29.
    MacFarlane, S., Sim, G., & Horton, M. (2005, June). Assessing usability and fun in educational software. In Proceedings of the 2005 conference on Interaction design and children, pp 103–109. ACM.Google Scholar
  30. 30.
    Martins, A. I., Queirós, A., Silva, A. G., & Rocha, N. P. (2014). Usability evaluation methods: A systematic review. Human Factors in Software Development and Design, 250. DOI:
  31. 31.
    Molich, R., & Nielsen, J. (1990). Improving a human-computer dialogue. Communications of the ACM, 33(3), pp 338–348. DOI: CrossRefGoogle Scholar
  32. 32.
    Nielsen, J., & Molich, R. (1989). Teaching user interface design based on usability engineering. ACM SIGCHI Bulletin, 21(1), pp 45–48. DOI: CrossRefGoogle Scholar
  33. 33.
    Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 249–256. ACM. DOI:
  34. 34.
    Nielsen, J. (1995). How to conduct a heuristic evaluation. Nielsen Norman Group. URL:
  35. 35.
    Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Proceedings ACM CHI’92 Conference (Monterey, CA, May 3-7), 373–380. DOI:
  36. 36.
    Nielsen, J. (1994a, April). Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, ACM, pp 152–158. DOI:
  37. 37.
    Nielsen, J. Heuristic Evaluation: How-To: Article by Jakob Nielsen. Nielsen Norman Group Available at: (Accessed: 2nd May 2018)Google Scholar
  38. 38.
    Nielsen, J. (2002). Becoming a usability professional. Nielsen Norman Group. URL:
  39. 39.
    Nielsen, J. (2012). Usability 101: Introduction to usability. Nielsen Norman Group. URL:
  40. 40.
    Othman, M. K., Mahudin, F., Ahaguk, C. H., & Rahman, M. F. A. (2014, September). Mobile guide technologies (smartphone apps): Collaborative Heuristic Evaluation (CHE) with expert and novice users. In User Science and Engineering (i-USEr), 2014 3rd International Conference on (pp. 232–236). IEEE. DOI:
  41. 41.
    Paz, F., & Pow-Sang, J. A. (2016). A systematic mapping review of usability evaluation methods for software development process. International Journal of Software Engineering and Its Applications, 10(1), pp 165–178.CrossRefGoogle Scholar
  42. 42.
    Petrie, H., & Buykx, L. (2010). Collaborative Heuristic Evaluation: improving the effectiveness of heuristic evaluation. In Proceedings of UPA 2010 International Conference.Google Scholar
  43. 43.
    Petrie, H., & Power, C. (2012, May). What do users really care about? a comparison of usability problems found by users and experts on highly interactive websites. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, pp 2107–2116. ACM. DOI:
  44. 44.
    Petter, S., DeLone, W., & McLean, E. R. (2013). Information systems success: The quest for the independent variables. Journal of Management Information Systems, 29(4), pp 7–62. DOI: Scholar
  45. 45.
    Popovic, V. (2004). Expertise development in product design—strategic and domain-specific knowledge connections. Design Studies, 25(5), (pp. 527–545). DOI: CrossRefGoogle Scholar
  46. 46.
    Read, J. (2015). Children as participants in design and evaluation. interactions, 22(2), pp 64–66. DOI: Scholar
  47. 47.
    Renzi, A. B., Chammas, A., Agner, L., & Greenshpan, J. (2015, August). Startup Rio: user experience and startups. International Conference of Design, User Experience, and Usability Springer, Cham, pp 339–347. DOI: Scholar
  48. 48.
    Rusu, C., Rusu, V., Roncagliolo, S., Apablaza, J., & Rusu, V. Z. (2015, August). User experience evaluations: challenges for newcomers. International Conference of Design, User Experience, and Usability Springer, Cham, pp 237–246. DOI: Scholar
  49. 49.
    Salian, K., & Sim, G. (2014, December). Simplifying heuristic evaluation for older children. In Proceedings of the India HCI 2014 Conference on Human Computer Interaction (p. 26). ACM. DOI:
  50. 50.
    Salian, K., Sim, G., & Read, J. C. (2013, September). Can children perform a heuristic evaluation? In Proceedings of the 11th Asia Pacific Conference on Computer Human Interaction, ACM, pp 137-141. DOI:
  51. 51.
    Wei, T., & Simko, V. (2016). corrplot: Visualization of a Correlation Matrix. R package version 0.77. CRAN, Vienna, Austria.Google Scholar
  52. 52.
    Wodike, O. A., Sim, G., & Horton, M. (2014, September). Empowering teenagers to perform a heuristic evaluation of a game. In Proceedings of the 28th International BCS Human Computer Interaction Conference on HCI 2014-Sand, Sea and Sky-Holiday HCI, BCS, pp 353–358.Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • André de Lima Salgado
    • 1
    Email author
  • Flávia de Souza Santos
    • 1
  • Renata Pontin de Mattos Fortes
    • 1
  • Patrick C. K. Hung
    • 2
  1. 1.Institute of Mathematical Science and ComputingUniversity of São Paulo (USP)São CarlosBrazil
  2. 2.Faculty of Business and Information TechnologyUniversity of Ontario Institute of Technology (UOIT)OshawaCanada

Personalised recommendations