Advertisement

Guiding Usability Newcomers to Understand the Context of Use: Towards Models of Collaborative Heuristic Evaluation

  • André de Lima SalgadoEmail author
  • Flávia de Souza Santos
  • Renata Pontin de Mattos Fortes
  • Patrick C. K. Hung
Chapter
Part of the International Series on Computer Entertainment and Media Technology book series (ISCEMT)

Abstract

Usability inspection methods are liable to the expertise effect among distinct evaluators. Hence, understanding the difficulties faced by evaluators of low expertise (novices and newcomers) is a requirement to move forward in the field. However, the following question remains: Which of the terms that compose usability (User, Goal, Effectiveness, Efficiency, Satisfaction, Context of Use and Task) is the most difficult for newcomers to understand? This exploratory study aims to compare usability newcomers’ difficulties on understanding different terms that compose the usability, based on the definition showed by the ISO/IEC 25066. To achieve this goal, we conducted a survey with 38 usability newcomers. Observations on our survey show the Context of Use may be the most difficult term to be understood by newcomers. Thus, we suggest the adoption of scenarios, storyboards, and domain-specific principles as the basis for newcomers in HEs, when practitioners cannot count on experts for the inspection. In addition, we suggest three (3) different models of Collaborative Heuristic Evaluation aimed to provide newcomers with important insights about Context of Use. Finally, we suggest as future works to study the validity of such models.

Keywords

Usability Heuristic evaluation Evaluators Novice Newcomer Expertise effect Evaluator effect 

Notes

Acknowledgments

This study was supported by the grants 2017/15239-0 and 2015/24525-0, São Paulo Research Foundation (FAPESP). We also thank to CAPES and University of São Paulo for their important support.

References

  1. 1.
    Boone, H. N., & Boone, D. A. (2012). Analyzing likert data. Journal of extension, 50(2), pp 1–5.Google Scholar
  2. 2.
    Brajnik, G., Yesilada, Y., & Harper, S. (2011). The expertise effect on web accessibility evaluation methods. Human–Computer Interaction, 26(3), pp 246–283. DOI: https://doi.org/10.1080/07370024.2011.601670 CrossRefGoogle Scholar
  3. 3.
    Bruun, A., & Stage, J. (2014). Barefoot usability evaluations. Behaviour & Information Technology, 33(11), pp 1148–1167. DOI: https://doi.org/10.1080/0144929X.2014.883552 CrossRefGoogle Scholar
  4. 4.
    Bruun, A., & Stage, J. (2015). New approaches to usability evaluation in software development: Barefoot and crowdsourcing. Journal of Systems and Software, 105, pp 40–53. DOI: https://doi.org/10.1016/j.jss.2015.03.043 CrossRefGoogle Scholar
  5. 5.
    Bryman, A. (2015). Social research methods. Oxford university press. ISBN: 978-0-19-968945-3Google Scholar
  6. 6.
    Buykx, L. (2009). Improving heuristic evaluation through collaborative working. Masters Dissertation. The University of York Department of Computer Science.Google Scholar
  7. 7.
    Chambel, T. (2016, November). Interactive and Immersive Media Experiences. In Proceedings of the 22nd Brazilian Symposium on Multimedia and the Web, pp 1–1. ACM. DOI: https://doi.org/10.1145/2976796.2984746
  8. 8.
    de Lima Salgado, A., & Fortes, R. P. M. (2016). Heuristic Evaluation for Novice Evaluators. In International Conference of Design, User Experience, and Usability, Springer International Publishing, pp 387–398. DOI:  https://doi.org/10.1007/978-3-319-40409-7_37 CrossRefGoogle Scholar
  9. 9.
    DeLone, W. H., & McLean, E. R. (1992). Information systems success: The quest for the dependent variable. Information systems research, 3(1), pp 60–95. DOI: https://doi.org/10.1287/isre.3.1.60 CrossRefGoogle Scholar
  10. 10.
    Delone, W. H., & McLean, E. R. (2003). The DeLone and McLean model of information systems success: a ten-year update. Journal of management information systems, 19(4), pp 9–30. DOI: https://doi.org/10.1080/07421222.2003.11045748 CrossRefGoogle Scholar
  11. 11.
    Demers, R. A. (1981). System design for usability. Communications of the ACM, 24(8), pp 494–501. DOI: https://doi.org/10.1145/358722.358730 CrossRefGoogle Scholar
  12. 12.
    Ebling, M. R., & Want, R. (2017). Pervasive Computing Revisited. IEEE Pervasive Computing, 16(3), pp 17–19. DOI: https://doi.org/10.1109/MPRV.2017.2940959 CrossRefGoogle Scholar
  13. 13.
    Ferreira, D. F. (2008). Estatística multivariada. Editora UFLA. ISBN: 978-85-87692-52-8Google Scholar
  14. 14.
    Georgsson, M., Weir, C., & Staggers, N. (2014). Revisiting heuristic evaluation methods to improve the reliability of findings. In MIE2014. IOS Press. DOI: https://doi.org/10.3233/978-1-61499-432-9-930 CrossRefGoogle Scholar
  15. 15.
    Hair, J. F., Anderson, R. E., Babin, B. J. & Black, W. C. (2010). . Multivariate data analysis: A global perspective Pearson Upper Saddle River, NJ. ISBN: 0133792684Google Scholar
  16. 16.
    Hermawati, S., & Lawson, G. (2016). Establishing usability heuristics for heuristics evaluation in a specific domain: Is there a consensus? Applied ergonomics, 56, pp 34–51. DOI: https://doi.org/10.1016/j.apergo.2015.11.016 CrossRefGoogle Scholar
  17. 17.
    Hertzum, M. (2017). Commentary: Usability—A Sensitizing Concept. Human–Computer Interaction, pp 1–4. DOI:  https://doi.org/10.1080/07370024.2017.1302800 CrossRefGoogle Scholar
  18. 18.
    Hertzum, M., & Jacobsen, N. E. (2001). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13(4), pp 421–443. DOI: https://doi.org/10.1207/S15327590IJHC1304_05 CrossRefGoogle Scholar
  19. 19.
    Hornbæk, K. (2010). Dogmas in the assessment of usability evaluation methods. Behaviour & Information Technology, 29(1), pp 97–111. DOI:  https://doi.org/10.1080/01449290801939400 CrossRefGoogle Scholar
  20. 20.
    Huang, B. (2012). A Comparison of Remote Collaborative Heuristic Evaluation by Novices and Experts with User-based Evaluation. Masters Dissertation. The University of York Department of Computer Science.Google Scholar
  21. 21.
    Hung P.C.K., Tang J.K.T., & Kanev K. (2017) Introduction. In: Tang J., Hung P. (eds) Computing in Smart Toys. International Series on Computer Entertainment and Media Technology Springer, Cham, pp 1–5. DOI:  https://doi.org/10.1007/978-3-319-62072-5_1 CrossRefGoogle Scholar
  22. 22.
    ISO 25066 (2016). Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Common Industry Format (CIF) for Usability - Evaluation Report.Google Scholar
  23. 23.
    ISO 9241-210 (2010). Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems.Google Scholar
  24. 24.
    Johannessen, G. H. J., & Hornbæk, K. (2014). Must evaluation methods be about usability? Devising and assessing the utility inspection method. Behaviour & Information Technology, 33(2), pp 195–206. DOI:  https://doi.org/10.1080/0144929X.2012.751708 CrossRefGoogle Scholar
  25. 25.
    Kangas, K., Seitamaa-Hakkarainen, P., & Hakkarainen, K. (2013). Design expert’s participation in elementary students’ collaborative design process. International Journal of Technology and Design Education, 23(2), pp 161–178. DOI:  https://doi.org/10.1007/s10798-011-9172-6 CrossRefGoogle Scholar
  26. 26.
    Lewis, J. R. (2014). Usability: lessons learned… and yet to be learned. International Journal of Human-Computer Interaction, 30(9), pp 663–684. DOI:  https://doi.org/10.1080/10447318.2014.930311 CrossRefGoogle Scholar
  27. 27.
    MacDonald, C. M., & Atwood, M. E. (2013, April). Changing perspectives on evaluation in HCI: past, present, and future. In CHI’13 Extended Abstracts on Human Factors in Computing Systems, pp 1969–1978. ACM. DOI: https://doi.org/10.1145/2468356.2468714
  28. 28.
    MacFarlane, S., & Pasiali, A. (2005). Adapting the heuristic evaluation method for use with children. In Workshop on child computer interaction: methodological research, Interact, pp 28–31.Google Scholar
  29. 29.
    MacFarlane, S., Sim, G., & Horton, M. (2005, June). Assessing usability and fun in educational software. In Proceedings of the 2005 conference on Interaction design and children, pp 103–109. ACM.Google Scholar
  30. 30.
    Martins, A. I., Queirós, A., Silva, A. G., & Rocha, N. P. (2014). Usability evaluation methods: A systematic review. Human Factors in Software Development and Design, 250. DOI: https://doi.org/10.4018/978-1-4666-6485-2.ch013
  31. 31.
    Molich, R., & Nielsen, J. (1990). Improving a human-computer dialogue. Communications of the ACM, 33(3), pp 338–348. DOI:  https://doi.org/10.1145/77481.77486 CrossRefGoogle Scholar
  32. 32.
    Nielsen, J., & Molich, R. (1989). Teaching user interface design based on usability engineering. ACM SIGCHI Bulletin, 21(1), pp 45–48. DOI: https://doi.org/10.1145/67880.67885 CrossRefGoogle Scholar
  33. 33.
    Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 249–256. ACM. DOI:  https://doi.org/10.1145/97243.97281
  34. 34.
    Nielsen, J. (1995). How to conduct a heuristic evaluation. Nielsen Norman Group. URL: https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/
  35. 35.
    Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Proceedings ACM CHI’92 Conference (Monterey, CA, May 3-7), 373–380. DOI: https://doi.org/10.1145/142750.142834
  36. 36.
    Nielsen, J. (1994a, April). Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, ACM, pp 152–158. DOI:  https://doi.org/10.1145/191666.191729
  37. 37.
    Nielsen, J. Heuristic Evaluation: How-To: Article by Jakob Nielsen. Nielsen Norman Group Available at: https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/. (Accessed: 2nd May 2018)Google Scholar
  38. 38.
    Nielsen, J. (2002). Becoming a usability professional. Nielsen Norman Group. URL: https://www.nngroup.com/articles/becoming-a-usability-professional/
  39. 39.
    Nielsen, J. (2012). Usability 101: Introduction to usability. Nielsen Norman Group. URL: https://www.nngroup.com/articles/usability-101-introduction-to-usability
  40. 40.
    Othman, M. K., Mahudin, F., Ahaguk, C. H., & Rahman, M. F. A. (2014, September). Mobile guide technologies (smartphone apps): Collaborative Heuristic Evaluation (CHE) with expert and novice users. In User Science and Engineering (i-USEr), 2014 3rd International Conference on (pp. 232–236). IEEE. DOI:  https://doi.org/10.1109/IUSER.2014.7002708
  41. 41.
    Paz, F., & Pow-Sang, J. A. (2016). A systematic mapping review of usability evaluation methods for software development process. International Journal of Software Engineering and Its Applications, 10(1), pp 165–178.CrossRefGoogle Scholar
  42. 42.
    Petrie, H., & Buykx, L. (2010). Collaborative Heuristic Evaluation: improving the effectiveness of heuristic evaluation. In Proceedings of UPA 2010 International Conference.Google Scholar
  43. 43.
    Petrie, H., & Power, C. (2012, May). What do users really care about? a comparison of usability problems found by users and experts on highly interactive websites. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, pp 2107–2116. ACM. DOI:  https://doi.org/10.1145/2207676.2208363
  44. 44.
    Petter, S., DeLone, W., & McLean, E. R. (2013). Information systems success: The quest for the independent variables. Journal of Management Information Systems, 29(4), pp 7–62. DOI:  https://doi.org/10.2753/MIS0742-1222290401.CrossRefGoogle Scholar
  45. 45.
    Popovic, V. (2004). Expertise development in product design—strategic and domain-specific knowledge connections. Design Studies, 25(5), (pp. 527–545). DOI:  https://doi.org/10.1016/j.destud.2004.05.006 CrossRefGoogle Scholar
  46. 46.
    Read, J. (2015). Children as participants in design and evaluation. interactions, 22(2), pp 64–66. DOI:  https://doi.org/10.1145/2735710.CrossRefGoogle Scholar
  47. 47.
    Renzi, A. B., Chammas, A., Agner, L., & Greenshpan, J. (2015, August). Startup Rio: user experience and startups. International Conference of Design, User Experience, and Usability Springer, Cham, pp 339–347. DOI:  https://doi.org/10.1007/978-3-319-20886-2_32.CrossRefGoogle Scholar
  48. 48.
    Rusu, C., Rusu, V., Roncagliolo, S., Apablaza, J., & Rusu, V. Z. (2015, August). User experience evaluations: challenges for newcomers. International Conference of Design, User Experience, and Usability Springer, Cham, pp 237–246. DOI:  https://doi.org/10.1007/978-3-319-20886-2_23.CrossRefGoogle Scholar
  49. 49.
    Salian, K., & Sim, G. (2014, December). Simplifying heuristic evaluation for older children. In Proceedings of the India HCI 2014 Conference on Human Computer Interaction (p. 26). ACM. DOI:  https://doi.org/10.1145/2676702.2676704.
  50. 50.
    Salian, K., Sim, G., & Read, J. C. (2013, September). Can children perform a heuristic evaluation? In Proceedings of the 11th Asia Pacific Conference on Computer Human Interaction, ACM, pp 137-141. DOI:  https://doi.org/10.1145/2525194.2525200
  51. 51.
    Wei, T., & Simko, V. (2016). corrplot: Visualization of a Correlation Matrix. R package version 0.77. CRAN, Vienna, Austria.Google Scholar
  52. 52.
    Wodike, O. A., Sim, G., & Horton, M. (2014, September). Empowering teenagers to perform a heuristic evaluation of a game. In Proceedings of the 28th International BCS Human Computer Interaction Conference on HCI 2014-Sand, Sea and Sky-Holiday HCI, BCS, pp 353–358.Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • André de Lima Salgado
    • 1
    Email author
  • Flávia de Souza Santos
    • 1
  • Renata Pontin de Mattos Fortes
    • 1
  • Patrick C. K. Hung
    • 2
  1. 1.Institute of Mathematical Science and ComputingUniversity of São Paulo (USP)São CarlosBrazil
  2. 2.Faculty of Business and Information TechnologyUniversity of Ontario Institute of Technology (UOIT)OshawaCanada

Personalised recommendations