Skip to main content
SpringerLink
Log in

Challenges and Opportunities on the Application of Heuristic Evaluations: A Systematic Literature Review

  • Conference paper
  • First Online:
Design, User Experience, and Usability: UX Research and Design (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12779))

Included in the following conference series:

Abstract

Heuristic evaluation belongs to the usability inspection methods and is considered one of the most popular methods since it allows to discover over 75% of the total usability problems involving only 3 to 5 usability experts, in comparison with user tests. However, certain problems and challenges have been identified at the time of their execution. In this study we present the results of conducting a Systematic Literature Review (SLR) to identify case studies, challenges, problems, and opportunities on the execution of heuristic evaluations in the context of a research for the automation and formalization of the process. For this SLR, we have employed the protocol proposed by Kitchenham and Charters. The research was carried out on September 7 of 2020 and retrieved a total of 167 studies of which 37 were selected for this review. The results show that the main challenges are related to the low suitability of the chosen set of heuristics and the low expertise of usability evaluators. Additionally, we have identified that exist very few software solutions that support and automate the process. Finally, we have found that there were many protocols to follow when applying a heuristic evaluation, like the definition of new usability heuristics for a given case of study. According to the results obtained, we can conclude that it is necessary to develop and validate a tool based on a formal protocol that supports and automate the heuristic evaluation process, that gives solutions to the challenges and opportunities identified in this research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Huang, Z.: Usability of tourism websites: a case study of heuristic evaluation. New Rev. Hypermedia Multimed. 26, 55–91 (2020). https://doi.org/10.1080/13614568.2020.1771436

    Article  Google Scholar 

  2. Holzinger, A.: Usability engineering methods for software developers. Commun. ACM. 48, 71–74 (2005). https://doi.org/10.1145/1039539.1039541

    Article  Google Scholar 

  3. Nielsen, J.: Usability Engineering. Morgan Kaufmann Publishers Inc., San Francisco (1993)

    Book  Google Scholar 

  4. Kitchenham, B., Charters, S.: Guidelines for performing Systematic Literature Reviews in Software Engineering (2007)

    Google Scholar 

  5. International Organization for Standardization: ISO 9241–210–2019 (2019)

    Google Scholar 

  6. Muhammed, M.T., Waeal J, O., Bijan, R.: Applying deep learning techniques for big data analytics: a systematic literature review. Arch. Inf. Sci. Technol. 1 (2018). https://doi.org/10.36959/863/756.

  7. Petticrew, M., Roberts, H.: Systematic reviews in the social sciences: a practical guide. Blackwell Publishing, Malden (2006). https://doi.org/10.1002/9780470754887

  8. Hasan, L., Morris, A.: Usability problem areas on key international and key arab e-commerce websites. J. Internet Commer. 16, 80–103 (2017). https://doi.org/10.1080/15332861.2017.1281706

    Article  Google Scholar 

  9. Davis, D., Jiang, S.: Usability evaluation of web-based interfaces for Type2 Diabetes Mellitus. In: IEOM 2015 - 5th International Industrial Engineering and Operations Management Proceeding (2015). https://doi.org/10.1109/IEOM.2015.7093713

  10. Lam, D., Sajjanhar, A.: Heuristic evaluations of cultural heritage websites. In: 2018 Int. Conf. Digit. Image Comput. Tech. Appl. DICTA 2018. (2019). https://doi.org/10.1109/DICTA.2018.8615847.

  11. Jiménez, C., Cid, H.A., Figueroa, I.: PROMETHEUS: procedural methodology for developing heuristics of usability. IEEE Lat. Am. Trans. 15, 541–549 (2017). https://doi.org/10.1109/TLA.2017.7867606

    Article  Google Scholar 

  12. Paz, F., Paz, F.A., Pow-Sang, J.A., Collazos, C.: A formal protocol to conduct usability heuristic evaluations in the context of the software development process. Int. J. Eng. Technol. 7, 10–19 (2018). https://doi.org/10.14419/ijet.v7i2.28.12874

  13. Sanabre, C., Pedraza-Jiménez, R., Codina, L.: WebSite Canvas Model: propuesta de un modelo visual para la ideación estratégica de sitios web. Rev. española Doc. Científica. 41, 221 (2018). https://doi.org/10.3989/redc.2018.4.1542

    Article  Google Scholar 

  14. Ghazali, M., Sivaji, A., Abdollah, N., Khean, C.N.: Visual Clarity Checker (VC2) to support heuristic evaluation: to what extent does VC2 help evaluators? In: Proceedings - 2016 4th International Conference on User Science and Engineering i-USEr 2016, pp. 182–187 (2017). https://doi.org/10.1109/IUSER.2016.7857957

  15. Maghfiroh, L.R.: Observation and heuristics evaluation of student web-based application of SIPADU-STIS. J. Phys. Conf. Ser. 1511, 0–10 (2020). https://doi.org/10.1088/1742-6596/1511/1/012019

  16. Pincay-Ponce, J., Caicedo-Ávila, V., Herrera-Tapia, J., Delgado-Muentes, W., Delgado-Franco, P.: Usabilidad en sitios web oficiales de las universidades del Ecuador. Rev. Ibérica Sist. e Tecnol. Informação, pp. 106–120 (2020)

    Google Scholar 

  17. Walsh, L., et al.: The e-health literacy demands of Australia’s my health record: a heuristic evaluation of Usability. Perspect. Heal. Inf. Manag. 14 (2017)

    Google Scholar 

  18. Díaz, J., Rusu, C., Collazos, C.A.: Experimental validation of a set of cultural-oriented usability heuristics: e-Commerce websites evaluation. Comput. Stand. Interfaces. 50, 160–178 (2017). https://doi.org/10.1016/j.csi.2016.09.013

    Article  Google Scholar 

  19. Chen, Y.N., Hwang, S.L.: A heuristic evaluation on the usability of health information websites. In: Bridging Research and Good Practices towards Patient Welfare - Proceedings of the 4th International Conference on HealthCare Systems Ergonomics and Patient Safety, HEPS 2014, pp. 109–116 (2015)

    Google Scholar 

  20. Kiourexidou, M., Antonopoulos, N., Kiourexidou, E., Piagkou, M., Kotsakis, R., Natsis, K.: Websites with multimedia content: a heuristic evaluation of the medical/anatomical museums. Multimodal Technol. Interact. 3 (2019). https://doi.org/10.3390/mti3020042

  21. Yeratziotis, A., Zaphiris, P.: A heuristic evaluation for deaf web user experience (HE4DWUX). Int. J. Hum. Comput. Interact. 34, 195–217 (2018). https://doi.org/10.1080/10447318.2017.1339940

    Article  Google Scholar 

  22. Eliseo, M.A., Casac, B.S., Gentil, G.R.: A comparative study of video content user interfaces based on heuristic evaluation. In: Iberian Conference on Information Systems and Technologies Cist. (2017). https://doi.org/10.23919/CISTI.2017.7975820

  23. Motlagh Tehrani, S.E., Zainuddin, N.M.M., Takavar, T.: Heuristic evaluation for Virtual Museum on smartphone. In: Proceedings - 2014 3rd International Conference on User Science and Engineering: Experience. Engineer Engage, i-USEr 2014. 227–231 (2015). https://doi.org/10.1109/IUSER.2014.7002707

  24. Puspitasari, I., Indah Cahyani, D.: Taufik: a user-centered design for redesigning e-government website in public health sector. In: Proceedings - 2018 International Seminar on Application for Technology of Information and Communication, Creation Technology Human Life, iSemantic 2018, pp. 219–224 (2018). https://doi.org/10.1109/ISEMANTIC.2018.8549726

  25. Quinones, D., Rusu, C., Roncagliolo, S., Rusu, V., Collazos, C.A.: Developing usability heuristics: a formal or informal process? IEEE Lat. Am. Trans. 14, 3400–3409 (2016). https://doi.org/10.1109/TLA.2016.7587648

    Article  Google Scholar 

  26. Ahmad, W.F.W., Sarlan, A., Ezekiel, A., Juanis, L.: Automated heuristic evaluator. J. Informatics Math. Sci. 8, 301–306 (2016)

    Google Scholar 

  27. Paz Espinoza, F.A.: Método para la evaluación de usabilidad de sitios web transaccionales basado en el proceso de inspección heurística. Pontif. Univ. Católica del Perú. 275 (2018). file:///C:/Users/UTM-BIBLIOTECA/Downloads/PAZ_FREDDY_USABILIDAD_SITIOS_WEB_%20INSPECCI%C3%93N_HEUR%C3%8DSTICA.pdf

    Google Scholar 

  28. Halaweh, M.: Usability testing of conferences websites: a case study of practical teaching. In: Uden, L., Hadzima, B., Ting, I.-H. (eds.) KMO 2018. CCIS, vol. 877, pp. 380–389. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-95204-8_32

    Chapter  Google Scholar 

  29. Silvis, I.M., Bothma, T.J.D., de Beer, K.J.W.: Evaluating the usability of the information architecture of academic library websites. Libr. Hi Tech. 37, 566–590 (2019). https://doi.org/10.1108/LHT-07-2017-0151

    Article  Google Scholar 

  30. Paz, F., Paz, F.A., Arenas, J.J., Rosas, C.: A Perception study of a new set of usability heuristics for transactional web sites. In: Karwowski, W., Ahram, T. (eds.) IHSI 2018. AISC, vol. 722, pp. 620–625. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73888-8_96

    Chapter  Google Scholar 

  31. Paz, F., Paz, F.A., Pow-Sang, J.A.: Comparing the effectiveness and accuracy of new usability heuristics. Adv. Intell. Syst. Comput. 497, 163–175 (2017). https://doi.org/10.1007/978-3-319-41956-5_16

    Article  Google Scholar 

  32. Chanchí Golondrino, G.E., Pérez Oliveros, D., Campo Muñoz, W.Y.: Automation of usability inspections for websites. In: Ruiz, P.H., Agredo-Delgado, V. (eds.) HCI-COLLAB 2019. CCIS, vol. 1114, pp. 124–137. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-37386-3_10

    Chapter  Google Scholar 

  33. Paz, F., Paz, F.A., Sánchez, M., Moquillaza, A., Collantes, L.: Quantifying the usability through a variant of the traditional heuristic evaluation process. In: Marcus, A., Wang, W. (eds.) DUXU 2018. LNCS, vol. 10918, pp. 496–508. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91797-9_36

    Chapter  Google Scholar 

  34. Inal, Y.: University students’ heuristic usability ınspection of the national library of Turkey website. Aslib J. Inf. Manag. 70, 66–77 (2018). https://doi.org/10.1108/AJIM-09-2017-0216

    Article  Google Scholar 

  35. dos Santos Pergentino, A.C., Canedo, E.D., Lima, F., de Mendonça, F.L.L.: Usability heuristics evaluation in search engine. In: Marcus, A., Rosenzweig, E. (eds.) HCII 2020. LNCS, vol. 12200, pp. 351–369. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49713-2_25

    Chapter  Google Scholar 

  36. Iadanza, E., Fabbri, R., Luschi, A., Melillo, P., Simonelli, F.: A collaborative RESTful cloud-based tool for management of chromatic pupillometry in a clinical trial. Health Technol. 10(1), 25–38 (2019). https://doi.org/10.1007/s12553-019-00362-z

    Article  Google Scholar 

  37. Federici, S., et al.: Heuristic evaluation of eGLU-box: a semi-automatic usability evaluation tool for public administrations. In: Kurosu, M. (ed.) HCII 2019. LNCS, vol. 11566, pp. 75–86. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22646-6_6

    Chapter  Google Scholar 

  38. de Menezes, M., Falco, M.: The relationship of the studies of ergonomic and human computer interfaces – a case study of graphical interfaces in e-commerce websites. In: Marcus, A., Wang, W. (eds.) HCII 2019. LNCS, vol. 11586, pp. 474–484. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23535-2_35

    Chapter  Google Scholar 

  39. Paz, F., Paz, F.A., Pow-Sang, J.A.: Evaluation of usability heuristics for transactionalweb sites: a comparative study. Adv. Intell. Syst. Comput. 448, 1063–1073 (2016). DOI: https://doi.org/10.1007/978-3-319-32467-8_92

  40. Retore, A.P., Guimarães, C., Leite, M.K.: Heuristics for grid and typography evaluation of art magazines websites. In: Kurosu, M. (ed.) HCI 2016. LNCS, vol. 9731, pp. 408–416. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39510-4_38

    Chapter  Google Scholar 

  41. Morales, J., Rusu, C., Botella, F., Quiñones, D.: Programmer eXperience: a set of heuristics for programming environments. In: Meiselwitz, G. (ed.) HCII 2020. LNCS, vol. 12195, pp. 205–216. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49576-3_15

    Chapter  Google Scholar 

  42. Okhovati, M., Karami, F., Khajouei, R.: Exploring the usability of the central library websites of medical sciences universities. J. Librariansh. Inf. Sci. 49, 246–255 (2017). https://doi.org/10.1177/0961000616650932

    Article  Google Scholar 

  43. Fung, R.H.Y., Chiu, D.K.W., Ko, E.H.T., Ho, K.K.W., Lo, P.: Heuristic usability evaluation of university of hong kong libraries’ mobile website. J. Acad. Librariansh. 42, 581–594 (2016). https://doi.org/10.1016/j.acalib.2016.06.004

    Article  Google Scholar 

Download references

Acknowledgments

This study was highly supported by the Section of Informatics Engineering and the research group “HCI, Design, User Experience, Accessibility & Innovation Technologies” (HCI-DUXAIT) of the Pontificia Universidad Católica del Perú (PUCP).

Author information

Authors and Affiliations

  1. Pontificia Universidad Católica del Perú, Av. Universitaria 1801, San Miguel, Lima 32, Lima, Peru

    Adrian Lecaros, Freddy Paz & Arturo Moquillaza

Authors
  1. Adrian Lecaros
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Freddy Paz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arturo Moquillaza
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adrian Lecaros .

Editor information

Editors and Affiliations

  1. Hunan University, School of Design, Changsha, China

    Marcelo M. Soares

  2. World Usability Day and Rabb School of Continuing Studies, Division of Graduate Professional Studies, Brandeis University, Newton Center, MA, USA

    Elizabeth Rosenzweig

  3. Aaron Marcus and Associates, Berkeley, CA, USA

    Aaron Marcus

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lecaros, A., Paz, F., Moquillaza, A. (2021). Challenges and Opportunities on the Application of Heuristic Evaluations: A Systematic Literature Review. In: Soares, M.M., Rosenzweig, E., Marcus, A. (eds) Design, User Experience, and Usability: UX Research and Design. HCII 2021. Lecture Notes in Computer Science(), vol 12779. Springer, Cham. https://doi.org/10.1007/978-3-030-78221-4_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-78221-4_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-78220-7

  • Online ISBN: 978-3-030-78221-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation