Heuristic Evaluation in Information Visualization Using Three Sets of Heuristics: An Exploratory Study

  • Beatriz Sousa SantosEmail author
  • Beatriz Quintino Ferreira
  • Paulo Dias
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9169)


Evaluation in Information Visualization is inherently complex, and it is still a challenge. Whereas it is possible to adapt evaluation methods from other fields, as Human-Computer Interaction, this adaptation may not be straightforward since visualization applications are very specific interactive systems.

This paper addresses issues in using heuristic evaluation to evaluate visualizations and visualization applications, and presents an exploratory study in two phases and involving 25 evaluators aimed at assessing the understandability and effectiveness of three sets of heuristics that have been used in Information Visualization.


InfoVis evaluation Usability, cognitive and visual heuristics Heuristic evaluation 



The authors are grateful to the participants. This work was partially funded by FEDER through the Operational Program Competitiveness Factors – COMPETE – by National Funds through Foundation for Science and Technology – FCT (references FCOMP-01-0124-FEDER-022682 and PEst-C/EEI/UI0127/2011).


  1. 1.
    Plaisant, C.: The challenge of information visualization evaluation. In: Proceedings of the International Conference on Advanced Visual Interfaces (AVI2004), pp. 109–116. ACM, New York (2004)Google Scholar
  2. 2.
    Chen, C.: Top 10 unsolved information visualization problems. IEEE Comput. Graph. Appl. 25(4), 12–16 (2005)CrossRefGoogle Scholar
  3. 3.
    Carpendale, S.: Evaluating information visualizations. In: Kerren, A., Stasko, J.T., Fekete, J.-D., North, C. (eds.) Information Visualization. LNCS, vol. 4950, pp. 19–45. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  4. 4.
    Sedlmair, M., Isemberg, P., Baur, D., Butz, A.: Information visualization evaluation in large companies : challenges, experiences and recommendations. Inf. Vis. J. 10(3), 248–266 (2011)CrossRefGoogle Scholar
  5. 5.
    Lam, H., Bertini, E., Isenberg, P., Plaisant, C., Carpendale, S.: Empirical studies in information visualization: seven scenarios. IEEE Trans. Vis. Comput. Graph. 18(9), 1520–1536 (2012)CrossRefGoogle Scholar
  6. 6.
    Aigner, W., Hoffmann, S., Rind, A.: EvalBench: a software library for visualization evaluation. Comput. Graph. Forum 32(3pt1), 41–50 (2013)CrossRefGoogle Scholar
  7. 7.
    McGrath, J.E.: Methodology matters: doing research in the social and behavioural sciences. In: Baecker, R., Grudin, J., Buxton, W., Greenberg, S. (eds.) Readings in Human-Computer Interaction: Toward the Year 2000, 2nd edn, pp. 152–169. Morgan Kaufmann Publishers, San Francisco (1995)Google Scholar
  8. 8.
    Dix, A., Finlay, J., Abowd, G., Beale, R.: Human-Computer Interaction, 3rd edn. Prentice Hall, Harlow (2004)Google Scholar
  9. 9.
    Nielsen, J.: Usability Engineering. Morgan Kaufmann Publishers, San Francisco (1993)zbMATHGoogle Scholar
  10. 10.
    Zuk, T., Carpendale, S.: Theoretical analysis of uncertainty visualizations. In: SPIE & IS&T Conference Electronic Imaging, Visualization and Data Analysis 2006, vol. 6060, pp. 606007. SPIE (2006)Google Scholar
  11. 11.
    Forsell, C., Johanson, J.: An heuristic set for evaluation in information visualization. In: Proceedings of AVI 2010, pp. 199–206. ACM, New York (2010)Google Scholar
  12. 12.
    Trainer, E., Quirk, S., Souza, C., Redmiles, D.: Analyzing a socio-technical visualization tool using usability inspection methods. In: 2008 IEEE Symposium on Visual Languages and Human-Centric Computing, pp. 78–81. IEEE (2008)Google Scholar
  13. 13.
    Trainer, E., Quirk, S., Souza, C., Redmiles, D.: Usability Inspection Method-based Analysis of a Socio-Technical Visualization Tool. Tech. report, ISR, University of California, Irvine (2012)Google Scholar
  14. 14.
    Andrews, K.: Evaluation comes in many guises. In: Proceedings of the 2008 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization, (BELIV 2008). ACM, New York (2008)Google Scholar
  15. 15.
    Sousa Santos, B., Dias, P.: Evaluation in visualization: some issues and best practices. In: SPIE Conference Electronic Imaging, Visualization and Data Analysis 2014, vol. 9017, pp. 90170O-1–8. SPIE, San Francisco (2014)Google Scholar
  16. 16.
    Munzner, T.: A nested model for visualization design and validation. IEEE Trans. Vis. Comput. Graph. 15(6), 921–928 (2009)CrossRefGoogle Scholar
  17. 17.
    Gutwin, C., Greenberg, S.: The mechanics of collaboration: developing low cost usability evaluation methods for shared workspaces. In: Proceedings of the IEEE 9th International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET ICE 2000), pp. 98–103. IEEE Computer Society, Washington (2000)Google Scholar
  18. 18.
    Baker, K., Greenberg, S., Gutwin, C.: Empirical development of a heuristic evaluation methodology for shared workspace groupware. In: Proceedings of the 2002 ACM Conference Computer Supported Cooperative Work (CSCW 2002), pp. 96–105. ACM, New York (2002)Google Scholar
  19. 19.
    Kuparinen, L., Silvennoinen, J., Isomäki, H.: Introducing usability heuristics for mobile map applications. In: Proceedings of the 26th International Cartographic Conference (ICC 2013) (2013)Google Scholar
  20. 20.
    Tory, M., Möller, T.: Human factors in visualization research. IEEE Trans. Vis. Comput. Graph. 10(1), 72–84 (2004)CrossRefGoogle Scholar
  21. 21.
    Tory, M., Möller, T.: Evaluating visualizations: do expert reviews work? IEEE Comput. Graph. Appl. 25(5), 8–11 (2005)CrossRefGoogle Scholar
  22. 22.
    Shneiderman, B.: The eyes have it: a task by data type taxonomy for information visualizations. In: Proceedings of the IEEE Symposium on Visual Language, pp. 336–343. IEEE (1996)Google Scholar
  23. 23.
    Ware, C.: Information Visualization: Perception for Design, 2nd edn. Morgan Kaufmann Publishers, Waltham (2004)Google Scholar
  24. 24.
    Amar, R., Stasko, J.: A knowledge task-based framework for design and evaluation of information visualizations. In: IEEE Symposium Info Vis, pp. 143–150. IEEE (2004)Google Scholar
  25. 25.
    Zuk, T., Schlesier, L., Neumann, P., Hancock, M.S., Carpendale, S.: Heuristics for information visualization evaluation. In: BELIV 2006, pp. 1–6. ACM, New York (2006)Google Scholar
  26. 26.
    Freitas, C.M., Luzzardi, P.R.G., Cava, R.A., Winckler, M.A.A., Pimenta, M.S., Nedel, L.P.: Evaluating usability of information visualization techniques. In: Proceedings of the 5th Symposium on Human Factors in Computer Systems (IHC 2002), pp. 40 − 51 (2002)Google Scholar
  27. 27.
    Tufte, E.: The Visual Display of Quantitative Information, 2nd edn. Graphics Press, Anaheim (2001)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Beatriz Sousa Santos
    • 1
    • 2
    Email author
  • Beatriz Quintino Ferreira
    • 2
  • Paulo Dias
    • 1
    • 2
  1. 1.Department of Electronics Telecommunications and InformaticsUniversity of AveiroAveiroPortugal
  2. 2.Institute of Electronics Engineering and Telematics of Aveiro/IEETAAveiroPortugal

Personalised recommendations