How to Investigate Interaction with Information Visualisation: An Overview of Methodologies
Advanced information visualisation systems offer many different forms of interaction. Nevertheless, we do not know how useful these interactions are. Researchers have suggested to develop a science of interaction. In this paper we discuss which research methods might be appropriate to study interaction with information visualisation systems. We suggest that thinking aloud, log files and eye tracking are promising candidates. These methods enable researchers to study interaction in more detail than other methods. All these methods have strengths and weaknesses. A combination of two or three of these methods might help to overcome the weaknesses.
KeywordsThinking aloud Eyetracking Log files Interaction patterns Triangulation
Advanced IT systems enable users to interact with them in multiple ways. Users of interactive information visualisation systems, for example, can filter data, show data in more or less detail or represent the data in different visual forms (e.g. as scatterplots or as a graph etc). Such interactions have to be designed appropriately to be useful. User-centered design can help to develop interactive features with a high usability.
The investigation of various forms of interaction has become more important in HCI in recent years . Mirel , for example, argues that human problem-solving and open-ended inquiry consist of different high-level activities (e.g. wayfinding, sensemaking, ...). It is necessary to identify interaction patterns, that is “recurring sets of actions and strategies that have a successful record in resolving particular types of problems” [2, p. 35]. The strategies the users adopt to solve problems or find relevant information consist of a certain number of such interaction patterns. Such investigations are necessary for HCI in general, but they are especially relevant in information visualisation and visual analytics. In these areas, systems are developed which are supposed to support human reasoning processes and open-ended exploration specifically. Therefore, Pike et al.  argue that a science of interaction is necessary. They assume that interaction and cognition are closely coupled and that InfoVis should be designed as dialogic systems where both users and computers pose questions and answers. To design such dialogic systems it is necessary to investigate interaction processes in detail.
The following chapter discusses possible methods of analysis which are especially appropriate for this kind of investigation. In this context, researchers need methods able to represent the various activities which the users engage in while they work with information visualisations. Therefore, methods like interviews or questionnaires are not really appropriate because they cannot give a detailed overview of sequences of activities. In contrast to that, there are other methods like thinking aloud, eye tracking or log files which provide a fairly comprehensive overview of these activities. Categorisation of activities or utterances is necessary to be able to analyse the results of these methods. Based on these categorisations are mathematical approaches like transition matrices (see e.g. Ratwani et al. ) and Markov models. In this chapter, we will first describe these methods (eye tracking, log files, thinking aloud). We will discuss their advantages and disadvantages for the analysis of sequences of interactions with information visualisations. We will briefly discuss whether it is beneficial to combine two or more of these methods. Finally, we will present a few examples of the application of these methods in information visualisation and visual analytics. The discussion of the application examples is not exhaustive and can only give a brief idea of how these methods are applied in this area.
2 Eye Tracking
Eye tracking has been used quite extensively in human-computer interaction and usability research (see e.g. ). Goldberg and Wichansky  summarize usability recommendations based on eye tracking research. These recommendations concern screen elements such as icons, menus, navigation etc. There is also some relevant research on cognitive load .
A fundamental assumption of eye tracking research is the so-called eye-mind hypothesis [6, 7] which posits that the gaze direction is an indication of what the user is currently thinking about. There is empirical research indicating that this is not always the case. Duchowski  argues that peripheral vision also plays an important role in perception. Subjects often remember objects only seen in peripheral vision . Another difficulty in this context is, that it is sometimes challenging to infer what users really are thinking from gaze directions and scanpaths. If a user looks at an object on the screen for an extended period of time, this might indicate that the object is interesting or, on the other hand, that its functionality is not clear.
Eye tracking has several advantages . Eye tracking provides a large amount of objective data about users’ attention processes. Such processes sometimes are very fast and unconscious and, therefore, difficult to investigate. It is fairly unobtrusive (in contrast to, e.g., thinking aloud). Eye trackers also usually come with software which provides researchers with interesting visualisations (e.g. heat maps).
Advantages and disadvantages of eye tracking.
Large amount of data about users’ attention processes
Eye trackers already provide simple statistics and visualisations
Labour-intensive data extraction
Difficulties in data interpretation
Advantages for the analysis of interaction with InfoVis
Reflects attention to visual stimuli
Yields very detailed information about users’ scanning strategies
Challenges for the analysis of interaction with InfoVis
Generalisations of results from scanpaths difficult because of individual differences
Definition of AOIs
Investigation of exploratory tasks
Difficulty of the analysis of dynamic data
In addition, it should be pointed out that there are usually considerable individual differences concerning the users’ scanpaths . This makes generalisation of results of eye tracking studies difficult. A possible solution for this problem is to use small and well defined tasks with clear solutions to get comparable results. Such tasks are usually not very realistic. Results for explorative, open-ended tasks usually differ considerably.
Eye tracking is especially appropriate for investigating interaction with information visualisation tools because it reflects attention to visual stimuli. It can provide detailed information about the users’ scanning strategies . Eye tracking can show sequences of the users’ activities. The interaction processes of the users with the InfoVis (information visualisation) tool can be analysed in great detail.
There are some specific challenges concerning the usage of eye tracking in information visualisation. Defining areas of interest (AOIs) or regions of interest (ROI) is especially important for visualisations (we will use the terms interchangeably). AOIs should be related to the research question . Unluckily, it is not possible to provide rules for the definition of AOIs. Another problem is the fact that exploratory tasks are especially important in information visualisation and visual analytics. As mentioned above, it is difficult to investigate such tasks with eye tracking methods. In addition, it is still challenging to analyse dynamic data with eye tracking methodologies . Such data are highly relevant for information visualisation and visual analytics. To conclude, eye tracking seems to have great potential for the analysis of interactions with information visualisations and visual analytics, but it is not yet entirely clear how to use this potential.
3 Log Files
Log files are a very well-known methodology of research in Human-Computer Interaction . They were originally used for the analysis of so-called WIMP interfaces (window, icon, menu, pointer), but nowadays their main application area is the assessment of the usability of websites .
The addition of log files to software to analyse the users’ behaviour is called instrumenting . The goal of most usability studies using log files, in contrast to that, is to find usability errors and improve the interface. To reach this goal, log files produced automatically by servers or by off-the-shelf software is usually appropriate.
Ivory and Hearst  give an overview of various complex systems for the analysis of log files. They distinguish between automated capturing of usage data and the automated analysis of these data. Log file analysis in a narrow sense is the analysis of log file data based on metrics or a mathematical model. The system AMME developed by Rauterberg et al.  e.g. uses Markov models and Petri nets to investigate the users’ problem-solving processes.
Information visualisation systems often offer interaction possibilities going beyond navigation. Users are supported in zooming and panning, filtering the data, choosing different ways of representing data on the screen or other interaction activities (for an overview of categorisations of interactions see e.g. Gotz and Zhou , Yi et al. ). Logging all these activities can enable researchers to get some insights about the users’ cognitive processes. It is, for example, interesting to know that users sometimes concentrate on details of the visualisation and sometimes on overall aspects and prediction of the behaviour of the whole system . In addition, log files also provide information about the sequence of activities. This enables researchers to investigate whether there are patterns in the users’ behaviour. In this context, it is essential to decide which data to capture and how to aggregate these data . It is possible to collect data on a keystroke level, but in many cases such data is not very informative. Higher-level data might be more interesting. To decide which data to collect, a clear hypothesis about the cognitive processes involved in the interaction with the information visualisation system is often necessary. These data often have to be aggregated (e.g., when the same goal can be reached by several methods — menus or keyboard shortcuts) and categorised to be useful for an analysis of cognitive processes. Such an analysis process of log files is seldom described or discussed in the literature (Table 2).
Advantages and disadvantages of log files.
Less time consuming than other methodologies
Reflects actual behaviour of the users
Not intrusive, does not influence the users’ behaviour
Difficult to interpret the data (lack of context)
Advantages for the analysis of interaction with InfoVis
Provides detailed information about sequences of interaction
Especially appropriate for the analysis of interactive and explorative InfoVis tools
Challenges for the analysis of interaction with InfoVis
Identification of the appropriate level of granularity of interactions
Identification of the appropriate system of categorisation for the activities
4 Thinking Aloud
Thinking aloud is a methodology which was developed by Ericsson and Simon . The original goal was the investigation of cognitive processes, especially in the context of problem solving activities. It is difficult to analyse such processes exclusively on the basis of observation of visible behaviour because only the results of such activities can be perceived, not the process itself. Ericsson and Simon looked for a methodology to get more information about the processes which happen during problem solving and the strategies problem solvers adopt to reach their goals. This model is based on several assumptions. The theoretical context of thinking aloud is the information processing model of cognition (see e.g. ), a theoretical approach in psychology which uses computer based models as metaphors for the explanation of human cognition. Related to this is the assumption that thinking is a serial process which takes place in the working memory and the assumption that thinking aloud provides a complete overview of the cognitive processes. These assumptions are controversial (for a description of these discussions see e.g. ).
Boren and Ramey  give a comprehensive overview of problems which might arise when thinking aloud is applied in usability research. They point out that cognitive psychology is quite different to usability research, and some of the problems encountered when thinking aloud is used in usability research is due to that fact. In usability research, researchers often encounter system crashes or bugs in using the system which is being investigated. The consequence of such problems is that thinking aloud is interrupted. It is not clear whether an investigation where many such crashes and bugs happen really gives an accurate impression of the users’ thought processes. In addition, difficulties in using novel and unstable prototypes often necessitates that researchers talk to subjects to explain relevant issues of using the system. Such behaviour is not acceptable in the context of the original methodology. Boren and Ramey  suggest another theoretical framework — speech communication — to allow researchers to conduct more consistent and well defined investigations in usability. The theoretical framework they propose accommodates the current practice in usability research much better than the original theoretical positions formulated by Ericsson and Simon . In addition, we would like to point out that there is another problem with thinking aloud which was already mentioned by Ericsson and Simon . They point out that verbalisation is difficult when problems are presented in a physical form (e.g. the problem of the towers of Hanoi has to be solved by manipulating the disks physically). Subjects concentrate on the manipulation of the objects and are less able to verbalise their thought processes. Using a computer program to solve problems might be a similar situation. Users interact with an artifact and concentrate on mouse movements and navigation, and less on their thought processes.
Nevertheless, thinking aloud has significant advantages compared to methods like eye tracking and log file analysis. It gives insights into the thought processes, goals and motivation of the users of information visualisations. It also provides context to activities of the users and helps researchers to understand what strategies users adopt. Thinking aloud also provides a good impression of sequences of actions, although the granularity of these actions is usually more coarse than the one of log files and eye tracking.
Advantages and disadvantages of thinking aloud.
Direct investigation of cognitive processes during problem-solving behaviour
Information about thought processes, goals and motivation of the users
Difficult to use in the context of HCI 
Disruptive and unnatural
Advantages for the analysis of interaction with InfoVis
Provides detailed information about sequences of interaction
Provides more direct information about strategies and reasoning processes of users of InfoVis tools
Provides context for the interpretation of data
Challenges for the analysis of interaction with InfoVis
Adaptation of the model of Boren and Ramey  for the analysis of interaction with InfoVis tools
5 Mixing Methodologies
The methodologies described above all have strengths and weaknesses. It has often been suggested that a combination of these methodologies might yield more valid results. Lazar et al. , e.g., argue that log files are difficult to interpret and seldom provide contextual information about users and their cognitive processes. They suggest to combine log file analysis with video recordings (analysis of videos of users’ interaction processes with software) or direct observation to get more information about the problem at hand. Gilhooly and Green , on the other hand, suggest the combination of thinking aloud and log files. Log files enable the researchers to get more detailed and fine grained information about the users’ activities. Sometimes, such data can clarify what users meant with their utterances. Thinking aloud can also be combined with eye tracking  to be able to interpret the results of eye tracking studies. Subjects are often motivated to talk while they work with the system. A problem occuring in this context is that their eye movements are affected by thinking aloud, and eye movement recordings do not provide a realistic representation of what people are attending to any more. Webb and Renshaw suggest that retrospective methods should be applied in this case (e.g. discussing gaze plots with the users after the experiment). Holmqvist et al.  argue that the accuracy of eye tracking data especially suffers in the case of a remote eye tracking system because such systems cannot compensate the effects of fast head movements which usually accompany verbal behaviour. In the case of head-mounted systems, the problem is less serious.
Such combinations of methodologies enable researchers to get more detailed and accurate data than when only one single methodology is used. Such an approach is called triangulation (see e.g. ). This also increases the reliability and validity of the results. In their book on mixing methods, Creswell and Plano Clark  give an overview of how different research methodologies can be combined. Usually, quantitative and qualitative approaches are mixed.
One example for mixing methods is the study conducted by Jakobsen and Hornbaek . They combined grounded theory, thinking aloud, activity logging, probes and interviews. They argue that the methods in combination “provide stronger evidence of participants’ adoption and use” of a specific software. Activity logging, e.g., does not provide any information about the subjects’ intents and the context of their work. Thinking aloud, therefore, complements the data from activity logging.
6 Applications in Information Visualization
To illustrate how these methods could be used we will now give some examples of how they were already applied in the field of information visualisation and Visual Analytics.
6.1 Eye Tracking
Goldberg and Helfman , compared three different graph types using eye tracking. They defined Regions-of-Interest (Areas-of-Interest) for the sub-elements of the graphs concerned. They measured the time until their participants’ fixations first met the ROIs in which the information necessary for the given task was encoded. They also looked at sequences of fixations, observing, for example, that “The second viewed bar graph was generally scanned left to right (except the first AOI), but the first viewed bar graph was not regularly scanned in a particular direction” [11, p. 77].
A similar approach is used by Siirtola et al.  where ROIs are defined for elements of a parallel coordinate visualisation, divided in ROIs deemed necessary and relevant to a given task and those which are not. From the eye tracking data they derive the number of fixations that happened before a fixation in certain ROIs, as well as the total number of fixations and the total time spent in it, comparing these for ROI- and task-type, and relating their results to the interactions offered by the visualisation .
Another interesting way to use eye tracking is described in Convertino et al.  who investigate multiple-view visualisations in different configurations. They use the data to derive a measure of how often their participants moved their focus between views, and relate these values to other measurements taken .
Huang et al.  on the other hand, consciously avoided using eye tracking data for quantitative analysis, after observing that “currently available measures” are “difficult to relate [...] to specific graph elements such as nodes, links or paths” and decided to employ “Eye movement videos”, that is, a video of the screen content overlaid with a marker for the gaze direction during the experiment [30, p. 3:3]. By analysing these they developed new theories concerning human behaviour reading these graphs and then tested the theory by designing and implementing experiments employing classical time-and-error measurements .
6.2 Thinking Aloud
As with eye tracking there are different kind data and results one can get with thinking aloud. In evaluating information visualisation systems the concept of insight often is used, though definitions (for a discussion see, for example, Chang et al. ) as well as categorization approaches vary, depending among other things on the goals and questions of the study concerned.
Bautista and Carenini , for example, use this approach as part of a qualitative and quantitative triangulation (see Sect. 5) in an effort to demonstrate usefulness and improve an information visualisation tool for preferential choice.
HCI often uses time to complete tasks and number of errors as variables of analysis. For explorative tasks in InfoVis other variables are also interesting, e.g. the insights gained during interaction processes. Saraiya et al.  conducted a study comparing this kind of evaluation to the more classical method of measuring time and error in the information visualisation domain. They compared three information visualisation techniques of the same dataset and the two methods between subjects. Concerning the insights they use the number of insights (total and in categories, developed in cooperation with domain specialists) and the time spent in the open ended exploration scenario in an analysis of variance. They also note that even though they did not prompt their participants for it they additionally got feedback concerning the usability and visualisations in the insight condition, which might be another interesting factor to consider in the context of methodology choice .
Thinking aloud data is usually collected in sessions spanning hours at most, but it can also be a part of a longer term field study, as with Jakobsen and Hornbaek , who studied real-life workplace adoption of a source code visualisation tool spanning multiple weeks employing thinking aloud and logging methods, among others. Comparing and contrasting the thinking aloud results with the other methods, they mention that thinking aloud “showed use of the interface across a range of tasks, including some surprising ad-hoc uses” [27, p. 1586].
6.3 Log Files
Log files can clarify many open questions about the interaction processes of users with information visualisations. Bautista and Carenini , for example, mention that their log files revealed to them the reason why a certain task showed no significant difference in time-to-complete, even though this task required a functionality that was hidden in one condition and quite prominent in the other. According to them the log files showed that their participants traded the time gained by quicker access to the function for time spent using it .
A study conducted at our institute  is an example for a more in-depth report of a statistical analysis of log files created during a study of an information visualisation tool (Gravi++).
Building on this, a later study  explored the issue of emerging patterns of interactions. Beside the log file data the authors also analysed log files generated by another information visualisation application (VisuExplore) for comparison. While the specific interactions afforded by different tools usually also differ, the authors were able to use a variant of the general categorisation scheme proposed by Yi et al.  to group the logged events into higher-level activities, to see if the same general patterns could be found in both tools. To that end they qualitatively identified recurring patterns and also computed transition probabilities between categories . Based on the theory of Distributed Cognition, the authors interpret these patterns as indicators of cognitive activities.
In the study by Convertino et al.  already mentioned as an example for eye tracking investigations the authors also statistically analysed their log file data, using the number of different interactions as dependent measure and finding some significant effects for condition and task type, and also relating them to completion time measurements .
One advantage of using log files is that the data can be potentially collected over a longer period of time and that the method is relatively non-intrusive, therefore viable for medium or long term field studies, as Jakobsen and Hornbaek  demonstrated in the study already mentioned above concerning the in-the-field adoption of a fisheye source code visualisation. Visualising and quantitatively analysing the logs, they could observe the real life adoption of the visualisation component over time and in the context of the surrounding workflow .
In this paper, we describe three methods for the analysis of interactions with information visualisations. All these methods have advantages and disadvantages. A possibility to overcome the disadvantages would be to combine some of these methods. A careful study of the users’ interaction processes with information visualisations could provide more information about cognitive processes accompanying interaction with information visualisation.
We think that in the future it is necessary to discuss methodological problems regarding the investigation of interactions with information visualisations in more detail. Important research has been done to clarify research methods for the evaluation of information visualisations in general . The investigation of interaction sequences is, however, a very specific problem requiring more detailed research.
This work is conducted in the context of the CVAST - Centre of Visual Analytics Science and Technology project. It is funded by the Austrian Federal Ministry of Economy, Family and Youth in the exceptional Laura Bassi Centres of Excellence initiative.
- 1.Preece, J., Rogers, Y., Sharp, H.: Interaction Design. Wiley, New York (2002)Google Scholar
- 2.Mirel, B.: Interaction Design for Complex Problem Solving. Elsevier/Morgan Kauffman, Amsterdam, Boston (2004)Google Scholar
- 5.Jacob, R.J.K., Karn, K.S.: Eye tracking in human-computer interaction and usability research: ready to deliver the promises. In: Hyona, J., Radach, R., Deubel, H. (eds.) The Mind’s Eye - Cognitive and Applied Aspects of Eye Movement Research, pp. 573–03. Elsevier, Amsterdam (2003)Google Scholar
- 6.Goldberg, J.H., Wichansky, A.: Eye tracking in usability evaluation: a practitioner’s guide. In: Hyona, J., Radach, R., Deubel, H. (eds.) The Mind’s Eye - Cognitive and Applied Aspects of Eye Movement Research, pp. 493–516. Elsevier, Amsterdam (2003)Google Scholar
- 7.Webb, N., Renshaw, T.: Eyetracking in HCI. In: Cairns, P., Cox, A.L. (eds.) Research Methods for Human-Computer Interaction, pp. 35–69. Cambridge University Press, Cambridge (2008)Google Scholar
- 8.Duchowski, A.: Eye Tracking Methodology - Theory and Practice, 2nd edn. Springer, London (2007)Google Scholar
- 9.Holmqvist, K., Nystrom, M., Andersson, R., Dewhurst, R., Jarodzka, H., van de Weijer, J.: Eye Tracking - A Comprehensive Guide to Methods and Measures. Oxford University Press, Oxford (2011)Google Scholar
- 10.Nielsen, J., Pernice, K.: Eyetracking Web Usability. New Riders Press, Berkeley (2010)Google Scholar
- 11.Goldberg, J.H., Helfman, J.I.: Comparing information graphics: a critical look at eye tracking. In: Proceedings of the 3rd BELIV’10 Workshop: BEyond Time and Errors: Novel EvaLuation Methods for Information Visualization, BELIV ’10, pp. 71–78. ACM, New York (2010)Google Scholar
- 12.Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., Carey, T.: Human-Computer Interaction. Addison Wesley, Reading (1994)Google Scholar
- 13.Kuniavsky, M.: Observing the User Experience - A Practitioner’s Guide to User Research. Morgan Kaufmann, San Francisco (2003)Google Scholar
- 15.Lazar, J., Feng, J.H., Hochheiser, H.: Research Methods in Human-Computer Interaction. Wiley, Chichester (2010)Google Scholar
- 17.Rauterberg, M.: A petri net based analysing and modelling tool kit for logfiles in human-computer interaction. In: Proceedings of Cognitive Systems Engineering in Process Control - CSEPC ’96 Conference, pp. 268–275 (1996)Google Scholar
- 21.Ericsson, K.A., Simon, H.A.: Protocol Analysis - Verbal Reports as Data - Revised Edition. MIT Press, Cambridge (1993)Google Scholar
- 22.Anderson, J.: Kognitive Psychologie, 3rd edn. Spektrum Akademischer Verlag, Heidelberg (2001)Google Scholar
- 23.Gilhooly, K., Green, C.: Protocol analysis: theoretical background. Handbook of Qualitative Research Methods for Psychology and the Social Sciences, pp. 43–54. BPS Blackwell, Malden (1996)Google Scholar
- 25.Gilhooly, K., Green, C.: Protocol analysis: practical implications? In: Richardson, J.T.E. (ed.) Handbook of Qualitative Research Methods for Psychology and the Social Sciences, pp. 55–74. BPS Blackwell, Malden (1996)Google Scholar
- 26.Creswell, J., Plano Clark, V.: Designing and Conducting Mixed Methods Research. Sage, Los Angeles (2011)Google Scholar
- 27.Jakobsen, M.R., Hornbaek, K.: Fisheyes in the field: using method triangulation to study the adoption and use of a source code visualization. In: Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHI ’09, pp. 1579–1588. ACM, New York (2009)Google Scholar
- 28.Siirtola, H., Laivo, T., Heimonen, T., Raiha, K.J.: Visual perception of parallel coordinate visualizations. In: 13th International Conference on Information Visualisation, 2009, pp. 3–9, July 2009Google Scholar
- 29.Convertino, G., Chen, J., Yost, B., Ryu, Y.S., North, C.: Exploring context switching and cognition in dual-view coordinated visualizations. In: Proceedings of International Conference on Coordinated and Multiple Views in Exploratory Visualization, pp .55–62, July 2003Google Scholar
- 30.Huang, W., Eades, P., Hong, S.H.: Beyond time and error: a cognitive approach to the evaluation of graph drawings. In: Proceedings of the 2008 Conference on BEyond Time and Errors: Novel EvaLuation Methods for Information Visualization, BELIV ’08, pp. 3:1–3:8. ACM, New York (2008)Google Scholar
- 32.Bautista, J., Carenini, G.: An empirical evaluation of interactive visualizations for preferential choice. In: Proceedings of the Working Conference on Advanced Visual Interfaces, AVI ’08, pp. 207–214. ACM, New York (2008)Google Scholar
- 33.Saraiya, P., North, C., Duca, K.: Comparing benchmark task and insight evaluation methods on timeseries graph visualizations. In: Proceedings of the 3rd BELIV’10 Workshop: BEyond Time and Errors: Novel EvaLuation Methods for Information Visualization, BELIV ’10, pp. 55–62. ACM, New York (2010)Google Scholar
- 34.Pohl, M., Wiltner, S., Miksch, S.: Exploring information visualization: describing different interaction patterns. In: Proceedings of the 3rd BELIV’10 Workshop: BEyond Time and Errors: Novel EvaLuation Methods for Information Visualization, BELIV ’10, pp. 16–23. ACM, New York (2010)Google Scholar
- 35.Carpendale, S.: Evaluating information visualization. In: Kerren, A., Stasko, J.T., Fekete, J.-D., North, C. (eds.) Information Visualization - Human-Centered Issues and Perspectives. Springer, Heidelberg (2008)Google Scholar