Decision making with visualizations: a cognitive framework across disciplines
- 213 Downloads
Visualizations—visual representations of information, depicted in graphics—are studied by researchers in numerous ways, ranging from the study of the basic principles of creating visualizations, to the cognitive processes underlying their use, as well as how visualizations communicate complex information (such as in medical risk or spatial patterns). However, findings from different domains are rarely shared across domains though there may be domain-general principles underlying visualizations and their use. The limited cross-domain communication may be due to a lack of a unifying cognitive framework. This review aims to address this gap by proposing an integrative model that is grounded in models of visualization comprehension and a dual-process account of decision making. We review empirical studies of decision making with static two-dimensional visualizations motivated by a wide range of research goals and find significant direct and indirect support for a dual-process account of decision making with visualizations. Consistent with a dual-process model, the first type of visualization decision mechanism produces fast, easy, and computationally light decisions with visualizations. The second facilitates slower, more contemplative, and effortful decisions with visualizations. We illustrate the utility of a dual-process account of decision making with visualizations using four cross-domain findings that may constitute universal visualization principles. Further, we offer guidance for future research, including novel areas of exploration and practical recommendations for visualization designers based on cognitive theory and empirical findings.
KeywordsDecision making with visualizations review Cognitive model Visual-spatial biases Graphs Geospatial visualizations Healthcare visualizations Weather forecast visualizations Uncertainty visualizations Graphical decision making Dual-process
People use visualizations to make large-scale decisions, such as whether to evacuate a town before a hurricane strike, and more personal decisions, such as which medical treatment to undergo. Given their widespread use and social impact, researchers in many domains, including cognitive psychology, information visualization, and medical decision making, study how we make decisions with visualizations. Even though researchers continue to develop a wealth of knowledge on decision making with visualizations, there are obstacles for scientists interested in integrating findings from other domains—including the lack of a cognitive model that accurately describes decision making with visualizations. Research that does not capitalize on all relevant findings progresses slower, lacks generalizability, and may miss novel solutions and insights. Considering the importance and impact of decisions made with visualizations, it is critical that researchers have the resources to utilize cross-domain findings on this topic. This review provides a cognitive model of decision making with visualizations that can be used to synthesize multiple approaches to visualization research. Further, it offers practical recommendations for visualization designers based on the reviewed studies while deepening our understanding of the cognitive processes involved when making decisions with visualizations.
Every day we make numerous decisions with the aid of visualizations, including selecting a driving route, deciding whether to undergo a medical treatment, and comparing figures in a research paper. Visualizations are external visual representations that are systematically related to the information that they represent (Bertin, 1983; Stenning & Oberlander, 1995). The information represented might be about objects, events, or more abstract information (Hegarty, 2011). The scope of the previously mentioned examples illustrates the diversity of disciplines that have a vested interest in the influence of visualizations on decision making. While the term decision has a range of meanings in everyday language, here decision making is defined as a choice between two or more competing courses of action (Balleine, 2007).
We argue that for visualizations to be most effective, researchers need to integrate decision-making frameworks into visualization cognition research. Reviews of decision making with visual-spatial uncertainty also agree there has been a general lack of emphasis on mental processes within the visualization decision-making literature (Kinkeldey, MacEachren, Riveiro, & Schiewe, 2017; Kinkeldey, MacEachren, & Schiewe, 2014). The framework that has dominated applied decision-making research for the last 30 years is a dual-process account of decision making. Dual-process theories propose that we have two types of decision processes: one for automatic, easy decisions (Type 1); and another for more contemplative decisions (Type 2) (Kahneman & Frederick, 2002; Stanovich, 1999).1 Even though many research areas involving higher-level cognition have made significant efforts to incorporate dual-process theories (Evans, 2008), visualization research has yet to directly test the application of current decision-making frameworks or develop an effective cognitive model for decision making with visualizations. The goal of this work is to integrate a dual-process account of decision making with established cognitive frameworks of visualization comprehension.
In this paper, we present an overview of current decision-making theories and existing visualization cognition frameworks, followed by a proposal for an integrated model of decision making with visualizations, and a selective review of visualization decision-making studies to determine if there is cross-domain support for a dual-process account of decision making with visualizations. As a preview, we will illustrate Type 1 and 2 processing in decision making with visualizations using four cross-domain findings that we observed in the literature review. Our focus here is on demonstrating how dual-processing can be a useful framework for examining visualization decision-making research. We selected the cross-domain findings as relevant demonstrations of Type 1 and 2 processing that were shared across the studies reviewed, but they do not represent all possible examples of dual-processing in visualization decision-making research. The review documents each of the cross-domain findings, in turn, using examples from studies in multiple domains. These cross-domain findings differ in their reliance on Type 1 and Type 2 processing. We conclude with recommendations for future work and implications for visualization designers.
Decision-making researchers have pursued two dominant research paths to study how humans make decisions under risk. The first assumes that humans make rational decisions, which are based on weighted and ordered probability functions and can be mathematically modeled (e.g. Kunz, 2004; Von Neumann, 1953). The second proposes that people often make intuitive decisions using heuristics (Gigerenzer, Todd, & ABC Research Group, 2000; Kahneman & Tversky, 1982). While there is fervent disagreement on the efficacy of heuristics and whether human behavior is rational (Vranas, 2000), there is more consensus that we can make both intuitive and strategic decisions (Epstein, Pacini, Denes-Raj, & Heier, 1996; Evans, 2008; Evans & Stanovich, 2013; cf. Keren & Schul, 2009). The capacity to make intuitive and strategic decisions is described by a dual-process account of decision making, which suggests that humans make fast, easy, and computationally light decisions (known as Type 1 processing) by default, but can also make slow, contemplative, and effortful decisions by employing Type 2 processing (Kahneman, 2011). Various versions of dual-processing theory exist, with the key distinctions being in the attributes associated with each type of process (for a more detailed review of dual-process theories, see Evans & Stanovich, 2013). For example, older dual-systems accounts of decision making suggest that each process is associated with specific cognitive or neurological systems. In contrast, dual-process (sometimes termed dual-type) theories propose that the processes are distinct but do not necessarily occur in separate cognitive or neurological systems (hence the use of process over system) (Evans & Stanovich, 2013).
Identifying processes that require significant working memory provides a definition of Type 2 processing with observable neural correlates. Therefore, in line with Evans and Stanovich (2013), in the remainder of this manuscript, we will use significant working memory capacity demands and significant need for cognitive control, as defined above, as the criterion for Type 2 processing. In the context of visualization decision making, processes that require significant working memory are those that depend on the deliberate application of working memory to function. Type 1 processing occurs outside of users’ conscious awareness and may utilize small amounts of working memory but does not rely on conscious processing in working memory to drive the process. It should be noted that Type 1 and 2 processing are not mutually exclusive and many real-world decisions likely incorporate all processes. This review will attempt to identify tasks in visualization decision making that require significant working memory and capacity (Type 2 processing) and those that rely more heavily on Type 1 processing, as a first step to combining decision theory with visualization cognition.
Then viewers use a match process, where the graph schema that is the most similar to the visual array is retrieved. When a matching graph schema is found, the schema becomes instantiated. The visualization conventions associated with the graph schema can then help the viewer interpret the visualization (message assembly process). For example, Fig. 3 illustrates comprehension of a bar chart using the Pinker (1990) model. In this example, the matched graph schema for a bar graph specifies that the dependent variable is on the Y-axis and the independent variable is on the X-axis; the instantiated graph schema incorporates the visual description and this additional information. The conceptual message is the resulting mental representation of the visualization that includes all supplemental information from long-term memory and any mental transformations the viewer may perform on the visualization. Viewers may need to transform their mental representation of the visualization based on their task or conceptual question. In this example, the viewer’s task is to find the average of A and B. To do this, the viewer must interpolate information in the bar chart and update the conceptual message with this additional information. The conceptual question can guide the construction of the mental representation through interrogation, which is the process of seeking out information that is necessary to answer the conceptual question. Top-down encoding mechanisms can influence each of the processes.
The influences of top-down processes are also emphasized in a previous attempt by Patterson et al. (2014) to extend visualization cognition theories to decision making. The Patterson et al. (2014) model illustrates how top-down cognitive processing influences encoding, pattern recognition, and working memory, but not decision making or the response. Patterson et al. (2014) use the multicomponent definition of working memory, proposed by Baddeley and Hitch (1974) and summarized by Cowan (2017) as a “multicomponent system that holds information temporarily and mediates its use in ongoing mental activities” (p. 1160). In this conception of working memory, a central executive controls the functions of working memory. The central executive can, among other functions, control attention and hold information in a visuo-spatial temporary store, which is where information can be maintained temporally for decision making without being stored in long-term memory (Baddeley & Hitch, 1974).
While incorporating working memory into a visualization decision-making model is valuable, the Patterson et al. (2014) model leaves some open questions about relationships between components and processes. For example, their model lacks a pathway for working memory to influence decisions based on top-down processing, which is inconsistent with well-established research in decision science (e.g. Gigerenzer & Todd, 1999; Kahneman & Tversky, 1982). Additionally, the normal processing pathway, depicted in the Patterson model, is an oversimplification of the interaction between top-down and bottom-up processing that is documented in a large body of literature (e.g. Engel, Fries, & Singer, 2001; Mechelli, Price, Friston, & Ishai, 2004).
A proposed integrated model of decision making with visualizations
Empirical studies of visualization decision making
Application area for the tasks used in the reviewed studies
Task application area
Meteorology, weather, and natural disaster forecasting, weather communication
Health research, medical images, health risk communication
Ancker et al. (2006); Fagerlin et al. (2005); Garcia-Retamero and Galesic (2009); Keehner et al. (2011); Keller et al. (2009); McCabe and Castel (2008); Okan et al. (2015); Okan, Garcia-Retamero, Cokely, and Maldonado (2012); Schirillo and Stone (2005); Stone et al. (2003); Stone et al. (1997); Waters et al. (2006); Waters et al. (2007)
Land-use planning, spatial planning, urban planning
Cost comparison, finance
Error-bar interpretation, graph comparison, statistics communication, science reasoning
Map reading, map perception
Social network, computer connections
Map-based threat identification, emergency management
Overview of the four cross-domain findings along with the type of processing that they reflect
Evidence for Type
Visualizations direct viewers’ bottom-up attention, which can both help and hinder decision making.
The visual encoding technique gives rise to visual-spatial biases.
Visualizations that have greater cognitive fit produce faster and more effective decisions.
Knowledge-driven processes can interact with the effects of the encoding technique.
Visual features used in the reviewed studies to attract bottom-up attention
In sum, the reviewed studies suggest that bottom-up attention has a profound influence on decision making with visualizations. This is noteworthy because bottom-up attention is a Type 1 process. At a minimum, the work suggests that Type 1 processing influences the first stages of decision making with visualizations. Further, the studies cited in this section provide support for the inclusion of bottom-up attention in our proposed model.
Biases documented in the reviewed studies
Belia et al. (2005)
Fagerlin et al. (2005)
Schirillo and Stone (2005)
Side effect aversion
We argue that visual-spatial biases reflect a Type 1 process, occurring automatically with minimal working memory. Work by Sanchez and Wiley (2006) provides direct evidence for this assertion using eye-tracking data to demonstrate that individuals with less working memory capacity attend to irrelevant images in a scientific article more than those with greater working memory capacity. The authors argue that we are naturally drawn to images (particularly high-quality depictions) and that significant working memory capacity is required to shift focus away from images that are task-irrelevant. The ease by which visualizations captivate our focus and direct our bottom-up attention to specific features likely increases the impact of these biases, which may be why some visual-spatial biases are notoriously difficult to override using working memory capacity (see Belia et al., 2005; Boone, Gunalp, & Hegarty, in press; Joslyn & LeClerc, 2013; Newman & Scholl, 2012). We speculate that some visual-spatial biases are intertwined with bottom-up attention—occurring early in the decision-making process and influencing the down-stream processes (see our model in Fig. 4 for reference), making them particularly unremitting.
Decision-making components that the reviewed studies evaluated the cognitive fit of
Cognitive fit examined
Visualization - > task
Visualization - > primed schema
Tversky et al. (2012)
Visualization - > learned schema
The aforementioned studies provide direct (Zhu & Watts, 2010) and indirect (Dennis & Carte, 1998; Feeney et al., 2000; Gattis & Holyoak, 1996; Huang et al., 2006; Joslyn & LeClerc, 2013; Smelcer & Carmel, 1997; Tversky et al., 2012; Vessey & Galletta, 1991) evidence that Type 2 processing recruits working memory to resolve misalignment between decision-making processes and the visualization that arise from default Type 1 processing. These examples of Type 2 processing using working memory to perform effortful mental computations are consistent with the assertions of Evans and Stanovich (2013) that Type 2 processes enact goal directed complex processing. However, it is not clear from the reviewed work how exactly the visualization and decision-making components are matched. Newman and Scholl (2012) propose that we match the schema and visualization based on the similarities between the salient visual features, although this proposal has not been tested. Further, work that assesses cognitive fit in terms of the visualization and task only examines the alignment of broad categories (i.e., spatial or semantic). Beyond these broad classifications, it is not clear how to predict if a task and visualization are aligned. In sum, there is not a sufficient cross-disciplinary theory for how mental schemas and tasks are matched to visualizations. However, it is apparent from the reviewed work that Type 2 processes (requiring working memory) can be recruited during the schema matching and inference processes.
Either type 1 and/or 2
In a review of map-reading cognition, Lobben (2004) states, “…research should focus not only on the needs of the map reader but also on their map-reading skills and abilities” (p. 271). In line with this statement, the final cross-domain finding is that the effects of knowledge can interact with the affordances or biases inherent in the visualization method. Knowledge may be held temporally in working memory (Type 2), held in long-term knowledge but effortfully used (Type 2), or held in long-term knowledge but automatically applied (Type 1). As a result, knowledge-driven processing can involve either Type 1 or Type 2 processes.
Both short- and long-term knowledge can influence visualization affordances and biases. However, it is difficult to distinguish whether Type 2 processing is using significant working memory capacity to temporarily hold knowledge or if participants have stored the relevant knowledge in long-term memory and processing is more automatic. Complicating the issue, knowledge stored in long-term memory can influence decision making with visualizations using both Type 1 and 2 processing. For example, if you try to remember Pythagorean’s Theorem, which you may have learned in high school or middle school, you may recall that a2 + b2 = c2, where c represents the length of the hypotenuse and a and b represent the lengths of the other two sides of a triangle. Unless you use geometry regularly, you likely had to strenuously search in long-term memory for the equation, which is a Type 2 process and requires significant working memory capacity. In contrast, if you are asked to recall your childhood phone number, the number might automatically come to mind with minimal working memory required (Type 1 processing).
Type of knowledge examined in each study
Short-term training, instructions
Galesic and Garcia-Retamero (2011); Galesic et al. (2009) Keller et al. (2009) Okan et al. (2015); Okan, Garcia-Retamero, Galesic, and Cokely (2012); Okan, Garcia-Retamero, Cokely, and Maldonado (2012); Okan, Garcia-Retamero, Galesic, and Cokely (2012); Reyna et al. (2009); Rodríguez et al. (2013)
Lee and Bednarz (2009)
These studies are good examples of how designers can create visualizations that capitalize on Type 1 processing to help viewers accurately make decisions with complex data even when they lack relevant knowledge. Based on the reviewed work, we speculate that well-designed visualizations that utilize Type 1 processing to intuitively illustrate task-relevant relationships in the data may be particularly beneficial for individuals with less numeracy and graph literacy, even for simple tasks. However, poorly designed visualizations that require superfluous mental transformations may be detrimental to the same individuals. Further, individual differences in expertise, such as graph literacy, which have received more attention in healthcare communication (Galesic & Garcia-Retamero, 2011; Nayak et al., 2016; Okan et al., 2015; Okan, Garcia-Retamero, Cokely, & Maldonado, 2012; Okan, Garcia-Retamero, Galesic, & Cokely, 2012; Rodríguez et al., 2013), may play a large role in how viewers complete even simple tasks in other domains such as map-reading (Kinkeldey et al., 2017).
Less consistent are findings on how more experienced users incorporate knowledge acquired over longer periods of time to make decisions with visualizations. Some research finds that students’ decision-making and spatial abilities improved during a semester-long course on Geographic Information Science (GIS) (Lee & Bednarz, 2009). Other work finds that experts perform the same as novices (Riveiro, 2016), experts can exhibit visual-spatial biases (St. John et al., 2001) and experts perform more poorly than expected in their domain of visual expertise (Belia et al., 2005). This inconsistency may be due in part to the difficulty in identifying when and if more experienced viewers are automatically applying their knowledge or employing working memory. For example, it is unclear if the students in the GIS course documented by Lee and Bednarz (2009) developed automatic responses (Type 1) or if they learned the information and used working memory capacity to apply their training (Type 2).
Interesting future work could limit experts’ time to complete a task (forcing Type 1 processing) and then determine if their judgments change when given more time to complete the task (allowing for Type 2 processing). To test this possibility further, a dual-task paradigm could be used such that experts’ working memory capacity is depleted by a difficult secondary task that also required working memory capacity. Some examples of secondary tasks in a dual-task paradigm include span tasks that require participants to remember or follow patterns of information, while completing the primary task, then report the remembered or relevant information from the pattern (for a full description of theoretical bases for a dual-task paradigm see Pashler, 1994). To our knowledge, only one study has used a dual-task paradigm to evaluate cognitive load of a visualization decision-making task (Bandlow et al., 2011). However, a growing body of research on other domains, such as wayfinding and spatial cognition, demonstrates the utility of using dual-task paradigms to understand the types of working memory that users employ for a task (Caffò, Picucci, Di Masi, & Bosco, 2011; Meilinger, Knauff, & Bülthoff, 2008; Ratliff & Newcombe, 2005; Trueswell & Papafragou, 2010).
In sum, this section documents cases where knowledge-driven processing does and does not influence decision making with visualizations. Notably, we describe numerous studies where well-designed visualizations (capitalizing on Type 1 processing) focus viewers’ attention on task-relevant relationships in the data, which improves decision accuracy for individuals with less developed health literacy, graph literacy, and numeracy. However, the current work does not test how knowledge-driven processing maps on to the dual-process model of decision making. Knowledge may be held temporally by working memory capacity (Type 2), held in long-term knowledge but strenuously utilized (Type 2), or held in long-term knowledge but automatically applied (Type 1). More work is needed to understand if a dual-process account of decision making accurately describes the influence of knowledge-driven processing on decision making with visualizations. Finally, we detailed an example of a dual-task paradigm as one way to evaluate if viewers are employing Type 1 processing.
Throughout this review, we have provided significant direct and indirect evidence that a dual-process account of decision making effectively describes prior findings from numerous domains interested in visualization decision making. The reviewed work provides support for specific processes in our proposed model including the influences of working memory, bottom-up attention, schema matching, inference processes, and decision making. Further, we identified key commonalities in the reviewed work relating to Type 1 and Type 2 processing, which we added to our proposed visualization decision-making model. The first is that utilizing Type 1 processing, visualizations serve to direct participants’ bottom-up attention to specific information, which can be either beneficial or detrimental for decision making (Fabrikant et al., 2010; Fagerlin et al., 2005; Hegarty et al., 2010; Hegarty et al., 2016; Padilla, Ruginski et al., 2017; Ruginski et al., 2016; Schirillo & Stone, 2005; Stone et al., 1997; Stone et al., 2003; Waters et al., 2007). Consistent with assertions from cognitive science and scientific visualization (Munzner, 2014), we propose that visualization designers should identify the critical information needed for a task and use a visual encoding technique that directs participants’ attention to this information. We encourage visualization designers who are interested in determining which elements in their visualizations will likely attract viewers’ bottom-up attention, to see the Itti et al. (1998) saliency model, which has been validated with eye-tracking measures (for implementation of this model along with Matlab code see Padilla, Ruginski et al., 2017). If deliberate effort is not made to capitalize on Type 1 processing by focusing the viewer’s attention on task-relevant information, then the viewer will likely focus on distractors via Type 1 processing, resulting in poor decision outcomes.
A second cross-domain finding is the introduction of a new concept, visual-spatial biases, which can also be both beneficial and detrimental to decision making. We define this term as a bias that elicits heuristics, which is a direct result of the visual encoding technique. We provide numerous examples of visual-spatial biases across domains (for implementation of this model along with Matlab code, see Padilla, Ruginski et al., 2017). The novel utility of identifying visual-spatial biases is that they potentially arise early in the decision-making process during bottom-up attention, thus influencing the entire downstream process, whereas standard heuristics do not exclusively occur at the first stage of decision making. This possibly accounts for the fact that visual-spatial biases have proven difficult to overcome (Belia et al., 2005; Grounds et al., 2017; Joslyn & LeClerc, 2013; Liu et al., 2016; McKenzie et al., 2016; Newman & Scholl, 2012; Padilla, Ruginski et al., 2017; Ruginski et al., 2016). Work by Tversky (2011) presents a taxonomy of visual-spatial communications that are intrinsically related to thought, which are likely the bases for visual-spatial biases.
We have also revealed cross-domain findings involving Type 2 processing, which suggest that if there is a mismatch between the visualization and a decision-making component, working memory is used to perform corrective mental transformations. In scenarios where the visualization is aligned with the mental schema and task, performance is fast and accurate (Joslyn & LeClerc, 2013). The types of mismatches observed in the reviewed literature are likely both domain-specific and domain-general. For example, situations where viewers employ the correct graph schema for the visualization, but the graph schema does not align with the task, are likely domain-specific (Dennis & Carte, 1998; Frownfelter-Lohrke, 1998; Gattis & Holyoak, 1996; Huang et al., 2006; Joslyn & LeClerc, 2013; Smelcer & Carmel, 1997; Tversky et al., 2012). However, other work demonstrates cases where viewers employ a graph schema that does not match the visualization, which is likely domain-general (e.g. Feeney et al., 2000; Gattis & Holyoak, 1996; Tversky et al., 2012). In these cases, viewers could accidentally use the wrong graph schema because it appears to match the visualization or they might not have learned a relevant schema. The likelihood of viewers making attribution errors because they do not know the corresponding schema increases when the visualization is less common, such as with uncertainty visualizations. When there is a mismatch, additional working memory is required resulting in increased time taken to complete the task and in some cases errors (e.g. Joslyn & LeClerc, 2013; McKenzie et al., 2016; Padilla, Ruginski et al., 2017). Based on these findings, we recommend that visualization designers should aim to create visualizations that most closely align with a viewer’s mental schema and task. However, additional empirical research is required to understand the nature of the alignment processes, including the exact method we use to mentally select a schema and the classifications of tasks that match visualizations.
The final cross-domain finding is that knowledge-driven processes can interact or override effects of visualization methods. We find that short-term (Dennis & Carte, 1998; Feeney et al., 2000; Gattis & Holyoak, 1996; Joslyn & LeClerc, 2013; Smelcer & Carmel, 1997; Tversky et al., 2012) and long-term knowledge acquisition (Shen et al., 2012) can influence decision making with visualizations. However, there are also examples of knowledge having little influence on decisions, even when prior knowledge could be used to improve performance (Galesic et al., 2009; Galesic & Garcia-Retamero, 2011; Keller et al., 2009; Lee & Bednarz, 2009; Okan et al., 2015; Okan, Garcia-Retamero, Cokely, & Maldonado, 2012; Okan, Garcia-Retamero, Galesic, & Cokely, 2012; Reyna et al., 2009; Rodríguez et al., 2013). We point out that prior knowledge seems to have more of an effect on non-visual-spatial biases, such as a familiarity bias (Belia et al., 2005; Joslyn & LeClerc, 2013; Riveiro, 2016; St. John et al., 2001), which suggests that visual-spatial biases may be closely related to bottom-up attention. Further, it is unclear from the reviewed work when knowledge switches from relying on working memory capacity for application to automatic application. We argue that Type 1 and 2 processing have unique advantages and disadvantages for visualization decision making. Therefore, it is valuable to understand which process users are applying for specific tasks in order to make visualizations that elicit optimal performance. In the case of experts and long-term knowledge, we propose that one interesting way to test if users are utilizing significant working memory capacity is to employ a dual-task paradigm (illustrated in Fig. 19). A dual-task paradigm can be used to evaluate the amount of working memory required and compare the relative working memory required between competing visualization techniques.
Identify the critical information needed for a task and use a visual encoding technique that directs participants’ attention to this information (“Bottom-up attention” section);
Aim to create visualizations that most closely align with a viewer’s mental schema and task demands (see “Visual-Spatial Biases”);
Work to reduce the number of transformations required in the decision-making process (see "Cognitive fit");
To understand if a viewer is using Type 1 or 2 processing employ a dual-task paradigm (see Fig. 19);
Consider evaluating the impact of individual differences such as graphic literacy and numeracy on visualization decision making.
We use visual information to inform many important decisions. To develop visualizations that account for real-life decision making, we must understand how and why we come to conclusions with visual information. We propose a dual-process cognitive framework expanding on visualization comprehension theory that is supported by empirical studies to describe the process of decision making with visualizations. We offer practical recommendations for visualization designers that take into account human decision-making processes. Finally, we propose a new avenue of research focused on the influence of visual-spatial biases on decision making.
This research is based upon work supported by the National Science Foundation under Grants 1212806, 1810498, and 1212577.
Availability of data and materials
No data were collected for this review.
LMP is the primary author of this study; she was central to the development, writing, and conclusions of this work. SHC, MH, and JS contributed to the theoretical development and manuscript preparation. All authors read and approved the final manuscript.
LMP is a Ph.D. student at the University of Utah in the Cognitive Neural Science department. LMP is a member of the Visual Perception and Spatial Cognition Research Group directed by Sarah Creem-Regehr, Ph.D., Jeanine Stefanucci, Ph.D., and William Thompson, Ph.D. Her work focuses on graphical cognition, decision making with visualizations, and visual perception. She works on large interdisciplinary projects with visualization scientists and anthropologists.
SHC is a Professor in the Psychology Department of the University of Utah. She received her MA and Ph.D. in Psychology from the University of Virginia. Her research serves joint goals of developing theories of perception-action processing mechanisms and applying these theories to relevant real-world problems in order to facilitate observers’ understanding of their spatial environments. In particular, her interests are in space perception, spatial cognition, embodied cognition, and virtual environments. She co-authored the book Visual Perception from a Computer Graphics Perspective; previously, she was Associate Editor of Psychonomic Bulletin & Review and Experimental Psychology: Human Perception and Performance.
MH is a Professor in the Department of Psychological & Brain Sciences at the University of California, Santa Barbara. She received her Ph.D. in Psychology from Carnegie Mellon University. Her research is concerned with spatial cognition, broadly defined, and includes research on small-scale spatial abilities (e.g. mental rotation and perspective taking), large-scale spatial abilities involved in navigation, comprehension of graphics, and the role of spatial cognition in STEM learning. She served as chair of the governing board of the Cognitive Science Society and is associate editor of Topics in Cognitive Science and past Associate Editor of Journal of Experimental Psychology: Applied.
JS is an Associate Professor in the Psychology Department at the University of Utah. She received her M.A. and Ph.D. in Psychology from the University of Virginia. Her research focuses on better understanding if a person’s bodily states, whether emotional, physiological, or physical, affects their spatial perception and cognition. She conducts this research in natural settings (outdoor or indoor) and in virtual environments. This work is inherently interdisciplinary given it spans research on emotion, health, spatial perception and cognition, and virtual environments. She is on the editorial boards for the Journal of Experimental Psychology: General and Virtual Environments: Frontiers in Robotics and AI. She also co-authored the book Visual Perception from a Computer Graphics Perspective.
Ethics approval and consent to participate
The research reported in this paper was conducted in adherence to the Declaration of Helsinki and received IRB approval from the University of Utah, #IRB_00057678. No human subject data were collected for this work; therefore, no consent to participate was acquired.
Consent for publication
Consent to publish was not required for this review.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- Bailey, K., Carswell, C. M., Grant, R., & Basham, L. (2007). Geospatial perspective-taking: how well do decision makers choose their views? In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 51, No. 18, pp. 1246-1248). Los Angeles: SAGE Publications.Google Scholar
- Bandlow, A., Matzen, L. E., Cole, K. S., Dornburg, C. C., Geiseler, C. J., Greenfield, J. A., … Stevens-Adams, S. M. (2011). Evaluating Information Visualizations with Working Memory Metrics. In HCI International 2011–Posters’ Extended Abstracts, (pp. 265–269).Google Scholar
- Bertin, J. (1983). Semiology of graphics: Diagrams, networks, maps. Madison: University of Wisconsin Press.Google Scholar
- Boone, A., Gunalp, P., & Hegarty, M. (in press). Explicit versus Actionable Knowledge: The Influence of Explaining Graphical Conventions on Interpretation of Hurricane Forecast Visualizations. Journal of Experimental Psychology: Applied.Google Scholar
- Caffò, A. O., Picucci, L., Di Masi, M. N., & Bosco, A. (2011). Working memory components and virtual reorientation: A dual-task study. In Working memory: capacity, developments and improvement techniques, (pp. 249–266). Hauppage: Nova Science Publishers.Google Scholar
- Card, S. K., Mackinlay, J. D., & Shneiderman, B. (1999). Readings in information visualization: using vision to think. San Francisco: Morgan Kaufmann Publishers Inc.Google Scholar
- Castro, S. C., Strayer, D. L., Matzke, D., & Heathcote, A. (2018). Cognitive Workload Measurement and Modeling Under Divided Attention. Journal of Experimental Psychology: General.Google Scholar
- Engle, R. W., Kane, M. J., & Tuholski, S. W. (1999). Individual differences in working memory capacity and what they tell us about controlled attention, general fluid intelligence, and functions of the prefrontal cortex. In A. Miyake & P. Shah (Eds.), Models of working memory: Mechanisms of active maintenance and executive control (pp. 102-134). New York: Cambridge University Press.Google Scholar
- Fabrikant, S. I., & Skupin, A. (2005). Cognitively plausible information visualization. In Exploring geovisualization, (pp. 667–690). Oxford: Elsevier.Google Scholar
- Feeney, A., Hola, A. K. W., Liversedge, S. P., Findlay, J. M., & Metcalf, R. (2000). How people extract information from graphs: Evidence from a sentence-graph verification paradigm. In International Conference on Theory and Application of Diagrams (pp. 149-161). Berlin, Heidelberg: Springer.Google Scholar
- Frownfelter-Lohrke, C. (1998). The effects of differing information presentations of general purpose financial statements on users’ decisions. Journal of Information Systems, 12(2), 99–107.Google Scholar
- Garcia-Retamero, R., & Galesic, M. (2009). Trust in healthcare. In Kattan (Ed.), Encyclopedia of medical decision making, (pp. 1153–1155). Thousand Oaks: SAGE Publications.Google Scholar
- Gigerenzer, G., Todd, P. M., & ABC Research Group (2000). Simple Heuristics That Make Us Smart. Oxford: Oxford University Press.Google Scholar
- Grounds, M. A., Joslyn, S., & Otsuka, K. (2017). Probabilistic interval forecasts: An individual differences approach to understanding forecast communication. Advances in Meteorology, 2017, 1-18.Google Scholar
- Harel, J. (2015, July 24, 2012). A Saliency Implementation in MATLAB. Retrieved from http://www.vision.caltech.edu/~harel/share/gbvs.php
- Kahneman, D. (2011). Thinking, fast and slow. (Vol. 1). New York: Farrar, Straus and Giroux.Google Scholar
- Kinkeldey, C., MacEachren, A. M., Riveiro, M., & Schiewe, J. (2017). Evaluating the effect of visually represented geodata uncertainty on decision-making: Systematic review, lessons learned, and recommendations. Cartography and Geographic Information Science, 44(1), 1–21. https://doi.org/10.1080/15230406.2015.1089792.CrossRefGoogle Scholar
- Kunz, V. (2004). Rational choice. Frankfurt: Campus Verlag.Google Scholar
- Lallanilla, M. (2014, April 24, 2014 10:15 am). Misleading Gun-Death Chart Draws Fire. https://www.livescience.com/45083-misleading-gun-death-chart.html
- Liu, L., Boone, A., Ruginski, I., Padilla, L., Hegarty, M., Creem-Regehr, S. H., … House, D. H. (2016). Uncertainty Visualization by Representative Sampling from Prediction Ensembles. IEEE transactions on visualization and computer graphics, 23(9), 2165-2178.Google Scholar
- Lobben, A. K. (2004). Tasks, strategies, and cognitive processes associated with navigational map reading: A review perspective. The Professional Geographer, 56(2), 270–281.Google Scholar
- Munzner, T. (2014). Visualization analysis and design. Boca Raton, FL: CRC Press.Google Scholar
- Okan, Y., Galesic, M., & Garcia-Retamero, R. (2015). How people with low and high graph literacy process health graphs: Evidence from eye-tracking. Journal of Behavioral Decision Making.Google Scholar
- Padilla, L., Ruginski, I. T., & Creem-Regehr, S. H. (2017). Effects of ensemble and summary displays on interpretations of geospatial uncertainty data. Cognitive Research: Principles and Implications, 2(1), 40.Google Scholar
- Pinker, S. (1990). A theory of graph comprehension. In Artificial intelligence and the future of testing, (pp. 73–126).Google Scholar
- Ratliff, K. R., & Newcombe, N. S. (2005). Human spatial reorientation using dual task paradigms. Paper presented at the Proceedings of the Annual Cognitive Science Society.Google Scholar
- Rodríguez, V., Andrade, A. D., García-Retamero, R., Anam, R., Rodríguez, R., Lisigurski, M., … Ruiz, J. G. (2013). Health literacy, numeracy, and graphical literacy among veterans in primary care and their effect on shared decision making and trust in physicians. Journal of Health Communication, 18(sup1), 273–289.PubMedPubMedCentralCrossRefGoogle Scholar
- Shah, P., Freedman, E. G., & Vekiri, I. (2005). The Comprehension of Quantitative Information in Graphical Displays. In P. Shah (Ed.) & A. Miyake, The Cambridge Handbook of Visuospatial Thinking (pp. 426-476). New York: Cambridge University Press.Google Scholar
- Sloman, S. A. (2002). Two systems of reasoning. In T. Gilovich, D. Griffin, & D. Kahneman (Eds.), Heuristics and biases: The psychology of intuitive judgment (pp. 379-396). New York: Cambridge University Press.Google Scholar
- Stanovich, K. E. (1999). Who is rational? Studies of individual differences in reasoning. New York City: Psychology Press.Google Scholar
- Stone, E. R., Yates, J. F., & Parker, A. M. (1997). Effects of numerical and graphical displays on professed risk-taking behavior. Journal of Experimental Psychology: Applied, 3(4), 243.Google Scholar
- Tversky, B. (2005). Visuospatial reasoning. In K. Holyoak and R. G. Morrison (eds.), The Cambridge Handbook of Thinking and Reasoning, (pp. 209-240). Cambridge: Cambridge University Press.Google Scholar
- Tversky, B., Corter, J. E., Yu, L., Mason, D. L., & Nickerson, J. V. (2012). Representing Category and Continuum: Visualizing Thought. Paper presented at the International Conference on Theory and Application of Diagrams, Berlin, Heidelberg.Google Scholar
- Vessey, I., Zhang, P., & Galletta, D. (2006). The theory of cognitive fit. In Human-computer interaction and management information systems: Foundations, (pp. 141–183).Google Scholar
- Von Neumann, J. (1953). Morgenstern, 0.(1944) theory of games and economic behavior. Princeton, NJ: Princeton UP.Google Scholar
- Wilkening, J., & Fabrikant, S. I. (2011). How do decision time and realism affect map-based decision making? Paper presented at the International Conference on Spatial Information Theory.Google Scholar
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.