Keywords

1 Introduction

The physical world around us is well known to most of us, we spent our early years learning about space, mass, movement, and direction. We often take space for granted so much that it comprises the framework of our way of speaking [1]. A user’s familiarity with real spaces makes designing virtual spaces problematic. Our innate sense of spatial orientation in the real word is undermined by cumbersome technology and the use of input/output devices when engaging in virtual environments. That said, the future looks bright—trends in games, virtual/mixed reality, robotics, and artificial intelligence promise to bring us closer to rich, technology-mediated experiences popularized by science fiction [2].

Designers and developers of technologies often cite popular culture and science fiction such as Ready Player One and Star Trek as sources of inspiration [3]. For example, Gene Roddenberry’s Star Trek series—popularized by American media and Hollywood—has been a cult phenomenon in the United States celebrated for decades by scientists, artists, and technologists alike. Audiences around the world have been intrigued by the potential of Star Trek “technologies” to change our society, relationships, and abilities. The holodeck, first introduced in a 1974 animated episode of Star Trek called “Practical Joker,” and later in the television series Star Trek Next Generation, has widely influenced discussions about the design and use of immersive, computationally augmented collaboration environments [4]. A holodeck is a smart virtual/architectural hybrid space that incorporates voice actuated computer interaction, artificial intelligence, storytelling, and holographic display of information. Holodeck simulations can be distinguished from reality only by their limitless programmability. The Star Trek Next Generation holodeck was the epitome of an interactive storyworld of illusions that could be stopped, started, redirected, recombined, and reused at will [5]. The holodeck is in fact “too good to be true.”

Behind the holodeck were some highly paid Hollywood scriptwriters expert at narrative and timing. By design, the holodeck was created to stretch audience imaginations beyond their boundaries of the physically familiar and technologically possible. Usability was seamless and the technology transparent in the Star Trek Next Generation holodeck. Unfortunately, virtual (VR), augmented (AR), and mixed reality (MR) environments, together often referred to as XR or cross reality [6], have not yet been particularly successful at achieving the high technical ideals set by literally adopting a holodeck metaphor.

Nevertheless, science fiction and the notion of the holodeck can be useful when applied in the socio-technical context of general user expectations [7]. The holodeck introduced audiences to possible social implications of futuristic non-traditional human-computer interfaces for learning, conveying information, and collaboration. For example, by immersing oneself in the holodeck environment, crewmembers simulated problem situations, evaluated alternative solutions, employed new tools, and painlessly explored the consequences of life-altering decisions. The holodeck induced provocative changes in the behaviors of the crew of the Starship Enterprise while exploring what the future could hold for human communication, cognition, and creativity. In fact, the holodeck technology was so seamless, it was practically invisible. That is to say, it was not necessarily the pointer to futuristic technologies of the holodeck that inspired audiences around the globe, but rather the narratives, or stories, created in it.

Although the holodeck is a powerful metaphor for the way computer simulations, artificial intelligence, XR, and the design of advanced information displays can augment human learning and collaboration, like any metaphor, it must be interpreted with care. What aspects of the holodeck are most important to learning through discovery or collaborative problem solving? Possibilities from which we might have chosen include the availability of real time simulations; practically unlimited access to information, bandwidth, and artificially intelligent collaborators; and the freeing of computational power from the encumbrances of screens, head mounted displays (HMDs), smartphones, touchpads, keyboards, mice, controllers, gloves, trackers, and other 3D input devices.

More importantly, however, in the present paper the holodeck metaphor is extended to illustrate how information designers, developers, and engineers may achieve some of the qualities of this imaginary computationally augmented environment in order to support real life, face-to-face and virtual human learning, collaboration, and digital storytelling that is information-rich, right now. Information design is a multidisciplinary endeavor involving data science, communication science, cognitive science, media, and technology [8]. A brief description of the holodeck metaphor serves as a device to explore how XR information design is a social construction of narrative. Second, the Simulation Experience Design Framework & Method [9] is introduced to organize key concepts relevant to designing XR environments and their associated challenges and opportunities. The notion of carefully blending both real and virtual spaces to achieve total immersion is discussed as the reader moves through the elements of the cyclical framework. Last, the conclusions describe a solution space leveraging applied cognitive science, information design, and transmedia learning that is neither high-tech nor low-tech, but all-tech. Transmedia learning is defined as the scalable system of messages representing a narrative or core experience that unfolds from the use of multiple media, emotionally engaging learners by involving them personally in the story [10]. As previously noted by [10, 11], the challenge facing contemporary XR designers and developers is to interleave information, technology, and multiple media into the human storytelling process, and implement it in a way that is memorable, robust, and extendable.

1.1 The Social Construction of Narrative

The meaning of any metaphor emerges from an interaction between the metaphor’s basis (in this case, the holodeck) and the goals, assumptions and constraints of its interpreters. In applying the holodeck metaphor to the design of collaborative XR environments, the authors’ interpretive bias is to de-emphasize the advanced technologies it describes in favor of the collaborative interactions it potentially supports. One of the dangers of the holodeck metaphor is that it might steer us toward trying to replicate Star Trek technology, while ignoring practicality, usability, human performance, learning, ethics, and the deeper structure of human collaborative work or play. This could easily lead to the construction of yet another technical showpiece, filled with costly, soon to be obsolete, hardware that is good for little more than carefully orchestrated demonstrations. If we are to build useful and meaningful XR environments, the human dimension must drive our design decisions.

“Social construction of narrative” is the creation, by a group of people, of systematic, coherent structures for organizing shared knowledge and developing future knowledge. Although the goal of social construction of narrative is applicable to nearly all forms of collaborative information work, the emphasis on narrative as a social construction is particularly relevant to the design of information in immersive experiences. User experience (UX) design addresses the synthesis of cognitive science, human-computer interaction, communication, and design thinking. User experience design puts the human at the center of the product or service design process. As XR designers and developers create more immersive XR and persistent transmedia learning [10] experiences, it can be useful to employ a metaphor to bring to the fore initial assumptions, biases, or notions of expectations integral to the design of immersive information experiences.

Narrative plays a powerful role in virtually all forms of human problem solving, theory formation, creative work, and play [12]. For example, examining seminal work in artificial intelligence has long recognized the power of scripts and other narrative structures in creating and organizing knowledge [13, 14]. Laurel [15] has shown that human computer interfaces can be improved by paying attention to the narrative structure of the interaction activity. Narrative has even been shown to underlie the formation of scientific theory. Historical studies of scientific practice confirm the role of metaphor and analogy in theory formation [16]; these processes derive their power from their narrative-like ability to organize knowledge into a systematic structure. Landau [17] offers further support for the role of narrative in science by analyzing various versions of the theory of evolution, to show that all of them have a common narrative structure that mirrors the universal hero myth.

Therefore, the position taken in the present paper is that it is not the promise of technology that ultimately appeals to users; it is idea of co-creating and living out stories. More than the promise of artificial intelligence, simulation, or information visualization, it is the support for social narrative construction and creativity that is the source of the metaphor’s power. This understanding gives us a basis for elaborating the holodeck metaphor in ways that may be relevant to information designers and developers of immersive transmedia learning experiences leveraging XR and other media.

2 Simulation Experience Design Framework & Method

Organizations tell stories to share learning, strategies, and knowledge. Daily operations, the communication of scientific results, and data science analyses are story-driven endeavors. Scenario and problem-based learning with simulations and games, in particular, often leverage story-driven experiential learning. The Simulation Experience Design Framework & Method [9] is a process that addresses design as a system of experiences that exists within an emergent, adaptive, cultural context that the designer shapes and facilitates throughout user engagement, before, during, between, and after the core experience has concluded. The word simulation in the name of the method refers to an experience in which the role of a human, environment, or both, can be simulated. The Simulation Experience Design Framework & Method, briefly described in the present paper, has been applied by the author and others to problem-based learning in virtual environments, serious game design [9, 18], and transmedia learning [10]. User experience design for XR (or any medium for that matter) requires that designers and developers understand what makes a good experience first, and then translate these principles, as efficiently as possible, into the desired medium without the technologies dictating the form of the experience. In simulated environments in which end users are creatively problem solving or playing together, one’s experience may be unpredictable, may not have a right or wrong approach, or may not be what the designer intends. The Simulation Experience Design Framework & Method can be helpful in framing the co-creation of open-ended, rich systems of experiences that fosters learning, understanding, and sense-making (Fig. 1).

Fig. 1.
figure 1

Simulation Experience Design Framework [9]

Full size image

The Simulation Experience Design Framework & Method suggests that supporting equitable intercultural communication and learning is comprised of several salient elements, among them (1) the interactions or type of communication (interpersonal, group, etc.), (2) the narratives that are co-created by interlocutors, (3) the place, or context, in which narratives occur, and (4) the culture that emerges from the social construction of experience [19]. Following the circular framework from upper left to upper right, design may then be considered as facilitating a journey, or connected learning experience from interactions to emergent culture that iteratively lead to new interactions spawned by the emergent culture. Use of the framework and method is intended to improve the quality of equitable intercultural communication and learning in collaborative, immersive environments such as XR, serious games, simulations, transmedia storytelling & learning ecosystems [9, 10].

Finally, by treating intercultural communication as a core value, the individual cultural backgrounds the players bring to their experiences are considered strengths, not design liabilities. As we strive to create engaging immersive XR experiences approaching the holodeck, differing cultural values of designers, developers, stakeholders, and players can create a myriad of complications and competing desires or expectations. The Simulation Experience Design Framework & Method can serve to socially construct narratives and establish a shared understanding for thoughtful analysis from which to better ground assessment and evaluation of human performance, creativity, and expertise.

3 Challenges and Opportunities in XR Information Design

The following sections describe some of the challenges and “low hanging fruit” opportunities in information design for engaging storytelling and socially constructing narratives in XR immersive environments. Challenges and opportunities are presented as the reader moves through the elements of the cyclical Simulation Experience Design Framework & Method [9]. As designers and developers strive to blend real and virtual spaces to achieve total immersion, key challenges include but are not limited to inter-leaving information, technology, and media into the human storytelling process, and supporting diverse cultural narratives in a way that is equitable, memorable, robust, and extendable.

3.1 Interaction

Challenge: The Communication Space.

Human communication is comprised of systems of utterances, acts, and messages that are verbal, nonverbal, and incorporate each of the senses. After the novelty has worn off, virtual environments are usually less interesting and less appealing to the senses than the real world. What can we do to make XR spaces more appealing to the senses and more easily inhabitable? We know that people can make spaces more interesting, and, conversely, empty spaces may be prone to bore users—how can we make use of the emptiness often associated with virtual environments?

Opportunity: Support Quiet Reflection and Active Immersion.

An immersive environment can encourage an appropriate mixing of virtual and co-present end users for public, private (alone or two), and semi-private (small group) interactions. Although we often equate immersion with activity, designing for quiet reflection can enhance learning, or problem-solving experiences. The holodeck allowed for active and reflective behavior, both physically and virtually. The “coordinated use of mind, language, and body is a fulfilling mode of being in the world” [20, p. 193]. A space that is experienced through reflection and action can enhance immersion and engagement, such that the space becomes a “place.” Additionally, telepresent creativity should also value the silent pauses between verbal and nonverbal communication as much as it values the communication itself. While proximity creates presence without constant communication, the telepresent space must, like the holodeck, support co-present reflection as much as co-present active communication.

Supporting XR immersion requires we be able to represent the emerging narrative (e.g., problem solution) in the shared physical space. Users may want to actively manipulate data, or quietly inhabit the same physical or virtual space while working side-by-side with colleagues. This suggests a judicious integration of both high-tech (XR or wall-size displays for AR data simulations and shared virtual spaces) and low-tech (white boards and butcher paper) representations.

For example, a review of AR research suggests its potential in the design of 3D and 2D information for quiet reflection and active immersion. AR refers to the display of virtual elements alongside those in the real world [21]. The real/virtual juxtaposition of information can be used when visualization would otherwise be difficult, such as the inclusion of 3D components alongside 2D media [22]. This combination of digital and physical elements creates immersion that facilitates critical thinking, problem solving, and communication [23]. Research further indicates that the introduction of AR can improve understanding and memory of material [24, 25] as well as raise the level of engagement during presentation of information [26].

Another potential benefit of AR is the reduction in cognitive load, or the mental effort involved in task performance [27]. Cognitive load may be divided into three types: intrinsic (related to the inherent level of difficulty in task performance), germane (the construction of schemas that facilitate task performance), and extraneous or incidental (effort required based on the method of presentation or instruction) [28]. While intrinsic cognitive load is beyond the manipulation of an instructor, and germane cognitive load is desirable as it relates to identification of problem-solving strategies, extraneous cognitive load can and should be minimized in order to reserve cognitive resources to process the intrinsic and germane aspects of a task [29]. Extraneous cognitive load is thought to be reduced with proper AR implementation [24, 27].

Using electroencephalography (EEG), theta activity can be used as an index of cognitive load [30], sustained vigilance [31], and can be used to build a classifier to detect mental fatigue [32]. Employing theta activity as an index of attention, fatigue, and cognitive load can allow for the comparison of XR versus 2D information processing in terms of how hard individuals are working to process complex information, how engaged they are attentionally, how much mental fatigue is present, and ultimately, how well information is understood and recalled. Advancements in neuroscience, data science, and communication provide the scientific foundation for information design of quiet reflection and active immersion supported by XR and environments employing transmedia learning to blend real and virtual elements.

3.2 Narrative

Challenge: The Storytelling Space.

Although we think of immersion as a property of advanced computer interface technology, immersion is actually a fundamental property of narratives that goes back earlier than Homer (e.g. The Iliad and The Odyssey). All good stories can draw us in to the virtual worlds they create. How can we facilitate the simultaneous co-creation of and immersion in a shared narrative?

Opportunity: Achieve Co-created Immersion With and Through Interactivity.

The medium of film has deeply immersive qualities and is both symbolic and spatial. Movies achieve immersion through fixed narrative, the representation of physical space and realistic audio, yet are also symbolic in that the viewer sees from a “Gods eye” view. Movies compress time to suit one’s limited attention span and warp psychological and physical distances to suit narrative flow. A film is not a “true” representation of reality, but it is compelling nonetheless. However, to explore the notion of the viewer as a co-creator of narrative we may look at how computer game technology achieves immersion with and through interactivity. Early pioneers like Brenda Laurel and Hal Barwood pointed out that computer games are more like plays, not films [19, 33]. Like plays, scenes in computer games are often viewed from single angles and from the same distance. Additionally, as actors leave the stage, audience members know that they still exist and are not out of the context of the plot. A computer game is similar in that the player buys into a narrative of off-screen armies plotting against him or her while s/he is battling evil aliens on-screen. Thus, the action of the game takes place on and off the screen [19]. Games are stories that are co-created by the player [5].

The goal of information design of immersive narratives is to combine the symbolic narratives of movies and the co-creating nature of games into a space that support flexible creative relationships and improvisation that characterize learning and human creative problem solving. Previous research points to real spaces that support creativity and innovation [20], and the way problem solvers use objects as part of the creative process [34]. Narratives often find their way into physical spaces—via storytelling artifacts and other attempts to capture an experience in a more permanent fashion. An immersive, information-rich problem space should provide for the spatializing of narratives and artifacts necessary to facilitate innovation and creativity.

Preliminary exploration at the authors’ institution in spatializing problem-solving narratives in XR environments, involves bringing legacy simulation frameworks into XR applications to create visceral experiences and provide learning takeaways unachievable through a traditional PC interface. For example, in the domain of physical security, 3D simulation frameworks have been developed for analyzing the effectiveness of security system layouts. Dante is a simulation framework that constructs a system-of-systems model of a facility, including accurate 3D representations of the facility along with modeling security assets [35]. Dante relies on its intelligent agent architecture to simulate thousands of possibilities through Monte Carlo simulation runs.

An AR prototype was developed to explore whether problem-solving “what if” narratives supported by Dante could become more hands-on engineering design aids with the integration of Microsoft HoloLens. The prototype enabled users to explore the placement of virtual security assets (objects) within an indoor space. Some virtual assets (e.g., simulated cameras) provided information on the sensor viewshed within the deployed space and updated the viewshed information in real-time as the objects within the physical space were moved. A viewshed is the visible area of a physical space from a sensor’s vantage point. For example, a camera may be able to view a parking lot but would not see what’s behind a parked car. The area in front of the parked car is within the viewshed, but the space behind the car would be outside of it. The application also supported a “what if” narrative which involved an attempt by a non-player character (NPC) to find, and take simulated objects placed by the end user in the indoor space. The NPC constructed its navigation plan based upon the physical layout instead of a human-generated 3D model of the facility. The prototype demonstrated that Dante agents and sensor modeling libraries may enable XR applications to access predictive simulation tools thereby adding additional dimension to the co-creation of a shared narrative to aid decision making.

3.3 Place

Challenge: The Space Isn’t a Place.

An immersive information space is not just a display of data; it should be a place where people act.

Opportunity: Achieve Immersion With Contextual Cues.

Learning is situated in implicit cultural and contextual information. This includes assumptions, values, goals, meanings, and history shared by learners. This information is often implicit and comprises the tacit knowledge that is learned through interactions over time. When collaborating, especially with those who are not co-located, we should attempt to include tacit knowledge in our computational space, although much essential tacit knowledge never successfully makes it into current computational environments.

The first author and others [36] have shown that introducing subtle cultural and contextual cues into immersive environments is an effective way to encourage certain group collaboration. Cultural and contextual cues may be embedded in 3D information, simulated objects, models and data from real-world experiments, posters, or each other. Worth further exploration is a comparison of human collaboration with virtual objects or data within the XR environment, to a real-world corollary which splits attention among data in the XR environment, physical objects outside the XR environment, and a collaborator inside and/or outside of the XR environment. Being co-located in the same virtual space may allow collaborators to explore spaces intuitively, transition focus more easily, and more naturally interact with physical entities in the real world.

Designers can also carefully design places that provide clues about the unfolding narrative by stimulating memories and emotions. Research by LeDoux [37] suggests that the human brain is wired to pick up on messages crafted as stories because we feel real emotions when we connect with content or a character in a story. One potential explanation is that the brain uses two mnemonic systems to process information [37]. The brain processes information both rationally and emotionally, although emotions about rational content are usually processed by the brain split seconds before rational or logical interventions by the cerebral cortex. Put simply, LeDoux’s research indicates we best remember information presented in the form of a story.

Information and user experience designers can accommodate learners’ need for cultural cues and situated information through a deep understanding of social work structures, careful application of intercultural communication principles, and a willingness to accept the possibility that when faced with a choice, end users may recognize limitations of XR and prefer that some activities happen outside the immersive environment [9].

3.4 Emergent Culture

Challenge: The Space for Diversity.

Immersive narrative technologies, especially simulation, have a tendency to present a single point of view so powerfully, and with such an illusion of reality, that other points of view are lost. One of the less fortunate effects of information and communication media is their tendency to homogenize different points of view. How can we ensure that learning in collaborative immersive environments celebrate creative differences and reward out-of-the-box thinking?

Opportunity: Support Multiple Points of View and Perceptions in Multiple Spaces.

In parallel with our growing acceptance of virtual spaces, we can begin to explore merging a spatial and symbolic paradigm. An XR space that is populated with intelligent agents may adapt to one’s perspective, cognitive perception, and indeed crucially, to specific communities and communication preferences. The medium may be the message, but we should also remember that in designing message-rich environments, the message is more important than the medium! In transmedia learning the interaction with messages communicated by each individual medium (XR, etc.) should reinforce performance improvement, reflection, and behavior change. Emergent culture is an opportunity to explore the broader story, or socially constructed narrative, in different ways to enrich the core transmedia learning experience [10]. We can use agents to tailor information and personalize the medium based on an understanding of the way learners perceive information, the narratives they co-create, and knowledge of how users juggle peripheral and focused information in multiple spaces [38].

4 Conclusion

We have captured what we believe to be the essential appeal of the holodeck metaphor, and also the essential constraints on XR environments. Recall that after the novelty has subsided, it is not the technology that appeals to users; it is idea of creating and living out stories with colleagues. This understanding provides a basis for elaborating the holodeck metaphor in ways that may be relevant to XR information design. The Simulation Experience Design Framework & Method was introduced to organize challenges and opportunities in the design of information for XR environments. The discussion of carefully blending both real and virtual spaces to achieve total immersion leveraged research from cognitive science, neuroscience, information design, and transmedia learning. Key challenges facing XR designers include but are not limited to inter-leaving information, technology, and media into the human storytelling process, and supporting diverse cultural narratives in a way that is equitable, memorable, robust, and extendable.