Keywords

1 Introduction

Digital computing has been pervasively penetrating into people’s everyday life. Growing research suggests that the vision of interaction design should extend from digital artifacts to everyday environments where interactive technologies are embedded and hence support embodied interaction [14, 16, 18]. Ambient media, as a novel media form, incorporate interactive technology into physical spaces, which creates various affordances for engaging audience in bodily and social interactions. While many studies have focused on designing for task-oriented interactions, few studies have centered on meaning making in interaction with ambient media. “Ambient” has its phenomenological essence grounded in embodiment, and embodied interactions are anchored in human’ intrinsic familiarity with “ambiently embodied interfaces” [18]. The relationship between ambient media and embodiment reflects a loop of intentional arc, where perception and action are tightly interconnected with together. This study is to address the following questions: What major factors can be identified in audience experience? How do bodily movements facilitate embodied metaphorical thinking and imagination? We introduce a high-fidelity prototype for experimental study. Qualitative data is collected through observation and semi-structured interviews. We summarize qualitative results of participants’ primary experience in terms of six themes. A model of movement-based meaning making is developed to illustrate the cognitive processing of embodied conceptual mapping and schema-based imagination.

2 Research Background

In this section, we briefly examine three main concepts as the theoretical basis, and related studies are also reviewed.

Embodied cognition emphasizes that human body’s state plays a central role in shaping cognition [1, 12], languages [13, 15, 17], understanding digital media [7], and dancer’s emotion [8]. This is against traditional Cartesian dualism about separation of mind-body which overlook the influence of bodily experience on perception and reasoning. Human language and imagination are rooted in bodily experience and constrained by a range of embodied schemas, which is formed from people’s recurring physical interactions with environments [15]. Embodied metaphors are the concrete extensions of a schema. When talking about abstract concepts, people usually use some expressions linked to bodily interactions to represent abstract concepts. People who did bodily actions corresponding to the metaphors can better understand verbal phrases than those performing actions not relevant to the metaphors and those not performing any actions [22]. Clay et al. [8] investigated emotional expression of movement-based improvisations by using augmented technology, which scientifically shows that dancer’s bodily movements directly affect emotion and expression.

Dourish [10] coined embodied interaction to illustrate meaning-making as a process of physical and social interactions. The interaction between human and machine is embedded in a composition of both physical habits and social (cultural) experience, and embodied experience is created from direct and situated engagement with living environments. Antle et al. [3] believe that people’s early bodily experiences (e.g., keeping balance on ground) help to form the BALANCE schema which can be applied to understand more abstract ideas (e.g., He balances his emotion). The embodied metaphors refer to those of conceptual metaphors rooted in an embodied schema arising from everyday repeated bodily interactions [2, 4]. The projection between source domain and target domain constructs an embodied conceptual mapping. Empirical studies partly prove that the meaning expression of product is also structured by embodied schemas, and user experience with products is highly related to bodily experience [21]. Building upon embodied interaction, the previous studies have explored the relationship between bodily engagement with ambient media and four experiential qualities [19, 20].

Distinctively differing from traditional media, ambient media are embedded in public spaces where people are more likely to be immersed in it with intuitive actions [14, 19]. There are three main attributes of ambient media: The novel physical forms (Spatiality) of ambient media attract attention of audience and provide affordances for bodily actions. The audience may experience unexpected feedbacks (Unexpectedness) when being engaged in interactions with ambient media (Engagement). Audience experience is an active process of engagement where the perceptual, emotional, kinesthetic, and cognitive responses are derived from interactions [5, 6, 20].

3 Research Design and Methods

To investigate the impacts of bodily movements on audience’s metaphorical thinking and imagination, a high-fidelity prototype called Piano Staircase was developed to enable participants to freely perform different actions with auditory feedbacks. The goal of conducting audience experience test is to gather empirical data for understanding the role of bodily movements in audience experience.

3.1 Design Criteria and Experience Prototyping

The criteria of prototyping are aimed to match the three attributes of ambient media. Firstly, in order to transform physical space into interaction interface, the prototype is proposed to be physically and socially situated in a public space rather than a controlled laboratory. The second criterion is to attract audience’s attention by evoking a surprising response, which is a neutral psychological status giving rise to emotional experience. Thirdly, ambient media should engage audience’s bodily actions by creating affordances that can potentially solicit interactions. Inspired by “Piano Stairs” [9], we utilized the staircase in a public space of a university building to set up the prototype “Piano Staircase” (PS) (Fig. 1). Aligned with design criteria, the prototype was developed like a real piano, and each stair corresponded to a certain musical note.

Fig. 1.
figure 1

Piano staircase (based on the DDB’s work [9]).

Full size image

3.2 Participants and Procedure

We invited 30 participants (13 males and 17 females) to participate in the experience tests. Their ages ranged from 18 to 35. Twenty-two were locals of Hong Kong, seven Mainland Chinese, one Indian. Most participants were recruited from a university in Hong Kong except for three passers randomly invited at site. Firstly, participants were informed that they can feel free to experience a physical space, and they can choose to quit the test at any time. A consent form was signed by them (informing them video recording would be used). Then, participants can freely move on the staircase, and their behaviors were videotaped by a hidden camera, aiming to reduce distracting their attention. Each participant interacted with the work for 3–6 min, and the duration depended on their own intention. After the interaction session, the participant immediately joined a face-to-face interview. Semi-structured interview questions were prepared to motivate participants to express their feelings and thinking. The recorded video was played during the interview to help them recall their experience. The dialogues between researchers and participants were recorded as audio files for further analysis.

3.3 Data Analysis

This study mainly analyzed the verbal reports about first-person experience, while video data served as evidence for observational behaviors. We conducted thematic analysis to identify themes related to embodied experience. Interview audios were transcribed as text files. After that we used HyperResearch to do coding by following bottom-up and top-down approaches [11]. Regarding the research questions, the researchers viewed the transcripts, made notes and marked key words to look for potential patterns. Then, the words or sentences related to the prior codes (with round dots in Fig. 2) were firstly highlighted in the coding book where the meaning of each code was also described (e.g., Engagement: feeling being physically and mentally engaged in interaction with it). Next, inductive codes were generated from verbal data rather than the prior template. These codes were updated with the coding progress. When a new code was defined, the researchers went back to check the previous transcripts for coding for possible segments. Finally, the segments labeled with same code were compared and across all cases for some refinements, and all codes were rechecked and organized.

Fig. 2.
figure 2

Themes and codes.

Full size image

4 Results

The coding scheme consists of six themes highlighted in bold (Fig. 2). Twenty codes were identified and defined for further analyzing participants’ embodied experience and engagement patterns.

We summarized results in terms of the six themes as follows.

  • Perceptual Experience and Bodily Experience

All participants started to experience the Piano Staircase (PS) with their perceptual responses. Many participants got their initial impression through visual and auditory perceptions. All participants noticed the features of music changing during the interaction process, such as pitch, rhythm, duration and continuity.

Although the interview questions did not directly ask participants to talk about their bodily movements, all participants described their bodily experience related to various actions, speed, force. The most frequent code is “Bodily Movement” with 75 times (Mean frequency is 2.5 per case). Many participants tried different actions (e.g., stride, jump, stamp) rather then just stepping. Five participants skipped over one or two stair(s) by taking a long stride. Some participants stayed on the staircase for a while or keep in a posture, such as opening arms for balance and keeping in a posture like standing.

  • Embodied Metaphor and Cognitive Experience

The descriptions of participants reflected different embodied mappings between metaphorical concepts and physical movements. For example, after a period of adapting to the work (walking up and down), most of the participants had a mapping between musical pitches and levels of stairs. P3 said: “After I heard the first sound, I wanted to walked up following the stairs. Then I also found it had a pattern like a spectrum of notes of keyboard.” Four participants (P4, P11, P16, P28) described that they experienced the changes of tempos when walking at different speeds. Some participants experienced a sense of force when walking in different directions. The mentioned words include “heavy”, “intense”, “relaxed”, “unconstrained”, “light”, “subside”.

The theme of cognitive experience includes six codes: Attention, Evaluation, Anticipation, Desire for exploration, Recall, and Imagination. Evaluation: Participants constantly evaluated the interactive experience at different moments. Four participants mentioned that the distance between two stairs (e.g., the first and the fourth) were too far, and they can not reach it through stretching legs to generate a tune or chord. Anticipation: Some participants, who declared they had some piano performance skill, had an intention to play a composition. Apart from playing compositions, two participants (P8, P9) intended to stepping on all the stairs and to see the musical effects. Imagination: When being asked about what brands could be associated with PS and their imaginations on the current experience, many participants mentioned physical exercises and sports brands. Two participants (P11, P29) mentioned specific brands about scooter and sneaker. Three participants (P8, P23, P28) also related it to raising awareness of health.

  • Emotional Experience and Ambient Media

Most of the participants described the overall experience was positive (interesting, happy, fun, exciting). For example, P1 said: “It was very interesting and funny, and I think it can bring romantic feeling to me.” Three participants felt that the higher pitches made them more excited. In contrast, two participants described their feeling gradually became deep and heavy when walking down. P9 intended to walk down fast to experience the feeling of fast tempo which made her pleased.

The qualitative data also reflected three attributes of ambient media as mentioned in Sect. 2. Spatiality: Some participants had a sense of public space. They recalled environments they had experienced before. For example, P6 mentioned a hotel hall where she heard the music of piano played by someone walking on the stairs in the hotel. In addition, many participants noticed the physical spatial features. Four participants mentioned the spatial relation of stairs. P3 described: “It became a dynamic thing from a flat surface. I noticed the pitch went up with the height of the stairs.” Unexpectedness: Unexpectedness occurred not only in the beginning but also in the whole process of interaction. Many participants felt surprised to hear the musical sounds when first stepping on. For example, P1 said: “After I stepped on it for the first time, I was really surprised by the sound.” Some participants did not feel much unexpectedness in the beginning, but they described various unexpected feedbacks when shifting different movements. Engagement: Many participants mentioned how their perceptions and bodily states affected their future intentions, and their cognitive states also determined further actions. For example, a participant (P19) mentioned he wanted to made the sound last longer and stayed on a stair for a while. P4 mentioned she move her whole body to produce music, and energy was engaged with it.

5 Discussion and the Model of Embodied Meaning Making

The results indicate that bodily movements are tightly correlated to embodied conceptual mappings and cognitive experience. In this section, we focus on further discussing the relationships between bodily experience, embodied conceptual mapping, and imagination, and other related studies are compared. Based on these analyses, we aim to develop a model of embodied meaning making, which provides a useful guidance for characterizing and designing experience with ambient media.

5.1 Embodied Conceptual Mapping

Embodied conceptual mapping can be activated by the audience’s immediate bodily movements. The embodied metaphor projects a concept of source domain (embodied schema-based) onto a concept of target domain (abstract) [2, 17, 21]. A range of conceptual mappings (Table 1) are closely tied to bodily states. The bodily movements (e.g., walking upward or downward) speed up a projection from bodily experience to the abstract and intangible concepts (pitch, tempo, flow, and force). Different from the purely cognitive mapping in linguistics and cognitive science, the conceptual metaphor in the interaction with ambient media maps the features of primary bodily experience (source domain) onto the abstract concept (target domain). The embodied interaction studies [2, 3] asked participants to finish a series of structured tasks, and the metaphorical concepts were prepared in advance. Compared with these procedures, our study does not have a structured task, as the metaphorical concepts were directly generated from lively bodily interactions with PS. We propose embodied interaction with ambient media is a process of meaning-making rather than mere meaning-understanding.

Table 1. Embodied conceptual mappings.
Full size table

5.2 Force-Related Schemas

The empirical findings support the theory of embodied schema-based product expressions [14, 21, 22]. The quantitative study [21] partly proved that the understanding of visual expression of products is rooted in embodied schemas which are based on a series of spatial relations (e.g., In-Out, Front-Back). Johnson [15] also examine those gestalts which embodies more internal structures of forceful interaction. Seven force-related schemas (e.g., Compulsion, Attraction) are demonstrated to play an important role in constructing meaning. The mappings of flow and force intensity are less structured by spatial relations rather closely tied to forceful experience. The first two mappings (Table 1) are based on the same schema SCALE (quantitative or qualitative level), as High-Low and Fast-Slow are more about a cumulative character and dependent on the spatial relations. However, in the other two mappings, many force-related words (heavy, intense, relaxed, light) are tightly related to forceful interaction and kinesthetic experience. The mapping from bodily movements to fluency is extended from the Gravitation which is defined as an experiential structure that the body is physically dragged toward the earth. Especially, one participant described her kinesthetic feeling about fluent moving of a waterfall, which is a typical Gravitation-based mapping. The mapping of bodily movements and force intensity is structured by Removal of restraint that is exerting a force to remove a barrier and the barrier is removed (light or relaxed). These mappings are in line with several qualities of force gestalts: vector, path of motion, degrees of force intensity. Force-related schemas are overlooked by the research on embodied interaction, and further exploration is needed.

5.3 Embodied Schema-Based Imagination

The embodied conceptual mapping not only elicits immediate concept generating and understanding, but also triggers further cognitive processing such as imagination and recall. Participants associated the experience with different objects (e.g., ladder) or environments (e.g., cathedral). These elicited imaginations also can be summarized as two types: orientation-related and force-related. One participants imagined being a cathedral, which maps her feeling of upward movement onto a holy imagery and forms a metaphorical concept (abstract) “Holy is up”. This abstract concept then is mapped onto an imagined cathedral (concrete). This mapping of imagination is typically organized by a orientational or vertical relation. The second type of imaginations is tied to the participant’s forceful bodily experience with PS. The mentioned tangible objects and environments include sneaker, waterfall, house-jumping, and jewelry, these are related to the force-related movements. The imagination of sports products results from the past kinesthetic experience, in which immediate bodily experience is linked to an image structure. The other three mappings also arise from the forceful experience: Blockage: The mapping of stamping on the stairs and a jewel dropping to a metal plate; Gravitation: The mapping of fast walking down and waterfall; Enablement: The mapping of bodily movements and game house-jumping.

Based on the discussion and related studies [3, 21, 22], we develop a model (Fig. 3) to demonstrate the process of bodily movement-based meaning making. Past long term perceptual and motor experience form various mental patterns (spatial and forceful relations) that can be extended to construct different meanings (Line 1). Bodily interactions can activate the conceptual mapping from immediate bodily experience onto an abstract concept (Line 2), which results in an embodied metaphor structured by an embodied schema (Line 3). Embodied metaphors also further imaginations related to imagined environments, brands, and objects (Line 4). Embodied metaphor and embodied imagination are both elicited by bodily movements (Line 5) and structured by embodied schema (Line 6). They share similar mapping process, but normally embodied metaphor is a mapping from a concrete physical experience (action and perception) to an abstract experiential concept, and embodied imagination sometimes maps an abstract concept onto an imagined concrete environment or object (e.g., holy - cathedral, fluent - waterfall). This model provides a useful guidance for understanding and designing meaningful experience with ambient media.

Fig. 3.
figure 3

The model of embodied meaning making.

Full size image

6 Conclusion

We identified the main factors of experience with ambient media through experimental data collection and thematic analysis, and discussed embodied meaning making by analyzing the relationships between bodily experience, conceptual mapping, and imagination. This study provides empirical evidence that bodily action is not only an input behavior for triggering interaction but also a major source of embodied conceptual mappings and facilitating meaningful experience. The findings extend previous studies on embodied interaction: Meaning making and imaginative activities in interaction with ambient media are not only structured by pervasive embodied schemas but also tied to immediate bodily states. The model of embodied meaning making provides a useful guidance for characterizing and designing experience with ambient media. The future work will explore how this model of embodied meaning making can benefit designing ambient media with regard to meaningful engagement. Relevant design principles can be revealed to support design practice through further empirical studies.