1 Introduction

Human–computer symbiosis is a relationship between a human and computer in a cooperative interactive space in which the computer acts as an organism different from the human and is not limited to inflexible dependence on predetermined programs [1]. Based on a recent review by Jacucci et al. [2], for a human–computer paradigm, symbiotic interaction is possible to achieve by using technological resources to understand users and to make those resources understandable to users. Computation, sensing technology, and interaction design are key aspects to monitoring and affecting users through the systems output in a closed-loop paradigm. Following the notion of symbiotic interaction in human–computer paradigms outlined by Jacucci et al. [2], we affixed an affective agent or system to form affective symbiosis.

An affective agent, in one respect, can be expressed as the embodied agent or screen-based anthropomorphic entity that can display emotional expressions to affect users emotionally. A number of studies have demonstrated the potential of using affective agents in interaction [3,4,5,6,7]. For example, the use of an embodied agent in an interface can enhance social interaction [8] and improve life-like interaction with agents [5]. Of course, not every situation requires affective agents: Embodied, affective agents in utilitarian contexts can result in negative user emotions (e.g., the Microsoft search puppy). On the other hand, competitive contexts, such as computer games, might employ these agents more appropriately. In these types of systems, it is also much more likely that a relationship among a user, virtual agents, and the system grows and changes over time. Moreover, earlier studies have shown that the competitiveness of the interaction is positively associated with emotional synchrony between human users [9]. In other words, people pay more attention and are more aware of their adversary’s emotions as the competitiveness of the interaction increases. In human–computer interaction, this could mean that the agent’s affective state becomes more salient for the user when increasing the competitiveness of the game.

However, creating an affective symbiosis between the user and agent requires more than simply integrating emotional expressions into a competitive context. The interaction with the system should be built on a dynamic loop that adapts the system to a user’s body, behavior, and experience [10]. In the present study, we took steps toward creating this type of system in a competitive game scenario, expecting that it will enhance engagement and immersion during interaction. At this point, the system’s adaptiveness is limited to the adversary’s performance, meaning that the performance of the adversary improves directly with the user’s performance. We expect that if a user engages in affective symbiosis, (a) the user’s behavior and physiological states should be affected strongly by system events, (b) the emotional state of the user should be affected by the emotional state of artificial agents in the system, and (c) engagement with the system should increase as a function of the cumulative degree of interaction. In the present study, we aimed to gain early insights on the relationship between these critical aspects of affective symbiosis. The hope is that identified shortcomings will inform us adequately to enable a full-scale affective symbiosis system in the future.

We implemented an interaction scenario similar to the table-top game of air hockey, featuring virtual reality, haptic feedback, psychophysiological measurements, and affective agents. The virtual agent displayed emotional expressions during play. We also built a haptic glove to display the different forms of tactile feedback during the game events. The air hockey game was selected because it represents a fast-changing competitive context in which the user is facing his/her opponent and is allowed to see his/her adversary’s facial expressions without disengaging attention from the game.

In order to investigate the user’s engagement in affective symbiosis, we measured the effect of emotional expressions displayed by affective agents. We expected that interaction with the agent could result in two types of emotional relationship, which one might consider, in the context of symbiotic interaction, as either parasitic or mutualistic. From a primarily utilitarian point of view, competition might entail a parasitic emotional effect: a negative expression of one’s opponent suggesting a gain and therefore resulting in positive affect. However, many studies have suggested that emotional expressions usually result in mirroring behavior, imitating the agent’s expression, which, according to some, might cause associated emotions [11]. In other words, positive emotions expressed by the agents might result in either negative (parasitical affective symbiosis) or positive (mutualistic affective symbiosis) affects in the user.

Because emotions are known to affect users unconsciously on occasion [12], if the system is affectively symbiotic, we expect that the more one interacts with the system, the more one becomes affected by game events. This refers primarily to events that happen to oneself (i.e., one wins or loses). To measure the emotional involvement, we complemented traditional self-report questionnaires with physiological data. To investigate how users implicitly responded to emotional events, we took measures of electro-dermal activity (EDA) and heart rate variability (HRV) during the game.

We think that immersion might improve affective symbiosis, because it might create a stronger impression of the importance of the game and events, thus increasing the salience of the agent’s emotions. For this reason, we used virtual reality [13] and haptic feedback [14]. The findings of the study will contribute to understandings of some crucial points to consider before developing an affective symbiosis adaptation and/or game model and will direct the possibilities.

2 Related Work

Both systems that adapt to user emotions and affective agents that influence user emotions have been popular topics of research in the field of human–computer interaction (e.g., [15,16,17,18,19]). Work regarding emotionally adapted games has demonstrated how adapting game events to a user’s physiological activity can improve the game experience [18]. Kuikkaniemi et al. [19], for instance, used EDA to control the level of difficulty in a first-person shooting game. Results revealed that players who were able to see their biosignal while playing were more immersed and positively affected by the game.

In addition to the adaptive games, affective agents’ emotional expressions also have been shown to influence users’ emotional states and social behavior [5, 7, 20]. De Melo and colleagues investigated social decision-making between a human and affective virtual agent, demonstrating how users were more willing to cooperate or make concessions if agents were displaying certain emotions [21, 22]. Besides making different decisions, users also have shown to mimic the expressions of affective agents automatically and respond to negative expressions with heightened autonomic arousal [23]. In conclusion, affective agents indeed affect users’ emotions and behavior.

Rather than using 2-D screen-based virtual reality, one can use head-mounted displays (HMDs) to achieve better immersion [24] by shutting out the outside world. Recently, advancements in HMD technologies have increased the use of virtual reality and its applications, mostly for gaming purposes [25, 26], to provide the experience of immersion and the enjoyment of full engagement. Though HMDs are used for the visualization of the virtual environment, the realistic feel comes from the feeling of touch and getting a response from the virtual objects [24], which can be achieved through haptic technologies. A haptic feedback technology increases the player’s sense of immersion [24] and provides interesting ways to interact with the game [15] while enhancing the entertainment value [27].

To summarize, systems that adapt to a user’s emotions and systems in which affective agents influence user emotions have been popular topics of research. However, an immersive system that incorporates adaptive gaming performance into affective expressions requires more research. The present work aims to pave the way toward affective symbiosis by investigating the changes in users’ physiology and emotional state in a competitive affective game environment.

3 The Game

An air hockey game (similar to Pong) was designed to investigate the effects of affective agents in an immersive, competitive context. A virtual puck was served from the center left of the air hockey table either toward the agent or the player, alternatively. Both the player and agent blocked the puck using a virtual mallet. There were slots on either end of the air hockey table that served as goal bars. The angle of the puck-serving direction was randomly generated between 15° to 75° angles toward either the agent or the player.

3.1 Player Interaction

We used Leap Motion, a tiny device used to sense a user’s hand and finger motions using infrared cameras in an interactive 3-D space, to track the user’s right-hand movements. This device was positioned underneath a glass table and enabled sight of the player’s hand in virtual reality. A transparent plastic sheet was used to help move the user’s right hand freely on the glass table. The default hand model, provided as part of Leap Motion’s Unity3D package (www.leapmotion.com), was used to enable interaction in virtual reality. In virtual reality, a user could grip the virtual mallet and move it to face the moving puck. A player’s left-hand index, middle, and ring fingers rested on the arrow keys of a PC keyboard and were used to provide answers to the self-reported items shown during gameplay (Fig. 3).

3.2 Agent Interaction

The agent faced the puck using a set of hand-movement animations. Twelve agents’ hand-movement animations were designed to reach 12 predefined target points on the game table (see Fig. 1). While the puck moved toward the agent, the agent tried to find the closest target point according to the puck’s movement direction, and the agent played the corresponding hand movement animation to bounce the puck. To make the game competitive based on player performance, we designed three levels of difficulty: easy, medium, and hard (Fig. 1). The adaptation of the competing model depended on the player’s performance so that players could gradually improve their skill along with the agent and continually enjoy the competition.

Fig. 1.
figure 1

Competing models. Yellow lines represent the medium model and its selected target points. Red lines represent the easy model. All target points were selected for the hard model. (Color figure online)

Full size image

3.3 Tactile Feedback

A recent work by Ahmed et al. [28] showed that differences in haptic technologies affect how simulated touch in virtual reality is perceived. They found that haptic exciters provided a more intense signal than the commonly used C2 vibrotactile actuators. However, mechanical actuators were found to make the touch feel more natural. This motivated us to use both a haptic exciter and a mechanical device to provide tactile feedback for different game events. We designed a glove (Fig. 2) using a piece of cotton fabric. We attached a micro servo motor (9 g, speed 0, 10 s/60, torque 1.3 kg/cm at 4.8 V) on top of the glove. Three elastic strips of tape were attached to the glove and connected by a thread to the horn of the servo motor. The servo motor rotated up to 180° (c), thereby pulling the thread, creating tension on the elastic tape (b), and applying pressure to the user’s hand. This mimicked the sensation of a player gripping the mallet. Two audio haptic exciters (Tectonic’s 14 mm, 8 O TEAX14 C02-8, www.tectonicelements.com/audio-exciters) were placed on top of the glove (a) to produce vibration on the user’s hand while the user bounces the puck. Because the skin on a human hand is sensitive to vibrations over 20 Hz and a square wave is perceived to be more intense than the commonly used sine wave [29], we selected a 35 Hz square wave of 50 ms vibration as the puck-bouncing feedback.

Fig. 2.
figure 2

Exciter-servo motor tactile glove.

Full size image
Fig. 3.
figure 3

Test setup and player interaction.

Full size image

4 Method/Evaluation

4.1 Participants

Seventeen university students and researchers (12 male, five female, 25.47 ± 4.4 years old) volunteered to take part in the study. They signed informed consent agreements prior to the start of the experiment and, afterward, received a movie ticket for their time.

4.2 Test Setup

The Unity 3-D game engine (version 4.5.4, www.unity3d.com) was used to implement the experimental system, providing the game scenario and questionnaires visually in virtual reality via the HMD, recording user responses, and communicating with the physiological apparatus. Virtual reality was enabled using the Oculus Rift VR HMD (Oculus Rift Developer Kit 2, Resolution 960 × 1080 per eye, Refresh Rate 75 Hz, 100° nominal field of view). The player’s head movements were tracked at 1000 Hz using a three-axis accelerometer, gyroscope, magnetometer, and external positional tracker (www.oculusvr.com).

Affective agents in virtual reality were designed by combining a face model and a body model. The face model used was the original one provided by Faceshift (www.faceshift.com), and the body was designed using the FUSE design tool provided by Mixamo (www.mixamo.com). The texture of the face and body were edited in order to build three different-looking agents. The agent displayed three emotional expressions (happiness, anger, neutral control condition). These were recorded prior to the study by capturing a live presentation by a professional actress using Faceshift algorithms. Each expression animation lasted 4 s, ending with neutral expression. Prior to the experiment, a pilot study was conducted in which a sample of human participants was used to classify the expressions based on categories of six basic emotions (disgust, fear, sadness, anger, happiness, and neutral). Expressions of anger, happiness, and neutrality were then selected for the final paradigm based on their relatively higher recognition accuracy (happy: 95 ± 14%, neutral: 90 ± 14%, angry: 90 ± 23%).

4.3 Procedure and Design

As illustrated in Fig. 4, at the beginning of the game, the player grabbed the mallet. The haptic glove applied tension to the user’s hand to mimic gripping the mallet physically. The agent’s facial expression was presented as a common starting point for every trial. The same facial animation was played throughout the entire trial. After 3 s, the puck was served toward either the agent or player in alternating order. A short vibration through the glove was displayed on the dorsal of the player’s hand while the player bounced the puck. The trial lasted until a goal was scored, after which the scores were updated. After 1 s, a short questionnaire (see Sect. 4.4) was presented.

Fig. 4.
figure 4

Trial sequences. Users grabbed a virtual mallet (A). At the beginning of the trial, the agent was displayed and started to show emotional expression (B). After 3 s, the puck was served (C) and the vibrotactile stimulus was presented as the user bounced the puck (D). After each goal, scores were displayed (E). After each block, a self-reported questionnaire was presented (F).

Full size image

Participants undertook nine blocks of seven trials. The virtual agent was changed between blocks whereas the order of agents was counterbalanced between participants. Trials were randomized fully across the 63 possibilities obtained by orthogonally crossing the 3 repetitions × 3 agents (angry, happy, neutral) × 7 serves.

4.4 Measurements

User experience was investigated using a quantitative analysis of both explicit self-reported Likert scales and implicit behavioral and physiological measures.

Self-reported Data.

Self-reported data on emotional interdependence, co-presence, game experience, and emotion-classification items were collected. Scales of emotional interdependence, co-presence, and game experience were single-item measures obtained from the social presence module of the game experience questionnaire (FUGA, [30]). Items are shown in Table 1. Participants filled out the questionnaires using their other (i.e., non-virtual) hand, resting on the arrow keys of a PC keyboard. The questionnaires used 5-point Likert-style scales, while the emotion classification used a three-alternative forced choice.

Table 1. Descriptive statistics of self-reported questionnaire (N = 17). A 5-point Likert scale (1 = not at all, 5 = very much) was used for item scoring. Note: values are means ± SEs
Full size table

Implicit Measures.

The users’ electrocardiography (ECG), EDA and right-handed biaxial accelerometer (ACC) were recorded using a Brain Products QuickAmp biosignal amplifier at a sample rate of 1,000 Hz (Fig. 5). ACC data were transformed to movement speed by taking the root mean square of the first derivative over both axes (i.e., the Euclidean distance traveled per ms). Physiological sensors to measure EDA were placed on the index and middle fingers of the left hand, and to measure ECG were placed on the chest. EDA commonly is used as an emotional arousal indicator [31] or to describe the intensity of an experience. Electronically, it represents the potential difference between two areas of the skin. Two key measurements of the EDA are tonic (low frequency baseline conductivity level indicates slow changes in arousal) and phasic (higher frequency indicates rapid changes in arousal). The average, raw EDA was taken as a tonic measure of autonomic activity. Furthermore, the measure was transformed to reflect phasic arousal related to non-specific events [31] by applying a continuous, local, positive peak (of at least 0.5 μS) detection algorithm and calculating the rate of skin-conductivity responses per second. Finally, the latency (in ms) between heartbeats, or inter-beat interval (IBI), was used to calculate the average IBI (inversely related to heart rate).

Fig. 5.
figure 5

Ten seconds of raw heart rate (ECG), EDA, and 2-D accelerator (ACCX and ACCY) data during gameplay. Red boxes show events of interest: (A) single heart-beat used for calculating heart rate variability, (B) phasic EDA response used for calculating skin conductivity response rate, and (C) movement across X and Y used for calculating movement speed. (Color figure online)

Full size image
Fig. 6.
figure 6

Effect of losing or winning on the first 3 s of movement speed, EDA/10, and skin conductivity response rate (SCR*10).

Full size image

5 Results

5.1 Emotion Recognition, Game Experience, and Presence Measures

Participants’ judgments concerning emotions, game experience, emotional interdependence, and co-presence were investigated using one-way repeated measure ANOVAs (RMANOVAs) with the agent’s emotional expression (neutral vs. angry vs. happy) as a factor. The agent’s emotional expressions were judged correctly in 91% of cases. Furthermore, some of the expressions were recognized significantly better than others, F(1.36, 21.82) = 4.34, p = .04. A post hoc LSD (Least Significant Difference) analysis revealed significantly higher recognition rates (p < .05) for happy (100%) compared to neutral (88 ± 14%) and angry (85 ± 23%) expressions. Moreover, the agent’s emotional expression had a significant effect on experienced emotional interdependence, F(2, 32) = 9.01, p = .001, η2 = .36, and co-presence, F(1.43, 22.84) = 15.31, p < .001, η2 = .49. A post hoc LSD analysis revealed that participants felt more influenced by the agent’s mood in happy (2.81 ± 0.78) and angry (2.54 ± 0.77) conditions than in the neutral (2.04 ± 0.59) condition (p < .05). Along similar lines, participants reported higher experiences of co-presence while playing against happy (3.45 ± 0.66) vs. angry (3.05 ± 0.59) agents, rating neutral (2.48 ± 0.70) agents as the least attention-capturing (both emotions > neutral, p < .05). Finally, the agent’s emotions affected the participant’s ratings on how deeply they concentrated on the game, F(2, 32) = 7.17, p = .003, η2 = .31. A post hoc LSD analysis revealed that participants were more concentrated during the happy (3.45 ± 0.66) than the angry condition (3.05 ± 0.59) and that the neutral expression (2.48 ± 0.70) was associated with poorest concentration (p < .01).

5.2 Experience of Game Events: Behavior and Physiology

Trials were included in the analysis only if both the agent and user successfully blocked or bounced the puck at least once. Seventeen users were tested, although the accelerator recordings were missing for one. Data for this user were omitted from movement-related analyses.

In order to find out whether the agent’s emotion had a general effect on behavioral and physiological responses, RMANOVAs were conducted with the agent’s emotion as factor and performance, movement, tonic EDA, average heart rate, HRV (standard deviation of the IBI), and skin conductance response rate as measures. Tonic EDA was found to be affected by the agent’s emotional expression significantly, F(2, 32) = 4.59, p = .03, but no effect was observed for any other measure, p > 0.1. Angry agents increased tonic EDA (883 µS) relative to neutral (829 µS) and happy (802 µS) conditions.

Next, we investigated how winning or losing affected the behavioral (movement data) and physiological (EDA, SCR, IBI) response. RMANOVAs were conducted on measures taken in the first 3 s after the game event of winning or losing with performance (win vs. loss) as factor. Performance significantly affected EDA, F(1, 16) = 15.20, p = .001, SCR(1, 16) = 17.92, p = .001, and movement, F(1, 15) = 20.15, p < .001. Losing increased EDA, SCR, and the amount of movement.

Finally, we explored emotional investment as the game progressed. We used the average SCR measure evoked by successfully blocking (bouncing) the puck across time. As shown in Fig. 7, the more often the puck was blocked (N bounced), that is, the further the game progressed, the higher the SCR related to the event itself. In other words, the more the user became excited by the event of blocking, the more the game progressed. Interestingly, this upward trend also was visible when the SCR was measured in response to the agent’s, rather than the user’s, blocking. Thus, this shows that the user is emotionally involved with both the game and the agent and that this emotional engagement evolves over time, suggesting affecting symbiosis (Fig. 7).

Fig. 7.
figure 7

Skin conductance response rate (SCR/s) related to blocking the puck as the game progresses.

Full size image

6 Discussion and Conclusion

The emergence of interest in human–computer symbiosis and affective communication calls for examinations into the possibilities of affective symbiosis in the human–computer paradigm. Here, we developed a competitive VR game environment working as a stepping stone toward a system that allows affective symbiosis. Previously, symbiotic interaction has been considered a beneficial aspect of collaborative interaction. However, recent findings from human–human gaming have demonstrated that affective linkage between users is crucial in competitive contexts [9]. Developing a similar affective linkage in a human–computer paradigm would require new sources of information, such as biosignals, and a system that is able to adapt the game events to such information. In the present study, we built a simple, behaviorally adaptive, competitive environment in order to investigate how the affective agent’s emotional expressions affected the user’s emotions and game performance.

The results of the present study showed that the agent’s expressions affected emotional interdependence and co-presence, indicating that the adversary’s happy and angry rather than neutral expressions caught users’ attention and affected their conscious emotional state. Moreover, users reported concentrating more on the game when the agent expressed happiness. However, neither the difficulty of the game nor other game-related experiences were influenced by the agent’s emotion. Thus, the results suggest that the computer agent’s affective expressions modified the user’s attention and emotional experience but did not change the degree to which the user found the game difficult or entertaining.

Additionally, the implicit measures supported this conclusion. Users’ tonic EDA level was higher when their adversaries were angry, whereas users’ game performance was not influenced by the agent’s emotion. EDA and other implicit measures also revealed that users were likewise affected by the game events. Losing was related to more amplified tonic and phasic skin conductance responses than winning. Finally, users’ EDA responses got stronger as the game progressed, indicating greater personal investment or engagement over time. The findings suggest that affective agents and game events influence the user’s emotional state on both the conscious and implicit levels. These effects do not, however, modify the user’s performance in the game or its perceived difficulty.

The limited effect of agents’ affect could be due to the competitive nature of the task. In a negotiation context, another person’s nonverbal cues might be more salient and thus influence the user’s behavior more [6]. In turn, in more competitive contexts, the influence of social cues is limited because the game events change more quickly and require continuous attention from the user. Nonetheless, the affectivity of the agent influenced users’ emotional state regardless of the competitive context. This effect was consistent with earlier findings [20]. The results thus suggest that affective expressions presented in a competitive context do not necessarily make humans perform better in the game but improves the overall social experience.

The findings of this study will help us understand users’ experiences while interacting with affective adversaries in an immersive competitive environment. The results also can be used to build an affectively adaptive system that integrates users’ behaviors to their emotional responses when calculating further game events. More knowledge of agents’ adaptive emotional responses and their effect on the user is required. In the future, one could, for instance, investigate whether varying the agents’ affective state according to game events would intensify the effect of the agent’s emotions on the user’s emotions and behavior.