Keywords

1 Introduction

Smart things, in the context of the digital games, offer opportunities for engaging in physical play outside the virtual world. Since the things have playful aspects, such as toys, they can promote interaction independently of being linked to a digital platform. This happens because of the artifact’s aesthetics, or due to the thing’s own technological components, e.g., sensors and actuators. Thus, these interactions promote a connection between the physical and the digital worlds since the scenario places playful meaning on the game interface. However, to design such experience, one must first investigate how to integrate these things into the gameplay.

Although there are works focusing on these physical interactions, especially in the context of serious games [1, 2], little has been achieved concerning individual aspects related to prototyping [3] and integration of artifacts with the game design [4, 5]. For that reason, this paper aims to bridge this gap by proposing a theoretical model for game development and prototyping based on case studies. Therefore, we conducted a systematic literature review, covering related publications of the last five years resulting in 35 selected papers, and in addition, we included 30 cases from the market. During data extraction, we analyzed the systems in order to identify the agents involved in the interaction, and how these integrate the real and virtual worlds.

2 Related Work

Coulton [5] uses the game spaces [6] definition to classify IoT systems for gaming, aiming to consider how and where the interaction takes place. Moreover, the author seeks to recognize when and how the interaction feedback must be presented. The author divides the scenarios into four categories: IoT object, IoT object in the player space, IoT object with tablet and IoT object with the screen. Despite that the author’s model covers different scenarios of smart play taking into account the relationship between things and spaces and the types of technology that promote interactions, little is explored regarding the impact of these things on the decisions of game design, introducing only a few considerations for the last two scenarios.

Moreover in the context of smart tangible things, van de Garde-Perik et al. [7] analyzed the relationships between input and output (I/O) seeking to understand the relations involving the user information perception to create meaningful I/O experiences. The model combines two parts: the State Change, referring to I/O from the user perception of the physical world, and the Information Access, related to the smart devices, in the sense of what information the devices collect (input) and how the system translate the data (output). Connecting the two I/O models there is the cognitive dimension regarding the user’s perspective, followed by the shared physical space, and the digital space. The framework, however, does not focus on gaming scenarios, but the study includes an educational game in one of the case studies.

Magerkurth [4] present a theoretical framework to describe the domains involved in hybrid gaming systems, making it based on the literature and observing aspects of board games and digital games. The author focuses on multiplayer gaming scenarios involving smart tangible things enabling to insert real-world information, with output provided by a screen and graphic-user-interface (GUI). They define that the goal of these games is to capture the traditional and co-located game experience of the physical spaces, enhanced non-intrusively using pervasive technology. The framework comprises three domains, starting with the social domain that is public, private or shared. Followed by the virtual domain associated with the social domain through the game state regarding the sharing of information. Finally, there is the physical domain accessed through the properties of the real world and tangibility.

In turn, Guo et al. [8] introduce a framework on the aspects that make a pervasive game based on literature review, and this consists of four perspectives, comprising systems such as smart-toys, affective gaming, augmented tabletop games, location-based games, and augmented reality games. The first is the temporality that classifies the game turns in closed, or open-ended where the player can join in the interaction at any time. The mobility perspective concerns games played at a fixed location, games with a large interactive environment and games with enough space to move the body. In the perception the study distinguishes how the system extracts the real-world information, and how the user and the system translate this information, also bringing an overview of the technologies. The social perspective reflects the relationship between the players, and in the social influence on the game purpose and thematic of the system. Finally, the authors use the parameters to set a score for the games, aiming to identify the pervasiveness of the each system.

3 Method

3.1 Systematic Literature Review

We conducted the systematic literature review following Kitchenham and Charters [9] guidelines, and we carried out the procedure between July and November of 2015 covering related publications between 2010 and 2015. Two researchers conducted the method, one primary researcher for select the papers, and other for validating the findings. We started with the automatic search using a search string including terms such as “tangible interface”, that on account of things integrate physical form to digital information [10], “playful interfaces” that are interfaces that invite the user to engage in playful experiences [11], and sure, “game-user-interface”. We applied the string in 5 search engines (ACM, Springer, IEEE, ScienceDirect, and Scopus) on 23rd July of 2015, returning a total of 483 papers, of which 62 were replicas, leaving 421 articles for the first phase of analysis. During the first stage, we applied the inclusion and exclusion criteria on the title and abstract only. We considered two criteria for inclusion, these are systems where things have playful aspects, such as toys, and gameful systems, as digital games, augmented board games, gamified multimedia applications, interactive storytelling, structured play activities, and open-ended play scenarios. So during the first phase, we classified 90 articles as “potentially accepted” and we set 68 “in doubt”. Moreover, we excluded 263 articles of which 2 were non-English written, plus we regarded 33 as gray literature, 15 restricted to download, 3 were work-in-progress papers, and finally, we identified 213 out of the scope of this research, e.g., systems without playful intervention, such as wands, devices for gesture recognition, mobile phones or smartphones, and haptics devices. In the second phase, after reading both the introduction section and conclusion of the papers, only 83 articles met the inclusion or exclusion criteria.

After this stage, we started the manual search conducting the snowball of references, adding 60 articles for further quality evaluation. Also, we added another 21 full papers after manually searching the proceedings of 3 related conferences (CHIPlay, ACE, and DiGRA). In summary, 165 articles were fully read out and evaluated by 14 criteria, which 10 we extracted from the guidelines provided by [9], and we assign them weight (N = 1), and the other 4 criteria were regarding details of systems such as kinds of artifacts, the technologies involved and the gameplay. So, for these we designated twice the weight (N = 2), looking for select studies that had enough information to answer the research questions. All criteria were then scored with “0” to “no”, “0.5” to “partially” or “1” to “yes”, and we considered the cutoff point of 75 %. During the quality evaluation we excluded 125 articles considering the scores, and in the final selection we identified 5 papers as redundant, e.g., articles of the same authoring, and that of equal systems, resulting in 35 papers for further data extraction. Overall, the final selection of articles covers topics such as games for learning [12], games for children with special needs [2], games for health and sports [1], playful augmented training [13], interactive storytelling [14], smart and robotic toys [15, 16], tangible-augmented reality games (AR games) [17], tabletop games [3, 18], remote playful experiences [19], outdoor games [20], virtual reality games (VR games) or immersive scenarios [21], and smart devices for open-end play [22].

3.2 Market Scenarios

The study aims to extend the model’s contribution to developing hybrid systems beyond the academic sphere, also including market cases. Therefore, we selected 30 cases of systems involving smart toys and digital games, doing this by analyzing the catalog of toy stores websites, and successful projects of crowdfunding. In summary, we found mostly smart toys for mobile gaming destined for babies and young children. However, we also found more complex games, e.g., Golem Arcana [23], an augmented board game with token pieces, and the tower game Fabulous Beats [24], including interactive toys for TV. Moreover, we name multiple similar cases, such as NFC collectible toys embedded with content for video games, mixed reality coloring applications as Disney Color and Play [25], touchpoints toys and stampers, and building blocks. Other common are the robot plush toys for pet management applications, e.g., the Furby Boom from Hasbro [26], and plush toy cases for smartphone and tablet, as the Didi Teddy Bear [27], including plastic toy-cases aiming for protecting the device and promoting playful tangible interaction with mini-games.

3.3 Data Extraction

Joining the 35 papers from the review and the 30 market cases, resulted in a total of 88 systems for analysis, since some papers presented more than one application or more of an artifact. Regarding the results of data extraction, we name the components that define the setup of the interaction, we started with the things, these are two kinds, the playful things, such as toys, and non-playful things, like things that we project playful appearance, or smart things able to capture information from the environment. Thus, we have the technologies associated with the things, and these are passive or active. Passive technologies include three scenarios, starting with things with markers, these are QR codes and fiducial, followed by technologies that detect things markerless, configuring a non-intrusive way, these scenarios use invisible ink [28] or detect properties of the things, e.g., shape, color, and texture. The third scenario comprises things with touchpoints, consisting of conductive points positioned on the base of things that provide unique patterns for multi-touch screen, making it by the distance between points. Active technologies differ in two groups, external technology as smart tags attached to things (RFID/NFC tags) [20], and embedded technology, where the components are inside the things. These are components such as sensors used to detect an external stimulus, e.g., gyroscopes, accelerometers, color (RGB) sensors, and these vary according to the requirements of the application. In addition, there are actuators, which are responsible for the feedback of things, e.g., screen displays or LEDs, vibrating motors, or servomotors that move parts of things, and speakers for auditory feedback. As for promoting the transfer of data between things, there are the connection modules, the most common are Bluetooth-low-energy (BLE), wi-fi module, and RFID/NFC readers. In that sense, the sensors collect information and process it into things, otherwise, the thing transfers this data to an external device to occur processing, and the feedback occurs. The feedback is visual, tactile or auditory, and can happen on things by it actuators, or in an external device, thereby promoting interaction.

Such technologies are responsible for collecting environment information to the system, and depending on the nature of the technology, these can collect a different kind. Regarding all the selected systems, we name the nature of the information collected, starting with the identification (ID), that is useful for things that have unique information and its support both active or passive technologies. The ID often is a single path, but active technologies allow updating the ID status by means of interaction. Besides, in scenarios with multiple similar objects, such as board games, individual ID is not that relevant, and then captures information such as color and shape of things is enough information to promote identification. Another recurring type of gathered information is the thing’s position, there is the 2D position to capture moving up to two axes: x, y. This information is often used to recognize thing’s position on the top of a display or a table, such as in tabletops, however, it also occurs off-screen. Yet another information is the 3D position, comprising moves in three axes: x, y, z, usually, the system detects such positioning outside the screen, but, researchers took efforts to recognize objects in 3D position on tabletops [18]. In scenarios that enable more free movement, usually without displaying on-screen, capturing additional information, e.g., rotation and inclination can help distinguish the 6D position of things, and of people. Although, it is noteworthy that there are no fixed rules to use the different position information, such as regard to the environment has a screen or not. In addition to this more recurring information, more specific sensors can gather a sort of data, e.g., ambient temperature, lighting, and proximity to other things. Besides, things can extract data from people, such as games that use heart rate as input.

4 A Relational Model for Hybrid Gameplay

Observing the universe of things and technologies, we realize that there are two setups of play promoted by things. There is the open-ended play, characterized by scenarios where the game rules are not predefined and the play occurs based on the interactions promoted by things and it sensors and actuators. Yet, there is the hybrid gameplay referring to scenarios where the game rules are previously established to promote interaction. Traditional toys and smart toys, when off-line, supports free-play, and the latter, when actuated, promote open-ended play, and in both cases when integrated into the game world, promote hybrid-gameplay experiences. For that reason, the study decided to focus only on scenarios with toys in the context of hybrid gameplay looking to discuss the gap of integrating things with the game design. Therefore, we propose the model presented in Fig. 1 and we done this considering the relationship between three entities: the things, people, and the environment. The model configuration enabled to fit all scenarios identified by the review. Then we explain each aspect of the model and an example is given using the model to recognize aspects of the playful system in the context of hybrid gameplay.

Fig. 1.
figure 1

A relational model for hybrid gameplay

Full size image

Things are toys, and when applicable, things include the external devices connected to toys that promote the interaction. There are three types of toys and all of them can communicate with each other actively or passively. First, traditional toys can connect to passive or active technologies, enabling them to interact with the game environment [12]. Secondly, smart toys promote interaction regardless connection with an external device, and it can provide feedback concerning the game action on both toy and in the associated device [26]. Finally, smart playground consists of a large toy, e.g., a tower [1] or an interactive floor [29], or it consists of multiple interconnected smart toys in the same environment, featuring the Internet of toys.

Now, regarding the things and the game environment, the system review allowed classifies interactive patterns involving these connections whether this is physical or virtual. Yet, the interactions rarely occur in isolation, allowing the same toy, game, or action to associate with different interactive patterns simultaneously.

Replicate.

The physical game object replicates in the game environment, doing it replicating information, e.g., positioning and orientation, as well as its physical features. For example, in the space-themed game for Portico platform [30] the ship toy has its appearance, with regard to color, shape and texture replicated in a virtual object, and the virtual ship replicates the toy’s orientation in real-time.

Replace.

The physical game object replaces another thing assuming its identity in the game world. For example, in games with touchpoints toys like Batman Apptivity Mattel [31], a physical toy replaces the virtual playable character in the game that interacts with other virtual objects on the screen.

Create.

The physical game object creates new objects in the game world, by instantiating things or creating individual objects. For example, in Camelot [20] the players use physical toys to create a castle, but first they must collect virtual resources engaging pieces on smart toys with RFID readers, after acquiring the resources, the user is able to attach the pieces to another to build the castle before the other team finishes it.

Destroy.

The physical game object destroys things in the game world. In the Ipad game Runaway using wooden blocks from Magikbee [32], the blocks can build bridges to the virtual character get from one side to the other in the game scenario, however the enemy that follows the protagonist can also use the toy bridge to trespass, then the player needs to overturn this in time, by destroying the passage.

Extend.

The physical game object extends itself in the game world, by extending information, e.g., trajectory, or connecting to a virtual game object extending its real appearance, other than replicate, extended objects composes the same object. In Rope Revolution [33] a physical rope extends its trajectory and its appearance in the game to fly a kite, so the base of the rope is physical, and the end of the line where is the kite is virtual. Besides, inside the virtual gaming environment, there are wind interactions, and the microcontrollers coupled to the rope simulates the pressure and force exerted on the rope, together with fans that synchronize winds in a real environment.

Update.

The physical game object sends its status or identity, to the virtual game world and receives feedback, updating its initial state. In that case, the user can update the status intentionally, or independently by interacting with other virtual or physical game objects. In the application for coloring Disney Color and Play [25], the child paints the characters in the physical coloring book, and using a tablet, the user sees the corresponding 3D virtual character and its colors updated in real-time, according to painting held by the user.

Augment.

The physical game object augments information non-intrusively, referring to the rules of the game, interfering in the game challenge, and promoting social interaction between players. These include systems to play over a distance that the technology supports social interaction among participants [19], applications for playful training [13], in which uses technology to augment the rules of a traditional game scenario as the Golem Arcana [23] board game, the player uses a pen to augment character tokens by displaying information rules that guide the game through a tablet.

Moreover, the environment is where the interaction occurs, which means the place where are people and things. The environment is private when each player has its own information and access to the toys or shared when provides to all players the same information and access to things [24]. People are in two ways in the environment, co-located, with the players present in the same physical environment, such as in tabletop games [3]. In contrast, when players are in different physical environments they are remote located. In the latter, the real space of the players is private, but they share the virtual space. So, people relate to each other in three forms, first by competition where players compete for the same purpose in the game. Followed by collaboration, when players cooperate to achieve the same game goal, and may perform the same or complementary actions [33]. The third is the parallel play, where people interact in the same environment, but performing actions that are independent [18].

People relate to things through physical interaction mediated by technology. Observing the scenarios we recognized four main interactions, these are the vision, since people visually access the environment and things, followed by the manipulation when people use their hands to move toys or parts of things. The third is the embodiment where people access the environment through the body, may providing the system with information inherent to the body [20], or interacting with things that require body movements [1]. Finally, there is immersion, where people are inside the things, like environments with multiple interconnected objects, or large interactive installations as smart playgrounds. Such interactions increase as a scale, however, these are not mutually exclusive, but complementary.

Once, the four main interactions relate to actions that the user takes in the context of reality, then, there are corresponding technologies connecting such activities with the virtuality. In the case of vision, usually some sort of display present visual feedback using a screen or multiple screens, or through projection or LEDs illumination. In that sense, in manipulation the things that support action are toys or hand-held devices, while in the embodiment when the player uses the body as input, the wearable technologies are prevalent. Finally, regarding the immersive interaction considering an environment composed of multiple interconnected smart things, there are pervasive technologies related to the Internet of Things (IoT), in our case, the Internet of toys. Finally, perceiving the relationships between people, things, and the environment, we recognize two domains that rule such interactions. The physical domain comprises the interaction between things and people, and between people and the environment and the social domain covers the interaction between people and people, once they promote social interaction.

To guarantee applicability to the extracted model we analyze all selected systems using the proposed configuration. Here, we describe an illustrative scenario in Fig. 2. The chosen was the Age Invaders [29] consisting of a smart playground for intergenerational play promoting physical and social interaction between grandparents (GP), parents (P), and children (C). Now analyzing the game by the proposed model, we start naming things that are part of the system, these are the 45 LED tiles configuring the smart playground, the smart toys which are BLE and RFID empowered teddy bear slippers, the personal LED displays, and the BLE handheld devices, therefore things communicate with each other in an active way. On integrating things with the environment, the system replicates the virtual tiles controlled by P to the tiles on the smart playground, also occurs replacement of P for virtual avatars on-screen. The same happens in the smart playground where GP and C wearing slippers replace the virtual playable character by interacting with virtual objects on display.

Fig. 2.
figure 2

The hybrid gameplay model for Age Invader [29]

Full size image

The players (GP and C) also create instantiated laser shots in the game environment, by clicking on their handheld devices, and when the laser hits a player, it destroys energy points of the participants. So, the player’s health status updates through collecting virtual resources or receiving hits, and the power state updates by collecting power-ups. The update also occurs in the location of players on the map by reading the RFID tags on the floor and sending this information back to the system. Finally, managing the rules of the game level the system augment information indicating possible movements for C interfering with the challenge of the game.

Thus, the environment is private for P to the other participants, and shared to the players in the smart playground. Concerning the people and the environment, people are co-located in the case of GP and C, and P is remote. Now, on the people, there is social interaction promoted by competition between the teams of GP and C, collaboration among team members and between P and the whole group, also, there is parallelism on each player’s actions in the game. Yet, on the axis of things with people, we recognize the four main interactions, starting with the vision, and manipulation for all participants. Followed by the embodiment for GP and C, as well as immersion since people are inside the things (smart playground and the smart slippers). In the technology perspective, there is display on-screen for all players and display by LEDs for GP and C both in the smart playground as on the personal displays. Also, the manipulation happens through hand-held devices carried by GP and C, and by the computer interface for P. Finally, there are wearable technologies to GP and C that wear the slippers and the individual displays attached to their clothing, and IoT technology for all, since many things interconnects via BLE and wi-fi in the environment.

5 Conclusion and Future Work

In short, the study structured a relational model based on case studies from the academy and the market, able to comprise different scenarios of hybrid gameplay involving interaction with toys in the context of digital games. The paper has succeeded in classifying the scenarios in interactive patterns regarding the interaction between three entities: things, people, and the environment. Things are toys interconnected by active and passive technologies, integrated or not with external devices. Such things can interact with the environment by replicating, replacing, extending, creating, destroying, updating and augmenting real and virtual information in the gaming environment, making it in a private or shared form to people. People interact socially with each other by collaboration, competition, and parallelism, and physically with things and the environment. Doing this through visualization of things and the environment, manipulation of things, body movement, and by movement inside outdoor or indoor environment. Finally, people act on things and things access the environment through technologies responsible for promoting interaction and providing feedback of information via displays, handheld devices, wearable, and connection through the Internet of things.

As future work, we will prototype scenarios that include 7 patterns of interaction identified by the model. The goal is to understand through a practical perspective, more details on integrating things with the gameplay and how toys influence in the game design decisions. Therefore, all prototypes must involve toys both with active and passive technologies, and so far we prototyped scenarios focused on replication, replacement, and creation, and all the games are competitive and of the same thematic. In addition, we will apply the relational model in the creative process of hybrid games, intervening in the phases of ideation, prototyping, and evaluation, over a course of four months with students of engineering, computer science, and design.