An exploratory study of the relationship between meaning-making and quality in games

Abstract

This study aimed at identifying and interpreting relevant relationships between players’ perceptions of meaning-making affordances and: 1) players’ reported quality of the gameplay experience; 2) general player game ratings; 3) expert game ratings. The theoretical framework underpinning this study conceptualized gameplay as an iterative and contextualized activity driven by meaning-making processes that integrate rational interpretation and affective valorization of key game aspects. This was used to evaluate 14 games using an ad-hoc questionnaire that was completed by experienced players and compared to scores of game quality provided by the players and by external sources. The results of this exploration suggest an association between meaning-making affordances and critics’ evaluations, but not with players’ game experience and player ratings. The analysis revealed that key methodological issues should be accounted for when investigating game features and their affective meaning-making implications on the perceived quality of a game and the gameplay experience. Insights on important theoretical and methodological issues that may orient and support future research are discussed.

1 Introduction

Different perspectives have been used to understand the player experience and the factors that affect it. This has led to diverse conceptualizations of game systems, the constituents of the player experience, relationships between game features and the player experience, and methodologies to assess the player experience [7, 8]. Over time, player-centrism and game-centrism have emerged as core trends, presenting clear benefits but also associated shortfalls [15]. On the one hand, game-centric approaches focused on the structural analysis of games and the formulation of mechanistic models of game systems [18, 26, 38]. Albeit useful to conceptualize what features define a game and how these features interact, game centrism may be insufficient to explain why players decide to engage in gameplay dynamics and enjoy the gameplay experience. Player-centrism is instead focused on the psychological factors that trigger and sustain players’ engagement in the gameplay activity, and their relationship with generic game features such as the intuitiveness of controls [9, 34]. Player-centric approaches have led to the identification of key motivational affordances that may explain engagement in the player experience, and the development of instruments to assess the player experience accordingly [37]. However, emphasis on isolated motivational affordances and generic game features may be insufficient to explain the systemic interplay of psychological aspects of the gameplay activity, and how this ultimately defines the players’ experience. The game-centrism vs. player-centrism dichotomy has by extension influenced the development of formal instruments to investigate games and the player experience. Game-centric instruments have been developed to investigate players’ perceptions on game features (e.g. narrative, competition, collaboration), overlooking the psychological processes that these features might trigger [2, 5, 24, 40, 42]. Player-centric instruments have been developed to investigate perceived aspects of psychological processes involved in the gameplay experience (e.g. motivation, engagement, learning, etc.), without emphasizing the relationship between these processes and concrete features of game systems [1, 7, 8, 10, 13, 29, 32, 33, 35, 41]. Thus, as the study of games and the player experience evolved, the need emerged for integrative perspectives and instruments, suitable to account for the complex interplay of game features and the diverse psychological processes that underpin the gameplay activity [8, 15, 39].

This situation echoes to some extent the evolution in the field of human-computer interaction (HCI) of the conceptualization of user experience and the approaches to investigate it. Until the 1970s the focus of HCI was predominantly (if not exclusively) set on utilitarian, non-discretional uses of technology, and the optimization of the performance of computer-mediated work tasks [16]. Back then, people used computers as work instruments because they had to, in order to achieve externally-defined goals. HCI was consequently focused on improving the design of computer systems and applications to facilitate the physical interaction with technology, and the cognitive processes underpinning the decision-making involved in its use. This situation changed dramatically in the 1980s. The advent of personal computing set the spotlight on discretional, non-utilitarian uses of technology [16]. People begun owning computers and using applications because they “wanted” to, rather than because the “had” to. Thus, the frequency and purposes of use of computer applications broadened, as computer uses were increasingly defined by users who were able to decide what to use, when, how and why, based on their needs, inclinations and circumstances. The field of HCI consequently evolved, acknowledging that users are complex autonomous actors who self-regulate their behaviors based on their psychology and circumstances, rather than passive components of human-machine systems whose role should be performing assigned computer-mediated functions [4, 16, 25]. This led to a paradigmatic shift in the field of HCI. It was acknowledged that the “user experience” is a complex phenomenon that transcends the boundaries of “usability” and the instrumental uses of technology, and that it should be regarded as a subjective relationship between the user, their environment and the technological systems involved in it, which depends on the user’s rationality as much as on their affectivity [17, 27, 30]. This context led to the increasing popularity of frameworks suitable to investigate the user experience as the product of technology-mediated meaningful activity, defined by the interplay of users’ cognition, affection and behavior, the context of activity, and the technologies involved in it [16, 25]. Activity Theory (AT) is one of such frameworks [23].

AT conceptualizes human activity as a process of purposeful interaction between subjects and their environment, driven by meaning-making that integrates cognitive, behavioral and affective processes to motivate, orient and drive conscious human acts [15]. According to AT, subjects pursue meaningful goals by attempting to transform objects in their environments, relying for this purpose on the mediation of tools and other subjects [6, 14, 28] (Fig. 1).

Fig. 1

Model of activity system (adapted from [15])

Tools are conceptual, material or digital artifacts that enable mental activity, communication with other activity participants, and the practical transformation of material objects. Community dynamics regulated by division of labor schemas generate collaboration possibilities that allow achievements otherwise unattainable by individual subjects. Interactions between a subject and other elements of the environment are regulated by rules. Besides tools and community acting as enablers, activity can be also influenced by artefacts and other entities that may interfere with a subject’s acts and hinder goal attainment [6, 23]. According to AT, subjects engage in activity motivated by basic physiological and/or psychological needs. Driven by these needs and based on their understanding of the external world, through meaning-making processes subjects form and adjust mental representations of reality, based on which they plan and evaluate purposeful interactions with the external world [15]. As they progress in their goal attainment attempts, subjects gain new knowledge about the external world, updating their mental images and plans accordingly [6, 28]. The mental images formed by subjects are influenced in equal measure by rational understanding of objective properties and relationships of relevant entities, comprehension of their socio-cultural valorization, and by motivational connotations that subjects may attribute to relevant aspects of reality, based on personal psychological factors such as desires, inclinations and perceived self-efficacy [3, 28, 36]. Thus, according to AT meaning-making can be regarded as the core driver of human acts, and integrates affection and cognition to define and regulate meaningful behavior.

In response to the need for integrative approaches to the study of the gameplay experience, and in line with the human-centric, integrative approaches of modern HCI, Fabricatore [15] proposed an activity-theoretical framework that identifies meaning-making as a central element of the gameplay activity. Fabricatore’s framework [15] conceptualized gameplay as an iterative and contextualized activity that is driven by meaning-making processes integrating rational interpretation and affective valorization of key game elements. Accordingly, the author proposed a systemic model of games reflecting the key game aspects that players interpret through meaning-making processes, complemented by guidelines to analyze and/or design game information flows suitable to promote and facilitate meaning-making processes.

The aim of the present study was to examine the relationship between the meaning-making processes supported by games and perceptions of their quality. For this, we drew on Fabricatore’s framework and guidelines [15] to identify potential features in games that could influence meaning-making in players. This information was used to evaluate 14 games using an ad-hoc questionnaire that was completed by experienced players. Results were then correlated to overall quality ratings provided by external sources and by the participants. In this paper we present findings and insights on important theoretical and methodological issues that may illuminate trajectories for future research.

2 A human factors perspective on meaning-making in games and the gameplay experience

2.1 Gameplay activity

Activity is an iterative process of interactions with reality driven by environmental information. Gameplay activity can be conceptualized as a multiple loop process consisting of hierarchical and iterative tasks. Through a gameplay task the player self-defines or accepts externally defined goals, evaluates environmental conditions, plans a course of action to attain the goals, executes the plan, and evaluates results. Information loops from the evaluation of conditions and results of action may lead to the re-formulation of goals and plans, or the re-evaluation of conditions and outcomes. The stages of a gameplay task are thus underpinned and driven by the processing of information flows provided by the game, either directly in response to player acts, or as environmental feedback representing the state of the game independent of player actions (Fig. 2).

Fig. 2

Gameplay as an iterative activity (adapted from [15])

Goal definition, planning and execution of actions, and evaluation of conditions and outcomes are sub-processes of a gameplay task that might be carried out simultaneously, leading to very quick decision-making and assessment of activity [6].

2.2 Gameplay activity network

As players perform a gameplay task, they iteratively attempt to transform a system of game entities (a target object) in order to achieve a desirable goal state. This process is mediated by a network of interacting game elements that function as enablers (tools and aiders) and hindrances (barriers and opponents), and is driven by information loops from provisional or definitive outcomes of action, and other aspects of the context in which the task unfolds (Fig. 3). Based on these loops, players evaluate outcomes of past acts, interpret possible functions of game elements, and plan future acts accordingly.

Fig. 3

Gameplay activity network (adapted from [15])

For example, an adventure game might require completing a quest through reaching a destination beyond a chasm. For this, through controlling an avatar the player might need to restore a hanging bridge using parts available in the game scene, and then cross the bridge to reach the destination. The player might also need to dodge nasty flying creatures that hover the chasm. In this case, the quest represents a task comprising two clear sub-tasks: restoring the bridge and crossing it. Throughout this whole process the avatar represents a tool that the player can directly control to interact with the environment (i.e. the player token). In the bridge restoration stage, the bridge represents the target object to be transformed (i.e. restored), while avatar and bridge parts represent tools to achieve this transformation. The avatar is a tool that the player controls directly, while parts are artifacts that the player can explore and assemble through interactions mediated by the avatar. In the bridge crossing stage, the avatar itself represents the target object to be transformed by the player through moving it to the target destination, and the whole bridge represents a tool required to mediate this transformation. Flying creatures represent opponents that hinder both the restoration and the crossing of the bridge, and must therefore be avoided by the avatar. Throughout the quest, the player will explore, evaluate and/or anticipate contextual conditions to identify parts required to restore the bridge, plan how to retrieve and assemble them, and plan how to cross the bridge, dodging opponents whenever needed.

2.3 Defining events

The state of the game is defined over time by events originated by the interaction of game entities, and influential on the player’s actions and decisions. Some of these events are the direct consequences of the player’s decisions and actions (player-triggered defining events – PTE). Other events may occur in the game as a result of interaction of game entities independent of the player’s decisions (non-player-triggered defining events - NPTE). In some cases, players might not be able to directly perceive an NPTE, even though they might be witness and be influenced by its consequences. As they unfold, NPTE generate a history of game state changes crucial for meaning-making processes.

2.4 Core gameplay schemas

Core gameplay schemas define, regulate and justify the player’s possibilities to transform elements of the game space, either directly or indirectly. Causal-mechanistic schemas determine interactions between game entities merely in terms of cause-effect rules. Socio-cultural schemas determine interactions within the community, between player and community, and among different social groups in the game. Workflow schemas determine gameplay progression through hierarchies of game tasks, establishing conditions required to initiate or complete a task, and dependencies between tasks. Core schemas may also determine interactions and occurrence of NPTE.

2.5 Game context

The game context can be considered as a system of interrelated conditions defining circumstances in which the player’s activity happens. All games can be regarded as contextualized systems of activities, but not all game contexts are equally rich. Comprehensive fictional contexts are defined by settings, storylines and overarching aims. Settings define key aspects of the time and place in which gameplay activities happen (e.g. natural environment, social communities, cultures, political systems and historical backdrops). Storylines can be viewed as narrative articulations of events related to the end game goals and the player’s role. Explicit game aims reflect the in-game socio-cultural meaning of game goals, and justify the necessity to achieve them.

Game contexts foster meaning-making through relating things and events within a specific spatiotemporal dimension, defined by socio-cultural, physical and historical environmental conditions. Contextual conditions allow the player to understand the core schemas that define and regulate interactions between entities, and to comprehend their significance of these interactions. Interpreting schemas in context allows to fully make sense of game entities, their relationships, the transformations required in order to achieve game goals, the enabling and hindering function that entities may have, and causes and effects of defining events.

The game context comprises a global and a local dimension. The local context consists of game elements and defining events that the player can directly perceive, and which may influence ongoing tasks and the planning of their development. In terms of meaning-making local elements and defining events are the player’s primary focus of attention. The global context consists of game elements and defining events which may influence the gameplay activity even though the player cannot perceive them directly. Comprehension of global aspects of the game context is therefore important for the player to understand implications of remote defining events, distant consequences of their acts, and consequently plan future tasks. The interplay between local and global aspects of the game context thus defines the player’s ability to understand the in-game proximal and distant “present”, make sense of the “past”, and have meaningful expectations regarding the “future”.

2.6 Gameplay and meaning-making

Through the iterative process of the gameplay activity meaning-making is required to define gameplay tasks, evaluate their outcomes and assess local and global environmental conditions and defining events. In order to pursue game goals, players need meaning-making to make sense of the state of game entities and their interactions, understanding “what” entities interact, “how” they interact, and “why” actions and interactions happen the way they do. Meaning-making thus drives the rational understanding of causal-mechanistic, socio-cultural and workflow schemas required to support effective and efficient interaction with game entities in pursuit of game goals. At the same time, players need meaning-making to comprehend what is deemed meaningful within the game. Players can then form a sense of what matters to them, attributing personal significance to their gameplay activities and fully embracing game goals and challenges. Meaning-making is therefore also pivotal for the player to attribute an affective valorization to the game space and the tasks they are involved in.

From an AT perspective games can be regarded as systems of meaning. Players continuously perceive and process relevant aspects of the game space, formulating, developing and updating meanings accordingly. Thus, all elements of the game space that the player can perceive may contribute to the generation of game information flows that ultimately feed the player’s meaning-making processes. Meaning-making processes are therefore bound by the extent to which the design of a game enables the player to actively explore things, events and their relationships, and consequently perceive relevant information on a timely and ongoing basis. Hence, games should be designed considering which game elements can contribute to game information flows, how, when and why [15], accounting for the guidelines presented in Table 1.

Table 1 Game information flow guidelines (adapted from [15])

3 Motivation and aims of this study

Based on Fabricatore’s framework [15], it is plausible to hypothesize that there is a relationship between affordances for meaning-making offered by different game features and:

• 1. Players’ perception of the quality of the gameplay experience (PP_QE);

• 2. The perceived quality of a game according to the general player public (PP_QG);

• 3. The perceived quality of the game according to experts (EP_QG).

Accordingly, we designed this study as an exploratory investigation aimed at identifying and interpreting relevant relationships between the above-listed variables. Variables PP_QE, PP_QG and EP_QG were identified as relevant for this study based on past research suggesting that player experience quality constructs partially overlap with quality ratings provided by experts or the general player public, and that there are significant differences in the way experts and players rate the overall quality of a game [21].

4 Methodology

4.1 Participants

The games were evaluated by 12 male and 2 female participants aged between 21 and 41 (mean = 27.71, SD = 4.9). They were from different countries including the UK (n = 4), Bulgaria (n = 4), Sweden (n = 2), and Croatia, Denmark, Italy and Romania (n = 1 each). Participant-players reported liking a wide range of game genres, being RPG, FPS, MMO and strategy games the most mentioned ones. Time investment in playing video games also varied between participants, ranging from 3 to 25 h per week (mean = 13.07, SD = 8.0). Participants’ fluency in English was judged as sufficient to understand and complete the items of the questionnaire.

4.2 Instruments

4.2.1 Meaning-making questionnaire

A bespoke online questionnaire was created in order to investigate participant-players’ perceptions regarding game features relevant to convey meanings, and the quality of their experience of playing that game. The questionnaire consisted in three sections. The first section aimed at gathering information about the participant-players and their satisfaction with the game. The second and third section aimed at collecting data regarding aspects of the game key to support meaning-making processes. All sections of the questionnaire were mandatory. In order to minimize the risk of inducing participant-players to focus on meaning-making when evaluating their experience with the game, participant-players’ evaluation of the overall quality of their experience with the game was collected before viewing questionnaire items specifically related to meaning-making, and could not be changed thereafter. In addition, the phrasing of the questionnaire items was formulated avoiding any explicit mention of meaning-making. This was also done to minimize the risk of biases, as we wanted to maximize participant-players’ possibilities to freely consider game features and evaluate their experience with the game.

The first section of the questionnaire presented items to identify the participant (nickname), the game being evaluated (title), and to gather general information on the participant’s demographics (age, gender and nationality) and play habits (weekly video gameplay time and preferred game genres). The perceived quality of the participant-player’s experience with the evaluated game was assessed through a single item on a five-point Likert scale (1: “very unsatisfying”; 5: “very satisfying”).

The second and third sections presented items organized in scales formulated based on the activity-theoretical, meaning-making-centered conceptualizations of game systems and the gameplay activity proposed by Fabricatore [15]. According to the gameplay process represented in Fig. 2, throughout the gameplay activity players self-define or accept externally-defined goals, evaluate environmental conditions that could affect goal attainment, plan methods to pursue the set goals and evaluate (provisional) outcomes of the actions undertaken. When evaluating environmental conditions and planning methods for goal attainment, players interpret how and why different elements (e.g. characters and objects) may mediate their interactions with the rest of the gaming world, either enabling or hindering goal attainment. Accordingly, players establish relationships between gameplay elements that mirror the activity network pattern represented in Fig. 3. A game should therefore continuously provide to players information useful to facilitate the definition of gameplay tasks [15], allowing players to understand, decide and/or forecast:

• i. “What” should be done (i.e. goals of gameplay tasks);

• ii. “Why” (i.e. purposes and values underpinning game goals and defining their meaningfulness and valorization in the game world);

• iii. “How and when” (i.e. methods to pursue game goals, and involved mediating elements);

• iv. “Consequences” that player actions and other defining events.

To measure the extent to which participants perceived that a game supported these four meaning-making functions, we formulated the following scales:

• 1. Goal definition/acceptance. This scale addressed the “what”, “how and when” and “why” meaning-making functions, by including three Likert items aimed at inquiring how frequently the game allowed the player to understand: goals of gameplay tasks; objects, characters or other entities that the player had to interact with in order to pursue task goals; reason why accomplishing game goals was valued and meaningful in the game world.

• 2. Workflow definition. This scale addressed the “how and when” meaning-making function in relation to activity planning. The scale includes four Likert items inquiring how frequently the game helped the player to comprehend: conditions allowing the initiation of a gameplay task; interdependencies between tasks, and their underpinning reasons; steps required to complete tasks.

• 3. Comprehension of mediating function of gameplay entities. This scale addressed the “what” and “how and when”, “why” and “consequences” meaning-making functions, by including five Likert items exploring how often the game allowed the player to comprehend: the state of objects, characters or other entities potentially influential for gameplay tasks; which objects, characters or other entities could interact with one another and/or with the player, how and why; enabling and hindering functions that objects, characters or other entities could have within gameplay tasks.

• 4. Comprehension of defining events. For the formulation of this scale defining events were conceptualized as events potentially influential on gameplay tasks [15]. This scale addressed the “how and when” and “why” meaning-making functions, through four Likert items exploring how frequently the game facilitated: comprehension and prediction of events that could influence the development of gameplay tasks and their meaningfulness; understanding of possibilities for the player to influence defining events; understanding storylines relevant to make sense of tasks’ progression and meaningfulness.

Game information flows should also support the evaluation of state and outcomes of gameplay tasks [15]. For this purpose, information flows should help players to evaluate:

• i. “Progression” state of gameplay tasks;

• ii. “Outcomes” of gameplay tasks;

• iii. “Consequences” that the outcome of a gameplay task could have might have.

To investigate whether participants perceived that a game supported these meaning-making functions, we formulated the following scales:

• 5. Comprehension of progression. This scale addressed the “progression” meaning-making function. For this, we formulated three Likert items inquiring how often the game allowed the player to comprehend when a task was initiated, completed, and the state of advancement of an ongoing task.

• 6. Comprehension of activity outcomes. This scale addressed the “outcomes” and “consequences” meaning-making functions, inquiring through two Likert items how frequently the game allowed players to understand or predict task outcomes and their consequences.

Information provided by a game is important to comprehend contextual conditions defining meanings of things, events and relationships that cannot be fully understood independently of their circumstances. Contextual conditions may be key to fully understand and/or predict context-dependent functions of gameplay entities, game events and their implications, and the valorization of tasks and events within the game world. Their comprehension may thus promote the player’s rational understanding of game entities, events and relationships, as well as the attribution of deeper significance and sense of purpose to gameplay tasks [15]. To support the comprehension of contextual conditions, game information flows should help players to understand:

• i. The “backdrop” of the game, consisting key in socio-cultural, historical and physical aspects, as applicable;

• ii. The “role” of the entity (or entities) controlled by the player in the game world;

• iii. “Value systems” depending on the backdrop and defining what is valued in the world, and why.

To investigate participants’ perceptions concerning how frequently a game supported these meaning-making functions, we developed the scale “Interpretation of task context: socio-cultural, historical and physical backdrop”, consisting in seven Likert items inquiring how frequently the game supported the above meaning-making functions.

All the Likert items in these scales were defined on a five-point scale recording frequency (1: “Never, or almost never”; 5: “Always, or almost always”). The option of “Not relevant for this game” was added to allow players to express the inapplicability of a specific statement to a specific game (for example, a statement regarding the comprehension of a storyline would be irrelevant for a game which did not involve any storyline).

Finally, the mode and timeliness of provision of information is key to support meaning-making in games. Based on Fabricatore’s framework [15], it can be argued that a game supports meaning-making processes by:

• i. Fostering the acquisition of information as the subject is actively engaged in exploring and transforming the game world, thus promoting learning about the game “by doing” rather than “by being told”;

• ii. Providing information timely;

• iii. Leveraging elements of the game world as direct sources of meaning-making information, either through the interactive game scene or through non-interactive cinematic events;

• iv. Leveraging the graphic user interface (GUI) as a key non-contextualized source of meaning-making information;

• v. Combining diverse means to convey meaning-making information to the player.

In order to investigate the extent to which players believed that the game implemented the above meaning-making functions, six Likert items were formulated, inquiring to what extent players agreed that the game supported the acquisition of information: through interaction with the game world rather than passive reception; using varied means; timely; leveraging elements of the game world (either from the interactive game scene or non-interactive cinematic events); leveraging decontextualized GUI elements. All these Likert items were defined on a five-point scale recording level of agreement (1: “Strongly disagree”; 5: “Strongly agree”). The option “Not relevant for this game” was added to allow players to express the inapplicability of a specific statement to a specific game. The full questionnaire is presented in Appendix 1.

4.2.2 Game ratings

In order to consider in our analyses the overall quality of the gameplay experience as evaluated by experts and the general player public, we used game ratings provided by the Metacritic website. This source was chosen because it publishes both professional and player aggregated game ratings [31]. Metacritic was also chosen because past research has found that its ratings are significantly representative of the quality of the user experience as measured through Player Experience of Need Satisfaction and the Game Experience Questionnaire [21].

Metacritic aggregated professional ratings (henceforth referred to as Metacritics) are calculated selecting and collating critics from reliable online sources, converting scores provided by the selected reviews into a score out of 100, and finally weighting these scores based on undisclosed criteria that account for the quality of the reviews. Aggregated user ratings (henceforth referred to as Metaplayers) are calculated as the unweighted average of scores provided by users [21].

For a given game, different versions may exist for different platforms. Metacritic provides ratings specific to each platform the game is available on. Different game versions may present major design adaptations due to platform/specific features and constraints (e.g. mobile vs. Virtual Reality vs. console versions). The ratings used in this study were calculated averaging Metacritic ratings for PC and console game versions. This was done in order to aggregate data only for game versions which were highly-comparable, and representative of the versions evaluated by the participants.

4.3 Procedure

4.3.1 Recruitment of participants

Participant-players were identified through snowball sampling. Snowballing is a non-probabilistic sampling approach whereby the researchers identify an initial small group of information-rich participants, who are then required to identify other participants with similar characteristics [11]. For this study an initial set of five key participants was identified, who then contacted other players inviting them to participate.

Participants were contacted, informed about key aspects of the research, and formally recruited after obtaining their informed consent. Participants were informed that this study was focused on researching quality in games and the player experience, and that they would be required to evaluate games they were familiar with, and complete questionnaires accordingly. No allusion was made to meaning-making topics, in order to avoid introducing biases that could condition participants’ responses to the questionnaires.

4.3.2 Selection and assignment of games

In order to select games for this study, each participant was asked to provide a list of games that they played over the past 24 months, and learned well enough to feel that they could explain to other players key aspects required to understand and successfully progress in the game (e.g. settings, goals, gameplay action possibilities, tasks to be accomplished to progress, etc.). Participants were advised to propose any type of game, including games that they actually did not like. The lists provided by participant-players were collated, and all games proposed by at least two different participants were identified. Accordingly, a total of 14 games were selected for this study (Table 2).

Table 2 Games evaluated

4.3.3 Data collection and analysis

Participant-players were assigned games ensuring that each game had two independent evaluators. Participants were then asked to complete one online questionnaire per game assigned. Participants were recommended to play and review the assigned games as much as they felt necessary in order to provide a reliable evaluation, and were given a maximum of five days to return the questionnaires. Players were blinded to the results of each other during the study period.

Data analysis was done in two phases. The first phase was aimed at examining the inter-rater reliability between participants, using intraclass correlation coefficients (ICC). A two-way mixed effect, consistency, average measures model was employed and rated using cut-offs qualifying values <0.4 as poor, between 0.40 and 0.59 as fair, between 0.6 and 0.74 as good, and > 0.75 as excellent [12]. Analysis was done including all the items of the questionnaire.

The second phase was intended to examine the relationship between the game features affording meaning-making processes, participant-players’ game experience and quality of those games as rated by critics and the general player public. In this phase, the analysis only considered games for which there was an acceptable level of agreement between participant-players, i.e. ICC values close or above 0.6. Meaning-making partial scores belonging to this subset of games were obtained by averaging items of the scales of the questionnaire. In addition, a total score was obtained by adding the scores of the scales (Overall MM). Questions assessing the timeliness and mode of provision of information were analyzed as single items. The standard Pearson product moment correlation was used to examine the relation between partial and total scores of the questionnaire, participant-players’ experience with the games, Metaplayers scores, and Metacritics scores.

5 Findings

5.1 Descriptive statistics

5.1.1 Inter-rater reliability (IRR)

IRR was used as an indicator that the evaluators ascertained similar meaning-making-affording features through their experience with a given game. Inter-rater reliability was rated as acceptable only in four of the analyzed games: Guild Wars 2 (ICC = 0.62), Overwatch (ICC = 0.65), Ori and the Blind Forest (ICC = 0.58) and Hearthstone (ICC = 0.55). Data related to these games were included in the second phase of analysis.

5.1.2 Differences between games

Table 3 presents the main results of the questionnaire scores referring to meaning-making affordances. The overall MM scores for the four games ranged from an average of 27.5 in the case of Hearthstone to 33.18 of Overwatch (maximum of 35), indicating that in general players perceived a relatively high frequency of affordances for meaning-making in all of the included games. Comprehension of Activity Progression and Comprehension of Activity Outcomes were the scales with highest mean scores (means = 4.90 and 4.88, SD = 0.20 and 0.23 respectively), while Interpretation of Task Context was the scale with the lowest mean scores and highest standard deviation (mean = 4.07, SD = 0.86).

Table 3 Overall and partial MM scores

Results from the single items assessing mode of provision and timeliness of information are presented in Table 4. Players’ perceptions varied across games, presumably indicating intrinsic differences due to genres and specific game design features. For example, players perceived that all the games except Overwatch greatly required exploration and interaction with the game world to make sense of the events and elements in the game. Conversely, Overwatch relies on other means for conveying meaning, such as GUI interfaces with icons and buttons. However, regardless of the sense-making affordance method, the provision of key information was considered timely in all the games.

Table 4 Mode of provision and timeliness of information

5.1.3 Player satisfaction and game quality

Overall game quality and player experience with the games was assessed by three independent measures, with varied level of similarity among them. Participants’ experience was reported as satisfying or very satisfying with all the games (Mean = 4.2/5, SD = 0.46). Expert ratings provided by the Metacritic website were also very high for all the included games (mean = 89.16/100, SD = 1.26), whereas Metaplayers scores tend to be less generous (mean = 7.28/10, SD = 1.20).

5.2 Correlations between meaning-making, player satisfaction and ratings of game quality

Results show that there are no significant correlations between the participant-players’ satisfaction with the game, Metaplayers and Metacritics ratings.

There is a high correlation between overall meaning-making scores and the Metacritics ratings (r(6) = 0.76, p = 0.027) but no significant correlations were found with the quality of participant-players’ experience or with the Metaplayers ratings. Metacritics ratings appeared to be positively correlated to the perceptions of affordances to understand tasks context (r(6) = 0.77, p = 0.025), while Metaplayers ratings appear as negatively correlated with the presence of game features supporting definition and acceptance of game goals (r(6) = −0.81, p = 0.16).

6 Discussion

This study aimed at identifying and interpreting relevant relationships between participant-players’ perceptions regarding meaning-making affordances in a game and: 1) participant-players’ perception of the quality of the gameplay experience; 2) the perceived quality of a game according to the general player public; 3) the perceived quality of the game according to experts. The meaning-making framework underpinning this study [15] conceptualizes gameplay as an iterative and contextualized activity, driven by meaning-making processes integrating rational interpretation and affective valorization of key game elements. This framework identifies meaning-making as central to any activity experience. However, the results of this study suggest that the relationship between the perceived quality of the gameplay experience and the overall game’s capability to afford meaning-making processes is unclear. Findings also make evident the challenges faced when attempting to study game-related aspects whose perceptions depend on personal interpretations and complex experiences.

6.1 Why are evaluators not agreeing?

An unexpected finding of our study was the low inter-rater reliability between players in 10 of the 14 games subjected to analysis. Low IRR may be due to a variety of interplaying reasons, some related to the formulation of the questionnaire, some due to the complex nature of the meaning-making constructs being explored, and some to the intrinsic variability of the player experience. Key reasons for disagreement between participants may include:

• 1. Games were non-linear. Thus, players might have experienced different aspects of the games, consequently forming different opinions on the elements that worked as meaning-making affordances, and their effects.

• 2. Perceived quality of the player experience was assessed through a single item. However, it is acknowledged that the player experience is a multifaceted phenomenon, and articulated instruments for its assessment have been developed accordingly (e.g. [20, 22]).

• 3. Meaning-making affordances were captured by Likert items evaluating the perceived frequency of meaning-making functions of game features. Perceptions on frequency might have been significantly different among participants, given that no reference was provided to indicate what each frequency level could mean. The games’ non-linearity could have also affected perceptions on frequency. By exploring different portions of an assigned game, evaluators could have been exposed to meaning-making affordances with different frequency.

• 4. The majority of games were multiplayer. Social communities are a key source of rational meanings and affective meaningfulness of activities [28, 36]. Depending on differences in players’ individual characteristics (e.g. inclination to socialize) and non-linear game situations explored (e.g. different quests), for some evaluators the importance of the community as a meaning-making affordance might have overshadowed the relevance of game features designed to serve the same purpose.

• 5. Players’ perception of the clarity of the questionnaire was not investigated, which may have resulted in them interpreting Likert constructs in unintended ways.

• 6. A low number of players was assigned to answer questions about each specific game, which may have caused a high degree of variance in the responses and imprecise results.

• 7. Meaning-making is a complex process that can be supported by how game components are designed and organized in the game world. However, elements external to the game world (e.g. personal interests, cultural beliefs, and the context in which games are played) can also impact players’ perception regarding the role that specific game features may have in subjective meaning-making processes [15]. In our study, the cultural diversity of participants may have affected the interpretation of concepts examined through the questionnaire items.

To address the above, we suggest that:

• 1. Controlled experiments should be conducted to more robustly test IRR, maximizing homogeneity in the scope, duration, and focus of the play experience. This could be done by developing a game review protocol requiring all reviewers to play specific stages of the selected games, in order to fulfill specific game objectives, and explore specific game areas, entities, events, and their relationships. The elements specified by the protocol could be selected prior to the experiment by expert evaluators, judging their potential relevance as instances of activities requiring meaning-making processes and/or affordances to support such processes.

• 2. The perceived quality of the gameplay experience could be assessed through a more articulated construct, as we discuss more in detail below.

• 3. Pilot testing of the questionnaire could be leveraged to further explore the conceptual soundness, cultural stability and perceived clarity of the items assessing the constructs.

7 References could be provided to facilitate a more objective evaluation of frequencies

7.1 Quality of the player experience and meaning-making: a multifaceted relationship?

The results suggest that the relationship between participant-players’ reported quality of the gameplay experience and the overall game’s capability to afford meaning-making processes is unclear. The lack of a significant association between these two factors is unexpected and may seem somewhat contradictory with the activity-theoretical framework underpinning this investigation, which suggests that meaning-making should be central to any activity experience [15, 23]. The results may be related to how the perceived quality of the experience has been evaluated (i.e. one single Likert item focused on overall appreciation). Past research has conceptualized and assessed the player experience as a multi-dimensional construct [7, 8, 27]. Some aspects of the player experience may depend on the satisfaction of psychological needs not strongly influenced by meaning-making affordance provided by a game, as in the case of the sense of social relatedness [22, 37]. This could justify, for example, why one of the participants reported having played and enjoyed the multiplayer game Hearthstone for years, but realized after completing the questionnaire that it was “badly designed”. It would be therefore interesting to assess the perceived quality of the player experience using multi-dimensional constructs such as the Player Experience of Need Satisfaction and the Game Experience Questionnaire [22], and then explore relationships between specific dimensions of the player experience and the overall capability of a game to support meaning-making.

An overemphasis on the cognitive dimension of meaning-making may have also influenced the participants’ evaluation of game affordances. Items were formulated focusing on the role of game features to “comprehend” game goals, gameplay and context, without stressing enough their importance on the feelings and value assigned by players to make sense of their own experience. Hence, it would be important to explore affective impacts of meaning-making affordances relevant to engagement and sense-making games. In this way, the different effects of rational and affective aspects of meaning-making processes on different dimensions of the player experience could be investigated in a more integrative way.

7.2 Meaning-making as a key to game quality: who are the best judges, players or experts?

We found a strong association between the overall capability of a game to support meaning-making and its quality as evaluated by experts. The relationship with the game quality evaluated by the general player public was instead weak and non-significant. The first correlation suggests that game meaning-making affordances may influence the perceived quality of a game. The weakness of the second association cannot be clearly explained based on the activity-theoretical framework underpinning this study. It does, however, call for reflection on possible differences in analytical skills between experts and players, and the methodological implications of this.

Johnson and colleagues [21] found that both players and experts’ Metacritic game quality scores are associated to different dimensions of the player experience measured through the Player Experience of Need Satisfaction and the Game Experience Questionnaire constructs. The authors also found that experts’ ratings are more strongly associated to dimensions of the player experience related to the perceived capability to choose and plan gameplay activities (autonomy), and accomplish gameplay tasks (competence). Based on these results and other associations found, the authors suggested that professionals may be “more clinical or objective than regular players – focusing on these formal aspects of the game, while regular players are relatively more influenced by their emotional response to the game” (p146). This highlights in the first place that experts might have attitudes and skills more suitable to identify and analyze important game features relevant to define the quality of a game. By extension, this suggests an interesting possibility. Competence and autonomy both relate to the ability to define, plan and evaluate activity, requiring a rational understanding of key elements of the game space. Higher analytical skills might allow experts to better identify and critically evaluate game aspects relevant to support this. Therefore, the experts’ evaluation of the overall quality of a game might reflect more strongly the game’s capability to support meaning-making processes related to task definition, planning, and evaluation. From a methodological perspective, this suggests that experts might be a better source of information in studies requiring the critical analysis of structure, functionalities, and impacts of games features accounting for both rational and affective aspects of the player experience. At the same time, it suggests that, as discussed in section 6.1, structured evaluation protocols might be beneficial to support and orient player evaluation processes, thus increasing the possibility of obtaining more comprehensive and focused player analyses of games.

7.3 What about “bad” games?

All the games evaluated in this study received positive player and expert evaluations in Metacritic (between 88 and 91 out of 100 for experts’ rankings, and between 6.1 and 8.7 out of 10 for players’). The evaluation of the player experience provided by participant-players through the questionnaire was positive as well (“satisfying” to “very satisfying”). It would be interesting to explore games that received negative evaluations. Players mostly satisfied with their play experience might miss relevant meaning-making aspects of games precisely because these are not problematic. In “bad” games players might struggle to build meaning, and this could allow them to more clearly identify game features that do/don’t support meaning-making as they should, and evaluate them accordingly. Hence, we believe that the use of both “good” and “bad” games could be beneficial for a more thorough investigation of the relationships between meaning-making and quality in games and the gameplay experience.

7.4 Limitations and future directions

This study presents several limitations, many of which have been already addressed in this discussion. In addition, we note that the questionnaire explored the perceived quality of the player experience but did not ask players specific questions about the perceived quality of the game. This could be influential on future research. Furthermore, issues regarding the sample size, questionnaire item development, and specific game characteristics are probably affecting the validity of the measurements and limiting the outreach of conclusions. However, the exploratory nature of the study did allow to examine the relationship between meaning-making and quality of games, and to propose lines of inquiry that may motivate further research.

Worthy of future research are issues of both theoretical and methodological nature, including conceptualizing the influence of game elements on emotional processes that affect meaning-making, and investigating the metric characteristics of the questionnaire’s items and scales. Further research should be focused on exploring the meaning-making influence of diegetic game elements - i.e. elements properly belonging to the fictional game scenario in which the game is set [19]. Supplementary data from this study not directly related to our aims suggest that diegetic game elements could have functioned as key meaning-making affordances. Staying focused on elements of the game world can enhance players’ cognitive involvement, sense of control and overall immersion [19, 37], and consequently their perceived quality of the gameplay experience [37]. This calls for further investigation of the importance of game scene elements to convey key meaning-making information to the player. From a methodological perspective, findings from this study provide valuable insights into the design of the questionnaire items. In future work, we plan to create a new version of this instrument and test its psychometric properties with a larger sample of participant-players.

In conclusion, we believe that this study and the underpinning meaning-making framework represent a relevant step to address the need for integrative approaches to the study of the gameplay experience, in line with the human-centric, integrative approaches of modern HCI.

Appendix 1

Table 5 Game meaning-making questionnaire