Behavioral Telemetry in Games User Research

Abstract

Within the past few years the adoption of business analytics has provided powerful new tools to the interactive entertainment industry, giving rise to the field of game analytics. Where traditionally user testing was limited to samples, it is today possible to obtain behavioral telemetry data from entire populations of players, and to map second-by-second interactions in the user testing lab. In this chapter, the focus is on the behavioral side of user experience in games, rather than user experience itself. The chapter outlines what behavioral telemetry is and its role in game user research from an introductory, top-down perspective. The chapter also introduces data mining as a toolbox that is available for analyzing large or small telemetry datasets. Finally, several case studies are used to showcase examples of how behavioral telemetry data can be used to evaluate game design in order to optimize the user experience.

The Pagonis Network

Conclusions

Behavior analysis via gameplay metrics analysis addresses one of the major challenges to game user research, namely that of tracking and analyzing user behavior when interacting with the very complex systems that contemporary computer games represent. As a user-oriented approach, it complements existing methods utilized in the industry, providing detailed and quantitative data to supplement qualitative and semi-quantitative data from other user research methods on the quality of the play experience (Lazzaro and Mellon 2005; Kim et al. 2008; Isbister and Schaffer 2008; Drachen and Canossa 2011; Seif El-Nasr et al. 2013). Alternatively, behavior analysis can be used as a method for inferring player experience, although care need to be taken when doing so and secondary validation is recommended (Lazzaro and Mellon 2005; Marsh et al. 2006; Nacke and Drachen 2011; Drachen and Canossa 2011).

There are many ways to work with and utilize behavioral telemetry data, during production and post-launch, and it can be challenging to generate systematic overviews of methods and principles: The choice of approach is influenced by the available resources for telemetry logging, user research and not the least the game design. For example, traditional boxed, fire-and-forget games are different from persistent world massively multi-player online games and social online games, where an important focus is analysis and synthesis directed at monitoring the player population, tuning game design on a running basis, continually optimizing the play experience to tune game design on a running basis, and calculating business-oriented game metrics such as the average revenue per user and churn rate (Fields and Cotton 2011). Even within the confines of a single type of game, e.g. first-person shooters, there is a substantial space for developing approaches to working with gameplay metrics. On top of this come the problems in correlating behavioral data with robust and broadly applicable measures of play experience (Kim et al. 2008). Despite these challenges, there is a general consensus forming in the game industry and game academia that telemetry-driven behavior analysis mesh well with other user-oriented approaches for evaluating and testing games (Kim et al. 2008; Zoeller 2011; Seif El-Nasr et al. 2013; Luton 2013; Seufert 2014). Game analytics is here to stay.

Box 1: Drill-Down Analysis

The first analyses done on gameplay telemetry are typically high-level analyses and descriptive statistics, meant to provide an overview of how the game is doing. For example, looking at the number of player entering and leaving the game (if a live game), or the amount of time spent across different levels. These types of high-level views on datasets from pre-launch playtesting or post-launch live games provides one of the early means for detecting potential problems. However, they rarely show why specific problems occur. For example, when players are spending twice as much time on level 5 as intended, something we did not plan for is happening. To find out why, it is necessary to dive down into the data and find the root causes of player ­behavior. This process is known as drill-down analysis” and is a core method in data mining (Larose 2004; Han et al. 2005; Drachen et al. 2012).

Drill-down analysis is one of the most common forms of operations carried out in game analytics, and the term is commonly heard in game data mining contexts.

This is because the root causes of behavioral patterns are often nested at deeper levels of the behavioral data than what is apparent at high-level aggregate analyses. For example, a specific checkpoint malfunctioning for 25 % of the players, or only on some platforms. Drill down analysis, as a data mining operation, basically means moving from summary/aggregate information to detailed data via a particular focus. For example, noticing that a group of players have suspiciously high gold income rates in an MMORPG, and working down through the summary data to investigate the raw data of these particular player, to figure out if cheating is happening or not. When performing drill-down, we are essentially performing analysis on the parent attribute (e.g. gold income rate), via investigating child attributes (e.g. time spent playing, sources of gold income). At the lowest level of a drill-down analysis are the raw data

In practice, when using business intelligence applications, drilling down is performed by selecting and querying data. This can be as simple as clicking a bar in a bar chart and getting the underlying data, to running queries in a relational database or writing scripts for a particular drill-down path. How far drill-down analysis can be taken depends on data granularity.

To take an example (Fig. 7.9) consider a simple breakdown of data consisting of a few variables is considered, e.g. average completion times for the levels of a game. At this top level, it may be noticed that a level appears to take longer to complete than others (see graphic). In order to explore why, the underlying data need to be exposed, in this case a breakdown of the completion times for individual components of the level. In this more detailed view of the data, it may be noticed that a specific sector of the level is where the players spend a lot of time (Main Hall). Drilling down further into the metrics data captured from the level, it may be found that the root cause is that players have trouble beating a specific type of enemy and keep dying (Evil Conductors), whose difficulty can be adjusted to accommodate.

If the cause of the observed pattern is not obvious from looking at the data, it can be useful to consider the actual game environment, as in the Halo cases mentioned in this chapter. This is about the closest we can get to the actual experience of playing the game, for example via mapping player telemetry directly into the virtual environment (Drachen and Canossa 2011).

Box 2: Managing the Allure of Numbers

For people working with game user research, behavioral telemetry is usually valuable. They form potentially highly detailed data about player behavior, and can be obtained from very large groups of people. Quantitative data take the form of numbers, and many popular metrics calculated in the game industry and –academia are intuitively understandable, e.g. Monthly Active Users (MAU) (Luton 2013; Fields and Cotton 2011; Seif El-Nasr et al. 2013). It can therefore be tempting to rely on behavioral telemetry alone following the launch of a game, and ignore lab- or field-based user research (see e.g.: Pagulayan et al. 2003; Isbister and Schaffer 2008). However, this is dangerous because behavioral telemetry does not permit analysis of e.g. user experience beyond what can be inferred from the telemetry trails themselves.

In game analytics, is it important to keep in mind that data can only say something about the data themselves, not about what other data that should be collected, or factors existing outside the dataset, which impact the results being generated. Furthermore, any measures of human behavior cannot include all the factors that influence the behavior. Assumptions therefore need to be made when working with behavioral telemetry in GUR just like for any other data source on human behavior and PX in games.

Strong knowledge of the principles for empirical research and knowledge acquisition are essential to the successful deployment of behavioral game telemetry in game user research.

Many common KPIs used in the game industry require contextual information to be used to drive decision making. For example, knowing that the number of Daily Active Users (DAU) increased 20 % last week, does not explain why this increase occurred, or whether it is sustainable, and it can even hide a problem—for example that the increase is due to a new feature, which however removes a key motivator for IAPs, thus reducing the Life Time Value (LTV) of the player base in the same period of time, endangering the financial survival of the game.

As noted by Drachen et al. (2012) critical thinking should always be applied when evaluating player behavior—sometimes the analysis will show one thing through a red color on a heat map or another suspicious pattern in the data, but the problem may actually rest in a minor design detail somewhere else. Small design changes in one area of a game can cause changes in behaviors in an entirely different section. Just because something looks good does not mean it is true, and this is why careful analysis is needed when suspicious or problematic patterns are detected in behavioral data. Heat maps and graphs are often intuitively understandable and travel better in organizations than two pages of text with detailed explanation of a specific finding from a comprehensive user test. However, data visualizations also make it easy to ignore other factors that could potentially hold an impact on whatever is being investigated, but which is not included in the metrics-based analysis in question. Metrics analysis, data mining, etc. all requires a human element—the numbers do not make things right, human interpretation does (Han et al. 2005). The key lesson here is that game analytics is not design (Fields and Cotton 2011; Seif El-Nasr et al. 2013), but can be an incredibly help to design, providing hard evidence on how design decisions are affecting player behavior. Just like any other method for game user research, good research practices should inform game analytics. (© Game Analytics, used with permission, www.gameanalytics.com)