Keywords

1 Introduction

With the rapid economic growth of the computer game industry, the rising popularity of games and complexity of game-play call for systematic examination on play experience (PX) and playability.

According to the discussions that computer game systems are different from utility software [2, 13, 19] in many aspects including the user experience (UX), we can reasonably argue that a different set of playability heuristics should be introduced for computer games. Moreover, the resulting set should cover an array of common game design elements besides user interface.

As of now, only a handful playability heuristics [3, 6, 13, 17] have been identified by previous research. There are three major flaws in these heuristics: (1) the number of heuristics is small compared to traditional usability guidelines; (2) these heuristics are usually derived from a small dataset. They may not cover the most critical problems from players’ perspective; and (3) these heuristics tend to focus on few types of games and may not be applicable to a wider spectrum of computer games.

Provided the success in exploring computer game traits by our revised lexical approach [24], the objective of this study is to initialize a set of playability heuristics through a similar process. Specifically, we argue that by incorporating nouns and gaming jargon terms, more context and subject information will become available for deciphering observed patterns. If a pattern proves to be prominent from interpreting the original reviews, it should be translated into a potential heuristics.

This study attempts to fill in the gap in game playability research and it aims at developing comprehensive playability heuristics through a lexical analysis of frequently-used nouns and adjectives by users. The paper is structured as follows: (1) playability and playability heuristics, (2) the lexical approach and prior lexical analysis, (3) lexical analysis of nouns and adjectives, and (4) results and discussions.

2 Playability and Playability Heuristics

There is no standard definition of playability presently. Usability-First [20] defines playability as “the degree to which a game is fun to play and is usable, with an emphasis on the interaction style and plot-quality of the game; the quality of gameplay”. Sánchez et al. [18] view playability as the UX in videogames and define it as “a set of properties that describe the Player Experience using a specific game system whose main objective is to provide enjoyment and entertainment, by being credible and satisfying, when the player plays alone or in company.” Paavilainen et al. [16] suggest that playability is the combination of gameplay and user interface, which can be further connected to concepts of intuitiveness, unobtrusiveness, fun, and challenge. Since the cores of these definitions lies on either game-play or usability, confusions exist between playability and usability. Above all, there are few playability heuristics that game professionals can easily apply with consideration of different aspects of gameplay.

To introduce some existing heuristics, Malone [13] pinpoints three motivational factors of designing enjoyable interfaces for educational games: (1) challenge, (2) fantasy, and (3) curiosity. The subsequent analysis generates a design framework, which has a far-reaching significance for ensuing playability research. Federoff [6] publishes the first playability heuristics set for the games of entertainment purposes. The set is developed in the light of traditional usability heuristics and game design principles. Desurvire et al. [3] create a heuristics set consisting of 43 items that fall into three categories. Korhonen and Koivisto [10, 11] become the first to develop playability heuristics for games running on mobile devices.

More recently, Pinelle et al. [17] develop a set of heuristics through analyzing 108 game reviews from GameSpot.com. Important issues are identified and classified from the reviews of certain PC games. They then translate initial findings into the principles at high level; however, the resulting set is more toward user interface problems thus less applicable when it comes to other design factors. With the trend of social networking games, Paavilainen [15] introduces an initial set of social game heuristics according to two design frameworks [9, 21]. The heading-level heuristics contain 10 items such as spontaneity and narrativity.

Each of aforementioned heuristics has its own strengths. However, they fail to cover views and opinions from the majority of players who have different tastes for games. This study strives to address this issue by examining a massive amount of online game reviews. Because these online reviews are tantamount to user-generated content from which players’ experience is directly reflected, they will likely provide rich information and serve as a fertile ground for developing playability heuristics and/or design guidelines.

3 The Lexical Approach and Prior Lexical Analysis

The concept of lexical approach stems from the lexical hypothesis to study personality taxonomy in personality research. The lexical hypothesis states that when salient individual differences become socially relevant to human life, these distinctive attributes are likely to be encoded into nature languages. If many people recognize a difference, the difference may be expressed by similar words. Therefore, personality traits can be identified by exploring personality descriptive adjectives in natural languages. Through three basic steps summarized by Ashton [1], the original lexical approach helps discover five factors that later become the well-received Big Five personality factors: openness, conscientiousness, extraversion, agreeableness, and neuroticism [8].

Motivated by the lexical approach to understand human personalities, a revised lexical analysis is adopted in this study to analyze adjectives in online game reviews [22, 23]. This revised method has extracted six significant factors in game play experience: playability, creativity, usability, competition, sensation, and strategy. Based on the adjectives, a simple game design framework can be drawn, specifically, competition, sensation, and strategy can be regarded as the stimuli factors stimulating game enjoyment; on the other hand, playability, creativity and usability can be used to measure to what extent a computer game is successful.

However, the analysis of adjectives alone does not fully disclose the subjects and contexts when players write the reviews. Therefore, playability heuristics (i.e. design guidelines for computer games) are still some distance away. In this study, we plan to incorporate nouns in the lexical analysis because nouns may indicate the subjects described by the adjectives and therefore suggest more context-specific information. It is argued that a lexical analysis of the combination of nouns and adjectives will likely achieve two objectives: (1) Discovery of general patterns among online game reviews and (2) Context-specific information concerning different aspects of player experience. These two findings will likely lead to a prolific set of playability heuristics that would be a great help to game developers.

4 Lexical Analysis of Nouns and Adjectives

Two phases were involved in developing playability heuristics based on nouns and adjectives.

4.1 Phase I: The Lexical Analysis

There are mainly four stages in the revised lexical approach adopted in our early work [22, 23]: (1) Collecting online reviews, (2) Building a list of game descriptive adjectives, (3) Extracting ratings of adjectives from game players, and (4) Exploring patterns through factor analyses.

In our previous work [22, 23], we have downloaded 821,122 reviews from three major game websites (e.g., gamespot.com, gamestop.com, ign.com). The nature of those reviews varies self-reflection on game-play to objective introduction of game design and/or technical facts. Since nouns (including game jargons) are added in this study, we took some additional steps based on the work we’ve done in Stages 2 and 3 before [22, 23]:

  • Step 1: Identifying nouns/noun phrases and jargon terms from original game reviews;

  • Step 2: Extracting player ratings of nouns and jargon;

  • Step 3: Refining the list and conducting factor analysis.

Step 1: Identifying Nouns/Noun Phrases and Game Jargon Terms.

Four tasks were involved in this stage: (1) Parsing individual lexicons from raw texts and checking the part of speech (PoS); (2) Detecting nouns and phrases as non-adjectives in the sentences; (3) Dropping stop-words; (4) Registering overall frequency and the number of game reviews containing a word. Natural Language Processing (NLP) applications were developed using Perl to complete these tasks.

Since user-generated reviews are often poorly written with informal languages, this study adopts a simple parsing strategy. The NLP program examines a word’s sense semantically instead of analyzing the entire sentence syntactically. WordNet [7, 14] is referred to as the main reference because it offers a wealthy lexical library that documents words of four PoS. Meanwhile, it provides a comprehensive set of senses for each word.

Based on 821,122 reviews, 21,535 distinctive nouns/noun phrases are found. The numbers of documents containing a noun range from 1 to 100,532. On the other hand, there are approximately 4,327 jargon terms carrying an absolute frequency ranging from 8,033 to 10.

Step 2: Extracting Player Ratings of Nouns.

At this step, the online reviews were transformed to a matrix by a computer program as follows: (1) Each noun or jargon was regarded as an individual item. The terms were relabeled as the field names of the matrix table (2) All online reviews were retrieved one at a time. Each review about one game was processed as an individual document. If a term appeared in this review, the value for this noun or jargon (field) was set to 1. Otherwise, a zero value was registered. There were 821,122 records from which the ratings were obtained.

After variances of all terms were computed, those low variance terms causing computation errors were excluded. As a result, 4,342 items were retained in the final analysis. These terms were combined with the adjectives list produced in our prior lexical analysis [22] to form a new list. This new list was used as source dataset for the subsequent factor analysis.

Step 3: Refining the List and Conducting Factor Analysis.

This step commenced from an exploratory factor analysis with Varimax rotation using SAS. Un-weighted least squares (ULS) method was employed, and communalities were estimated by square multiple correlations (SMC). 147 factors were obtained from the first round of analysis. Redundant information was noticed among the 147 factors. In other words, many factors seemed similar. To address this problem, we consolidated nouns with similar or identical meanings as what we dealt with adjectives. The process generated 3,044 groups of nouns. The items within each group are not only semantically relevant, but also statistically correlated.

After combining the 788 adjective groups from prior analysis, 1298 popular game jargons, and 3,044 noun groups, a new hybrid list of 5,130 terms was finally formed. It served as the base for an updated rating matrix by a slightly different conversion process. In particular, for each review, the number of distinct adjective or noun terms from the same group appearing in the same review was used as the value of this group in this review. A second factor analysis was then carried out.

The lexical analysis of nouns and adjectives led to 97 factors. Each factor was loaded with a group of nouns and adjectives relevant to each other for some reason. The next task was to interpret why and how these nouns and adjectives are related.

4.2 Phase II: Developing Playability Heuristics

In Phase II, each factor was analyzed in the following four steps in order to formulate playability heuristics: (1) Step 1: Sorting terms in a factor according to factor loadings, (2) Step 2: Identifying terms that may contribute to potential heuristics, (3) Step 3: Composing initial heuristics, (4) Step 4: Refining playability heuristics. Factor #31 is used as an example to illustrate these different steps.

Step 1: Sorting Terms in a Factor According to Factor Loadings.

Ascript was written to rank the terms in a factor based on terms’ loading values. The Part-of-Speech (PoS) tags were retained during sorting. It became immediately noticeable that highly-loaded terms in a factor may suggest a prominent game play context. Taking Factor 31 for example, the first three factor descriptors - “Japanese”, “Anime” and “Turn-based” clearly represent the inherent characteristics of Japanese-RPGs. It is reasonable for them to stand out among other terms as players may have frequently used these three as generic descriptors. Other words such as “Anime-like” and “Japanese-style” with lower loadings also confirm this presumption. Those context-oriented terms introduce a primary context/background to understand the factor. Table 1 lists the words loaded on Factor #31.

Table 1. Nouns, adjectives and jargon terms loaded on Factor #31
Full size table

Step 2: Identifying Terms that Contribute to Potential Heuristics.

The list of each factor usually consists of nouns and adjectives with continuous factor loadings. This step is designed to short-list truly useful words for drafting playability heuristics. The entire list was studied in relation to the context. To ensure the validity of any playability heuristic to be developed, original player reviews were systematically examined.

Commonly-used words that hardly convey much useful information were first excluded. The following five types of words were considered irrelevant: (1) Terms reflecting a component in the game such as plot and character. In factor 31, some of the words are “daughter”, “girl”, “son”, “dragon”, “gust”, and “incest”. (2) Terms used in the title of a game or sequel. They can be a popular element in game storylines as well. With regard to Factor #31, some of these items are: “boy”, “atelier”, “sonata”, “fencer”, “saga”, and “radiance”; (3) Terms entailing meta-data related to a game or sequel. “Composer” as a generic noun term is often mentioned in reviews of Final Fantasy. Reviewers keep mentioning its composer because of his excellent work on the soundtrack. Another term “Square-Enix” is the developer of Final Fantasy; (4) Terms emphasizing the theme or style of a game or sequel. In Factor #31, those items are: “spiky-haired”, “anime-like”, “over-world”, “Japanese-style”, “rpg-style”, “adult-oriented”, “j-pop”, “zelda-like”, and “anime-ish”. (5) Terms describing generic features of gameplay. Those items loaded in Factor #31 are: “mid-battle”, “action-rpgs”, “friendship” and “auto-battle”.

As the result of this step, a shortened list of words was used to query the original reviews. Finally for Factor #31, 16 words were chosen for this purpose (see Table 1), and the first three context descriptors on this list are “Japanese”, “anime”, and “turn-based”.

Step 3: Composing Initial Heuristics.

Original reviews combined with nouns and adjectives belonging to a factor were systematically examined. The initial interpretations were then converted into playability heuristics in the light of general user-interface design principles. Playability heuristics were stated as specific as possible in order to preserve pertaining contextual information.

The process started with examining reviews that contain highly-ranked terms. In factor 31, the words “Japanese”, “Anime”, and “English” were used to query original reviews. This pattern in fact expresses players’ support for keeping original voice-acting in certain themed games. Although language options are often available, most players prefer the original “Japanese” voice-acting, as it narrates stories better than “English”. Since storytelling is an effective tool in user engagement, this finding is surely a unique discovery about game play. 146 online reviews with similar suggestions supported this heuristic.

“Side-quests” and “side-quest” encode the request for a fair playfulness in non-primary tasks. The reviews tell us that there should be plenty of fun side-quests so players will stay entertained. In addition, these secondary tasks may help players better explore the game world. 2,823 online reviews with similar suggestions supported this heuristic.

For factor #31, 12 out of 71 terms contributed to 10 raw playability heuristics. Table 2 presents two sample heuristics.

Table 2. Sample playability heuristics based on Factor #31
Full size table

Step 4: Refining Playability Heuristics.

Two tasks were performed to refine the first draft of playability heuristics:

  • Additional information was collected from original reviews to substantiate each drafted heuristic. The number and content of reviews related to each heuristic were analyzed. Sample reviews supporting a heuristic are presented along with it. Heuristics were re-worded to better reflect player comments. Heuristics without strong support in original reviews were removed.

  • Similar heuristics were combined together and consolidated.

For Factor #31, 10 drafted items from previous phase were consolidated into 7 heuristics. In total, 116 playability heuristics were developed based on all 97 factors. They are grouped into three categories: playability, creativity, and usability. Table 3 shows some basic statistics about them. Through retrieving relevant reviews, it is noted that the number of online game reviews supporting a heuristic ranges from 51 to 7,611. Although some of these numbers might seem small in comparison to the total amount of reviews analyzed, they should be viewed in a very specific context rendered by a unique creative vision. The number of reviews per game ranged from 1 to 6,021 with an average of 56. Some playability heuristics may aim at certain type of games. Nevertheless, the factor analysis in the early lexical analysis has statistically substantiated the lexicon patterns that were used to derive the playability heuristics.

Table 3. Statistics about playability heuristics
Full size table

5 Results and Discussions

The lexical analysis of nouns and adjectives produced 116 playability heuristics. All of these heuristics have been verified and are supported by original player reviews. As shown in Table 3, 50 of these playability heuristics replicate the results from previous studies [4, 6, 10, 13, 17]. For instance, factors 17 and 26 are concerned about intruding or interrupting game-play. After reading original reviews, the corresponding heuristic in usability category was phrased as “a game should minimize the odds of pausing or interrupting user play during a mission.” A similar rule, “the interface should be as non-intrusive as possible,” has been discussed before under the scope of game interface [6]. Factor 51 helps us find a heuristic about character likability in game stories. As the result of examining original player reviews, the heuristic in the category of playability was formed as “a game can be humorous through its characters and stories”. Although it sounds uncommon, a similar rule has been accepted as a final heuristic after large-scale survey studies [4]. The successful replication of 50 heuristics suggests that our findings are consistent with prior research and that the heuristics identified in this study are valid.

66 heuristics are newly discovered. This study substantially extends the pool of playability heuristics that game developers may apply. Since these heuristics were developed based on a fairly large amount of player reviews on a wide spectrum of computer games, they are able to cover many aspects of game-play that have never been possible in a-prior research. For instance, one heuristics advises designers to incorporate more connectivity features to support multiplayers mode. This becomes more demanding given the recent trend of mobile games, especially in a collocated setting. On the other hand, a couple of new heuristics may have been inexplicitly mentioned before, but they are much more elaborated this time. One example would be about replayability. Although Federoff [6] calls for designing re-playable games, there is a lack of details on how to achieve replayability. This study was able to produce 9 heuristics for re-playability, each of which proposes a specific strategy to improve replayability.

The playability heuristics have been developed to preserve specifics as much as possible. Although some researcher prefer a short list of heuristic rules, we argue that more specific heuristics are easier to use and are less likely misinterpreted by game developers. Because these playability heuristics were developed based on direct feedback from player, the specifics in the heuristics are accurate and will no doubt be a helpful source to game developers.

Overall, the discovery of the 116 playability heuristics indicates that the revised lexical approach is in fact effective and valid. By combining nouns and adjectives, the lexical analysis leads to discovery of lexicon patterns that contain specific contextual information. These patterns can then be used to develop valid playability heuristics. To summarize, the main advantages of the playability heuristics developed in this study include: (1) They are supported by a large number of online reviews and they truly represent player views on computer games. (2) They are based on the most critical issues in player experience since only the factors accounting for the most variances would be able to emerge from the factor analysis. (3) They provide much-needed specifics that cannot be easily derived from general design heuristics or theories. (4) They are comprehensive. These playability heuristics encompass a wide spectrum of computer games and different play scenarios. (5) The playability heuristics are expressed primarily by player language. They are easier to comprehend for game developers who often are also players themselves.

6 Conclusions

This study employed a revised lexical approach to analyze nouns and adjectives from over 800, 000 online game reviews. As a result, 116 playability heuristics were proposed. While these playability heuristics encapsulate an increasing amount of specifics, they also cover a wide range of topics about computer games. These playability heuristics can serve as useful design guidelines for different types of games. They clearly demonstrate the practical contributions of this study. From the theoretical perspective, this study provides strong evidence proving the effectiveness of the revised lexical approach. When mingling nouns and adjectives together, the revised lexical approach is able to discover the most important patterns in user experience with rich contextual information. These patterns can then be referenced to prepare design guidelines of a system or product. This new method can be easily extended to other fields for analyzing a large amount of online reviews on any user-oriented system or product.

To reiterate the rationale of this study, it is suggested that a lexical analysis of the combination of nouns and adjectives lead to two types of assets: (1) general patterns among players’ reviews and (2) context-sensitive information associated with different aspects of players’ experience. These two findings help introduce a prolific set of playability heuristics that would be of a great practical value to game researchers and professionals. Considering such a large dataset directly from game players, the resulting heuristics provide much-needed specifics to game professionals. Meantime, they cover a wide range of topics about computer games of various genres.

This new approach equips qualitative researchers with a rigorous, quantitative, and more controllable solution to analyze large amount of user-generated content. Its implications to the field of HCI and IS in general are profound.

Nevertheless, this study has its limitations as any other study would have. The present study used one coder to interpret the lexicon patterns and to develop playability heuristics. Although the playability heuristics proposed so far are all supported and verified by original player reviews, some potential heuristics might have been missed due the subjective reading of lexicon patterns. The language in the heuristics may also be refined to better serve game designers and/or developers. As the future research, we plan to hire additional game experts to continue the analysis of the lexicon patterns and develop new playability heuristics based on our revised lexical approach.