1 Introduction

This paper presents a proposal for audiogame evaluation based on usability, accessibility and gameplay heuristics. For the scope of this work, audiogames are defined as audio-based games that have blind persons as final users. As a result, we expect the proposed evaluation method to be used during the development cycle of audiogames. This way, we seek to reduce the negative aspects of user experience and, at the same time, to identify and maintain the positive ones that may go beyond usable audiogame design.

Game usability is defined as the degree to which a player is capable of learning, controlling and understanding a game [1]. A system’s usability is often defined in terms of effectiveness, efficiency and user satisfaction in a given usage context, along with the ease of using and learning to use the system [2]. Regarding games, these premises are questionable, since they must be more pleasant and entertaining when presenting users with challenges. Thus, the traditional definition of usability is insufficient to characterize the quality of interaction in digital games [3], not to mention that efficiency and effectiveness are secondary criteria for user satisfaction [4] when talking about games. Therefore, applying the general usability heuristics [5, 6] for game evaluation is not enough.

Many game evaluation methods originated in [1, 4, 7, 8]. Federoff [4] created 42 heuristics categorized into game interface, game mechanics and gameplay. Based on the study in [4], Desurvire et al. [7, 8] proposed heuristics organized into gameplay, game story, game mechanics and usability. Pinelle et al. [1] proposed 10 heuristics aiming at usability problems.

For audiogame evaluation, accessibility issues must be included [9]. Miao et al. [10] discussed the application of different usability tests with blind, partially blind and sighted users. In their study, they presented the differences in conducting tests and in users’ preferences. Petrie and Bevan [11] introduced a range of methods to evaluate accessibility, usability and user experience with information about their appropriate use, strengths and weaknesses. Sánchez et al. [12–14] evaluated different audiogames with blind persons based on usability and user satisfaction criteria. Park and Kim [15] suggested serious game guidelines based on game accessibility, which includes web accessibility, game contents accessibility and gameplay interface accessibility. Campos and Silveira [16] and Yuan et al. [17] described obstacles to digital accessibility that should be avoided in interactive systems and presented alternative input and output devices and interface styles, which attempt to provide a better experience for users with disabilities.

Based on these studies, we chose to adopt evaluation by inspection, in which evaluators explore the game interface using a set of heuristics. The process for developing the usability, accessibility and gameplay heuristics was based on the methodology presented by [18–20], which contains 6 steps. These topics, as well as the detailed heuristics for the evaluation of audiogames for blind persons, are presented and discussed in this paper.

This paper is organized in the following way: Sect. 2 presents works related to game evaluation; Sect. 3 presents the research process employed in formulating heuristics; Sect. 4 presents a proposal for usability, accessibility and gameplay heuristics for audiogames; finally, Sect. 5 presents some conclusions and future work.

2 Related Work

The ISO 9241-11 standard defines usability based on efficiency, effectiveness and user satisfaction criteria. The accessibility issues taken into account by [6] are learnability, memorability, efficiency, safety and satisfaction. However, for games, another analysis is required: one that focuses on user satisfaction over efficiency and effectiveness. [21] defined gameplay as an evaluation tool with 4 components: functional, structural, audiovisual and social playability. Gameplay is related to intuition, fun, challenge and social interaction, when the game is multiplayer [22]. This way, a game has good gameplay when its interface is intuitive and discreet, so that the player can focus on playing, which should be adequately difficult and engaging.

Many game evaluation methods originated in [1, 4, 7, 8]. Federoff [4] discusses usability criteria applied to games and emphasizes that a product’s usability cannot be evaluated without considering its context. Thus, she concludes that effectiveness, efficiency and satisfaction are not equally important or applicable when talking about games. Federoff [4] initially conceived 30 heuristics, which were revised after the pilot test with game design professionals. The compiled list of game heuristics was organized into 3 categories and 40 heuristics: game interface (13), game mechanics (2) and gameplay (23). To them are also added those which fit into more than one category, which are game interface and play and game mechanics and play.

Based on the study in [4], Desurvire et al. [7, 8] proposed heuristics organized into Gameplay, Game Story, Game Mechanics and Usability. Desurvire et al. [7] adapted a set of heuristics for productivity software to games. The result was Heuristic Evaluation for Playability (HEP). The HEP heuristics were grouped into four categories: Gameplay (16), Game Story (8), Mechanics (7) and Usability (12). According to [7], HEP is helpful in early game design because it facilitates thinking about the design from the user’s point of view. [8] have refined HEP, producing a new list called Game Usability Heuristics (PLAY), intended to help game developers during the entire design process, particularly at the beginning of the concept phase when changes to the design are less costly. The general principles were grouped into categories of heuristics evaluated by question: Gameplay (6 heuristics, 22 questions total), Coolness/Entertainment/Humor/Emotional Immersion (4 heuristics, 4 questions total), Usability and Game Mechanics (9 heuristics, 24 questions total). Some examples of heuristics in the Gameplay category include: enduring play; challenge, strategy and pace; consistency in game world; goals; variety of players and game styles; players’ perception of control. One of the advantages described by authors [8] is that the PLAY proposal is modular. This way, a story-less game may be evaluated without the questions related to this heuristic.

Pinelle et al. [1] proposed 10 heuristics aiming at usability problems. The process used to develop the heuristics included identifying usability problems in game design, developing a set of categories by grouping similar usability problems together and creating heuristics for avoiding common usability problems in games.

There are other guidelines for game evaluation. Korhonen et al. [22] propose 29 heuristics organized into three modules: Gameplay, Game Usability and Mobility. Soomro et al. [23] propose 4 categories (Gameplay, Usability, Mobility and Multiplayer), which add up to 10 heuristics for mobile games, which can also be used as guidelines by game developers. Korhonen and Koivisto [24] present and describe playability heuristics for mobile multiplayer games. Game accessibility is presented and discussed by [9, 15, 17, 25].

3 Usability, Accessibility and Gameplay Heuristics to Evaluate Audiogames

For this study, we created a set of heuristics that can be used as guidelines during the stages of modeling, developing and evaluating an audiogame.

3.1 Research Process

We used the methodology proposed by [18–20] to create the usability, accessibility and gameplay heuristics for the evaluation of audiogames for blind persons. This methodology comprises 6 steps:

  1. 1.

    Exploration: to collect bibliography related to the main topics of the research. We selected bibliography concerning audiogames [14] and game evaluation [1, 4, 7, 8], as well as 18 audiogames and 33 games, which were said to conform to accessibility recommendations. Three evaluators with experience in usability and games in general took part in this step.

  2. 2.

    Description: to highlight the most important characteristics of the previously collected information, in order to formalize the main concepts associated with the research. Evaluators faced usability issues that were listed in order to be categorized according to game heuristics and Nielsen’s heuristics [5, 6], without revising or modifying them.

  3. 3.

    Correlation: to identify the characteristics that the usability heuristics for specific applications should have, based on traditional heuristics and case studies analysis. One evaluator from the previous step took part in this stage, along with 23 undergraduate students who were enrolled in the Human-Computer Interaction course. 102 questions were written to be used as a guide in an evaluation by inspection.

  4. 4.

    Explanation: to formally specify the set of proposed heuristics. This stage allowed the evaluation instrument to be reorganized, eliminating questions that were still present in more than one category. The instrument was reduced to 72 questions.

  5. 5.

    Experimental validation: to check new heuristics against traditional heuristics by experiments, through heuristic evaluations performed on selected case studies. Using the instrument modified by step 4, 6 audiogames were re-evaluated.

  6. 6.

    Refinement, based on the feedback from the validation stage. We carried out successive evaluations to consolidate the set of heuristics proposed for audiogames evaluation. This step provided for a better description of the categories of the proposed heuristics and the commonly identified problems. The instrument was reduced to 44 questions. As a result, we produced a document with identification, definition, explanation, examples and advantages for each heuristic, along with a description of every possible problem that can arise when the heuristic is not taken into account.

4 A Set of Usability, Accessibility and Gameplay Heuristics for Audiogames

Based on the method used, we constructed a list of heuristics for audiogame evaluation, containing usability, accessibility and gameplay heuristics. For each heuristic, there is a definition, an explanation on how to conform to it, example questions, what to avoid when developing audiogames and the related benefits and problems. This method evaluates audiogames that do not have a graphical interface, due to the scope of the research. Should the evaluation be carried out on audiogames with a graphical interface, then questions concerning layout, size and shape of the elements must be included.

Heuristic 1: Visibility of System Status

Definition. The audiogame should keep the user informed through audio about actions that are relevant to the game.

Explanation. This heuristic should prioritize the means through which users will check their score and level and how objects and other characters move. One option is to allow the score and the level to be checked via keyboard shortcuts. For the location of objects and characters, a good option is to use 3D audio.

Avoid. Keys with too many functions. Each key should have a specific function. If many functions are accessed with a single key, the user should be notified about this change.

Example questions. Can the user know their score and level during the game? If approaching or moving objects, is it possible to perceive their location during the game? In mobile audiogames, are vibration effects recognizable?

Benefits. Better use experience, because the user may react to each situation accordingly while playing the audiogame. The user can be aware of the general context of the game.

Problems. When this heuristic is not observed, the user may not know about their progress during the game, possibly feeling frustrated.

Heuristic 2: Similarity Between System and Real World

Definition. The audiogame should use the most natural language possible.

Explanation. The audiogame should avoid the use of technical jargon during the game. Information should be presented in a natural and logical order. The language is advised to be that of the targeted audience or of the game’s genre.

Avoid. Words in another language or phrases that belong to other gaming genres.

Example question. Are the concepts used in the audiogame comprehensible?

Benefits. Errors are minimized because the player is used to audiogame concepts. Better understanding of the concepts related to audiogames.

Problems. The user may feel confused when playing the audiogame.

Heuristic 3: User Control and Freedom

Definition. The user should feel in control of the audiogame.

Explanation. The audiogame should allow the user to leave, save different states, return, pause and cancel actions at any time. The application should allow the audio to be adjusted. Keyboard shortcuts are advised for such actions. The state saving functionality can be automatic and, when the game starts, it can load the status and scenery according to the user’s level.

Avoid. The mouse should be avoided as a tool to select elements from the interface, as well as to move objects that follow the cursor’s position, due to blind persons’ lack of precision when using a mouse for this purpose.

Example questions. Does the user feel in control of the application? Can they save the game? Can they return to a previous point? Can they fast-forward the audio queue? Can they rewind the audio queue? Can they adjust audio playback speed? Is it possible to adjust the volume? Can the user select objects on the screen? Can they move the objects?

Benefits. Better use experience. The user feels free to explore the application.

Problems. Without the necessary commands, the user will be unable to control the game.

Heuristic 4: Consistency and Standardization

Definition. Audiogames should be executed via consistent and standardized actions.

Explanation. For games that have a graphical interface, this heuristic may relate to element layout on screen. In the case of audiogames, which do not have graphical interfaces, consistency and standardization relate to the actions that the user must carry out in order to interact with the game. In this context, it is associated with clicking and interacting with menus such as “play”, “return”, etc. Controls for taking actions should, when applicable, follow the game industry standards, always produce the same results and be available throughout the entire game. The choice of interaction mechanism should remain the same during the game, including key bindings and menu option layout, for example.

Avoid. Varying the control interface and audio features during the game and choosing unusual key combinations to perform actions.

Example questions. Is there coherence between game controls and their actions? Do keyboard shortcuts follow the game industry standard, when one exists? Do controls remain the same throughout the game? Is menu option navigation standardized? Are the key bindings consistent? Is there a standard for audio volume? Does an element’s type of audio remain the same throughout the game?

Benefits. User’s confidence may be increased, since actions can become more intuitive.

Problems. User may not understand what must be done in the game and what level they are in.

Heuristic 5: Error Prevention

Definition. The audiogame should prevent the user from making mistakes.

Explanation. In games that have a graphical interface, error prevention can be achieved by disabling unnecessary menu options without removing them from the interface. For audiogames, it is advised that disabled menu options be read with a different kind of voice or in another tone. Keys that perform no action during the game should be disabled. This prevents involuntary user actions from causing errors.

Avoid. Keeping options enabled when they should be disabled. Avoid irreversible actions.

Example questions. Can the user identify when a menu option is disabled? Does the audiogame disable keys that are not used during the game? When the user chooses to leave the game, is confirmation required? Is it necessary to save the game manually?

Benefits. Prevent errors from happening.

Problems. The amount of errors during gameplay may be increased.

Heuristic 6: Recognition Rather than Recall

Definition. The user should recognize what to do when playing the audiogame rather than memorizing it.

Explanation. To allow the user to enhance their skills, the game should be simple to play and memory should not be required for every interaction. For games with a graphical interface, options’ icons should be a reminder of their functions. In audiogames that have a menu, options should convey their function through audio. Keyboard shortcuts should be in accordance with what they represent. Future skills should be acquired before they are needed. Sounds for game elements should remind the player of what they mean, especially icon sounds.

Avoid. Menus with too many options or long explanations.

Example questions. Is the menu easy to understand? Are keyboard shortcuts easy to remember? Do objects’ sounds remind the player of what they mean?

Benefits. Reducing the player’s memory requirement also reduces their mental effort to understand the game.

Problems. The user may feel tired during the game, for they will have to rely on their memory in order to access information on how to play better.

Heuristic 7: Flexibility and Efficiency of Use

Definition. The audiogame should be flexible and efficient, so it can be used by various user profiles.

Explanation. Games usually have different levels of difficulty that require different sets of skills. The game should be flexible so that skills can be developed during the game. It is important for it to have customization mechanisms such as setting key bindings, volume control and sound playback speed. The user should be capable of skipping to the desired piece of information without listening to the entire audio.

Avoid. Large number of keys used simultaneously.

Example questions. Are the controls customizable? Are keystroke sequences easy to perform? Are all controls necessary? Is the choice of simultaneous keystrokes adequate? Does it provide for efficient use by different user profiles?

Benefits. Greater efficiency, helping advanced players and beginners alike.

Problems. May cause the player to lose interest in the game.

Heuristic 8: Aesthetic and Minimalistic Design

Definition. The audiogame should have an aesthetic and minimalistic design.

Explanation. An aesthetic and minimalistic design refers to presenting strictly necessary information to the user, including audio content, visual elements and control actions. In audiogames, this heuristic applies directly to the diversity of sound types and quality, as well as to the use of a haptic interface, when available. The user should be able to tell distinct sounds apart and identify the different situations in which the haptic interface is enabled.

Avoid. Excessive use of different types of audio. Avoid long messages.

Example question. Is the sound quality appropriate? Is the amount of sounds appropriate? Is the usage of the haptic interface appropriate? Is its vibration intensity appropriate?

Benefits. User exhaustion is minimized: too many sounds and vibrations in unnecessary moments cause the user to be upset or distracted.

Problems. Users may have problems telling the various sounds or vibrations apart.

Heuristic 9: Help Users Recognize, Diagnose, and Recover from Errors

Definition. The user should understand when an error occurs and be able to recover from it.

Explanation. In games that have a graphical interface, when the user makes a mistake, a window may be displayed containing further information and an option to close it. In audiogames, information is always conveyed in the form of audio. Thus, messages should be clear and the user should have the choice to pause, repeat or fast-forward them. Furthermore, the message should tell the user clearly what to do to return to a safe situation in the game, for example.

Avoid. Interference or excessive use of error message audio alongside gameplay audio. Screen change for help items.

Example questions. Can the user recover from an error? Does the audiogame provide information on how to leave an undesired state?

Benefits. May minimize frustration, since knowing how to recover from an error reduces the annoyance of it ever happening.

Problems. User may not know how to leave an undesired situation, which may cause frustration.

Heuristic 10: Help and Documentation

Definition. The audiogame should provide help and documentation for the user.

Explanation. Audiogames, as well as games in general, should supply relevant information so the user can learn how to play. Information in an audiogame should be conveyed via audio, especially things such as which keys to use during the game and how to interpret each sound. Help should be available at all times during the game and it should be supplied according to context. Also, a tutorial may be available.

Avoid. Exceedingly long explanation audio.

Example questions. At the start of the game, is the user given enough information to understand it? Is the user given help information according to the context they are in?

Benefits. Better understanding of the game and how to play it. Beginners may become more experienced by going through the documentation.

Problems. Increased error rate, since the user may not know how to use the application.

Heuristic 11: Gameplay

Definition. The audiogame should have gameplay.

Explanation. This refers to having an inviting story, with given goals and rules, levels of difficulty and rewards for the player. Additionally, the game may offer several ways to achieve the same goal.

Avoid. Having the user discover required skills by themselves.

Example questions. Does the audiogame have a clear goal? Does it present different levels of difficulty? Does it present challenges? Does it reward experience, i.e. does the character become stronger as levels are surpassed and secondary goals are achieved? Does the audiogame allow the user to practice a skill, be it physical, mental or social? Does the audiogame offer different ways to achieve its goals?

Benefits. The game becomes more entertaining.

Problems. User will not have motivation to carry on playing.

Heuristic 12: Accessibility

Definition. The audiogame should be accessible to the user.

Explanation. The audiogame should have the capability of being used equally, securely and autonomously by users with disabilities. The chosen kind of platform must be regarded, since the way to interact with devices may vary. On desktops, the keyboard is prioritized, whereas on mobile devices the only interface is the touchscreen. Moreover, mobile devices can be used in different screen orientations (portrait/landscape), so the content must adjust to them and be easily reachable in graphical interfaces. More important pieces of information should be presented first.

Avoid. Conflicting information in graphical interface. On mobile devices, avoid the use of navigation actions that differ from the smartphone’s native screen reader standards.

Example questions. Are the more important options presented first? Can the user access options quickly? Can the user start the game from the sound interface without having to activate the screen reader? Should a screen reader be used, is the information accessible?

Benefits. More equal and autonomous use. Obstacles that prevent access to information and gameplay are eliminated.

Problems. Gameplay may be made difficult or unfeasible.

5 Conclusions

Many sets of heuristics for game evaluation have been proposed [1, 4, 7, 8, 22, 24, 26]. However, there are no heuristics currently formalized for audiogame evaluation. In this work, we adapted existing heuristics and created others to cover usability, accessibility and gameplay for audiogame evaluation. They were presented and discussed.

The methodology used to develop the heuristics allowed us to perform an iterative process [18–20], resulting in the refinement of the heuristics and the reduction of the number of questions from 103 to 44. The presented heuristics are reliable for use in software development cycles, but it is still important to perfect them and re-validate them in future work.

As for future work, we intend to analyze as developers and evaluators from the field of game design use these heuristics.