Keywords

1 Introduction

Within the field of Human-Computer Interaction (HCI), expert-based usability inspection methods are well established. The Heuristic Evaluation method, defined originally by Jakob Nielsen and Rolf Molich and later modified by Nielsen [1], in particular is widely known for being fast, relatively inexpensive and easy to learn.

In an interview with Jenny Preece [2], Nielsen stated “You can identify a lot of issues with a phone or other mobile user experience by using exactly the same heuristics as you would for any other platform”. While we agree that a Heuristic Evaluation could work well in the mobile domain, we do not believe that the set of heuristics defined by Nielsen is the best set of heuristics to evaluate the usability of smartphone applications.

To this end, this paper reports on an investigation of Nielsen’s statement by analysing 105 peer-reviewed papers in the field of usability evaluation, particularly an Heuristic Evaluation. The analysis informed the investigation into the applicability of each of Nielsen’s heuristics when applied to the usability evaluation of smartphone applications. This approach helps to point out where some of Nielsen’s heuristics might be deemed inapplicable to smartphone applications and where perceived gaps might occur during a usability evaluation.

Modifying this traditional set of heuristics to ensure as many usability issues as possible are discovered, as well as creating new heuristics to cover the perceived gaps, could prove effective in the usability evaluation of smartphone applications. Thus, it is hoped that this work will be most beneficial to HCI practitioners, educators and researchers as they seek to develop new and modified usability evaluation paradigms for smartphone applications. Ultimately this effort might lead to smartphone applications that are easier to use and easier to learn, resulting in less frustrated smartphone users.

2 Related Work

Other research teams have also investigated the applicability of Nielsen’s traditional heuristics to other domains. Examples include Educational Media [3]; Groupware Based on the Mechanics of Collaboration [4]; Ambient Displays [5]; Adaptive Learning Environments [6]; Playability of Games [7]; Intensive Care Unit Infusion Pumps [8]; Intrusion Detection Systems [9]; Virtual Reality Applications [10]; Electronic Shopping [11]; and Child E-learning Applications [12].

As mobile devices became more popular, researchers recognized that usability methods, including Nielsen’s traditional heuristics, did not work well within the mobile domain [13–15]. This led to research into a set of heuristics for the mobile domain, most notably from:

  • Bertini et al. [16], which concentrated on the operating system, the loss of, and the ergonomics of the mobile device;

  • Inostroza et al. [17], which concentrated on touchscreen devices, changing little from Nielsen’s heuristics, in a study which led to statistically insignificant results.

While also concentrating on the mobile domain, this investigation differs from previous research by concentrating on how Nielsen’s heuristics apply to smartphone applications. The resulting research can be used by HCI practitioners, educators and researchers in their quest to define a new set of heuristics dedicated to the usability evaluation of smartphone applications.

3 Investigation

The following investigation takes each of Nielsen’s heuristics in turn, investigating their applicability to the usability evaluation of smartphone applications based on an analysis of 105 peer-reviewed papers in the field of usability evaluation, particularly Heuristic Evaluation.

Traditional Heuristic 1: Visibility of System Status

– The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.

Investigation.

Keeping the user informed is vital on any system [18]. In addition to Nielsen’s first heuristic, Tognazzini [19] also included “Latency Reduction” in his First Principles of Interaction Design. However, within the mobile domain, this heuristic in itself might not be enough as smartphone application users tend to be even more impatient than desktop application users [20]. Therefore, ‘a reasonable timeframe’ might need to be much shorter for smartphone applications in order to avoid frustrating and confusing users. This, however, needs to be balanced as too many notifications will simply annoy users.

Traditional Heuristic 2: Match between System and the Real World

– The system should speak the users’ language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.

Investigation.

This is an important heuristic for smartphone applications. Developers should use techniques such as scenarios [21, 22] and list key characteristics of users [23] when initially starting work on the creation of a smartphone application. This allows for the identification of the type of tasks and users that will use the application. From this knowledge, a style guide containing the words, phrases and concepts familiar to the user would be selected for use on the interface [24]. The style guide will also help to ensure words, phrases and concepts are used consistently and in a natural and logical order throughout the application [25, 26]. Platform and industry standards can also be used as guidelines to apply consistent mapping to user interactions, including touchscreen gesture motions, to ensure interactions occur as users expect.

Traditional Heuristic 3: User Control and Freedom

– Users often choose system functions by mistake and will need a clearly marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.

Investigation.

The title of this heuristic can be ambiguous. For instance, within the mobile domain it could be associated with the ability to customize a smartphone application. In fact, the heuristic is related to navigation and user errors, both of which need to be heuristics in themselves. Therefore, when defining a set of heuristics for smartphone applications, a better solution might be to specifically call out these two areas separately. As heuristics about errors will be covered below, a heuristic that considers navigation would be clearer for evaluators.

Traditional Heuristic 4: Consistency and Standards

– Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.

Investigation.

As discussed in the second heuristic, consistency is important. This is where a style guide of terms to be used and look-and-feel can be beneficial. The guidelines set out by each mobile Operating System developer, including Apple and Google can be very helpful in creating smartphone applications that conform to expected standards.

While it is important to stay with known conventions, it is also important to experience to see what works well in the mobile domain. Therefore, this heuristic should not be seen as a barrier to the introduction to new ideas. A short FAQ or interactive help functionality might be used to introduce first time users to terms and gestures they may be unfamiliar with [27].

Traditional Heuristic 5: Error Prevention

– Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.

Investigation.

Both of Nielsen’s heuristics 5 and 9 concentrate on errors [1]. Bertini et al. [16] recognized this, deciding to combine both into a simpler heuristic titled “Realistic Error Management”, which is also our recommendation.

Yet, while combining errors into one heuristic is a simpler approach, as Nielsen’s alluded to, there are two important parts to errors; namely, preventing errors, and assisting users should an error occur in a way that is easily understood. Both need to be addressed within smartphone application heuristics.

In relation to error prevention, Hoekman [28] calls for smartphone application developers to practice poka-yoke. The term “Poka-yoke” roughly translates from Japanese as “mistake proofing”. Hence, a poka-yoke user interface is designed to prevent a mistake from being made. Myszewski [29] further defines poka-yoke by stating that “A poka-yoke solution makes the issue of human error irrelevant regardless what are causes and effects of the error. Elimination of human error becomes a technical question. If there is no possibility to apply poka-yoke, then some less effective solutions have to be considered”. Smartphone application developers can therefore create a more robust, user-friendly application by considering the errors users might make, including when they are using the smartphone in various contexts such as sitting or walking, then ensuring everything is done to eliminate the possibility of the error occurring.

If a situation exists where poka-yoke cannot be applied whereby potential errors cannot be prevented, such as a navigation error or an incorrectly typed word, users should be allowed to undo their mistake, go back to the previous screen or to exit the application easily if they wish to do so. Should the user try to attempt to proceed when it is not possible to do so, the application should display an error message that is understandable, letting the user know what the problem is, how to solve it and to quickly move on.

Traditional Heuristic 6: Recognition Rather than Recall

– Minimize the user’s memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.

Investigation.

Mobile interfaces tend to be small (generally between 2” to 6”). Therefore, making objects, actions, and options visible, as this heuristic requests, might not be a good idea within a smartphone application. A better practise is to ensure that only the elements and information needed right away should be displayed at any one time. However, it is important to ensure that vital information needed by the user is part of the objects, actions, and options displayed, with any other elements that are not immediately required being removed. This is also touched upon in heuristic 8.

Traditional Heuristic 7: Flexibility and Efficiency of Use

– Accelerators – unseen by the novice user – may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.

Investigation.

Accelerators are important to smartphone application users. Consider, for instance, the ability to install widgets on their device interfaces for easier access to important information. Smartphone application developers might also consider context along with customizability. Applications that fall into this category are for example those that change to a brighter screen in a dark room, or change interface elements to larger, easier to tap buttons when a user is driving or running.

However, this heuristic might be confused with heuristic 3: “User Control and Freedom”, which would need to be considered when defining a set of heuristics for smartphone applications. Whereas, heuristic 3 concentrates on the ability to recover from errors with undo, as well as navigation issues, this heuristic concentrates on customisability. This should be made clearer in any newly defined smartphone heuristics.

Traditional Heuristic 8: Aesthetic and Minimalist Design

– Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.

Investigation.

Aesthetic and minimalist design lends itself to a simple, easy-to-use interface, and is prominent in other design guidelines including those from Gerhardt-Powals [30] whereby one of the User Interface Design principles state “Include in the displays only that information needed by the user at a given time”. This is more difficult to achieve in practice, but should be a goal for all designers.

Focusing on the mobile domain, this heuristic seems applicable in its current form. However, the heuristic does not quite capture exactly what we need in the design of a smartphone application, which creates gaps when considering a set of heuristics for this domain. Ideally, a set of heuristics for smartphone applications should consider that applications need to be:

  • Simple – Hoekman [28] supports this by applying the Japanese system of the 5S’s (described in detail by Ortiz [31]) to mobile interfaces to decide what is absolutely necessary and what can be disregarded, arguing that every element chosen causes cognitive load on users.

  • Focused on one task – Though simple, the interface should have all the elements required to complete a singular task, be that to display the information a user needs, to show options or settings, to allow the user to interact with the application and so on. This allows single tasks, even if part of a larger application, to be completed quickly and easily, even by people on the move. Clark [27] reinforces this thought by reminding us that “The best [mobile] apps fold neatly into a busy schedule”.

  • Visually pleasing – While Clark [27] states that the beauty of an application is primarily in its functionality, Gong and Tarasewich [32] suggest that a mobile application “will stand out if it is attractive”. In practical terms, both are correct. A mobile application needs to be functional and aesthetic. Marinacci [33] even suggests that if an application is visually pleasing, a user would be more inclined to wait longer as they wait for information to download, more so than a mobile application that does the same thing, but does not look as good.

  • Learnable and Intuitive – Given the lack of in-depth help and documentation available for mobile devices, as well as the interruptions users face as they interact with mobile applications, an intuitive, easy-to-learn interface is vital [34]. In fact, it has been suggested that even though the learnability of a software application has always been important, it is even more important in the mobile context [35].

  • Glanceable by allowing main information to be taken in quickly – Chittaro [36] is an advocate of mobile application displays that can be viewed at a glance for people on the go. However, while Bertini et al. [16] also recognized the importance of glanceable mobile interfaces, the researchers decided to create one heuristic to cover very different problems in their heuristic “Ease of input, screen readability and glanceability”. Similarly another heuristic was created named “Good ergonomics and minimalist design”, whereby other aspects of the interface, as well as the design of the actual device itself, were discussed. Some of these issues are beyond the scope of a smartphone-deployed mobile application evaluation and the influence of application developers, as well as the fact that separate heuristics considered the mobile interface, it is proposed to create one heuristic that considers the items discussed in this section, while omitting items beyond the scope of a mobile application evaluation.

Traditional Heuristic 9: Help Users Recognize, Diagnose, and Recover from Errors

– Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.

Investigation.

This heuristic is vital for smartphone applications as much as it is for any type of interface. However, as Nielsen’s heuristics 5 and 9 consider errors, we have covered this in our investigation of “Traditional Heuristic 5: Error prevention”. We recommend that a single heuristic is defined for error prevention and assisting users should errors occur to simplify an eventual set of smartphone application heuristics.

Traditional Heuristic 10: Help and Documentation

– Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large.

Investigation.

While it would seem unlikely that a usable smartphone application would require Help and Documentation, from his findings Bertini et al. [16] suggested that “people using mobile applications still expect such applications to provide help. Though they might prefer that the help be ‘interactive’, non-distractive, not be a separate task.”

On a related note, Clark [27] observed that the use of an interactive welcome mat for first-timers showing an overlay pointing to the main features of the interface and how to interact with the application made the “apps first screen more inviting and helpful”. As a welcome mat is interactive, non-distractive and not be a separate task, it ties in with the findings from Bertini et al. Once first-time users interact with the application and discover its main features, they can dismiss the welcome mat and later become familiar with the more intricate settings, should they wish to do so.

In addition, there are as yet no established gesture guidelines for smartphone applications [38, 40]. Therefore, Help and documentation and/or first time user tutorials may offer a way for new users to learn how to use a smartphone application [27]. This is echoed by research that has suggested that many first time users will avail of tutorials to some extent [39, 40].

4 Closing the Gaps

Having investigated Nielsen’s traditional heuristics for their applicability in the usability evaluation of smartphone applications, we find that we are missing vital areas related both to the smartphone applications themselves, as well as their environment. To that end, HCI practitioners, researchers and educators interested in the development of usability heuristics for smartphone applications should also consider the following:

  • Context of use – The importance of glanceability was already discussed, whereby the context of use could be a busy user possibly mobile themselves while walking, cycling or driving. Other types of context of use include where users have to contend with poor lighting conditions and high ambient noise [41]. It is improbable to cover all scenarios and environments where a smartphone application is likely to be used, yet a set of heuristics dedicated to this domain should include a form of context-of-use so it is at least considered by evaluators.

  • Content Input – Smartphone applications are difficult to use from a content input perspective. Input tends to be more difficult and slower while on the move [42]. A set of heuristics for smartphone applications should consider this. For instance, to ensure users can input the content accurately, an application might have keyboard buttons that are as large as possible, and offer multimodal types of input entry [43].

  • Use of microphone, camera and sensors – Smartphone applications contain a multitude of features and complex sensors, such as a microphone, camera, GPS, accelerometer, and gyroscope [41, 42]. These can be used in any number of situations and should therefore be considered when developing smartphone application heuristics. For instance, consider applications that use these features and sensors to:

    • assist in content input;

    • display more data in landscape mode;

    • adapt an interface while running or driving;

    • inform users when friends are close by;

    • move objects in a game.

    • Identifiable Icon – an icon for a smartphone application should be aesthetic and identifiable as it is the only item a user sees when searching a mobile device interface for the application they wish to launch [27].

5 Conclusion

An Heuristic Evaluation is a low cost, effective, relatively fast usability evaluation method that has been popular since the early 1990’s. In this paper, we argue that without modification the method might not be as effective when evaluating the usability of smartphone applications. To back up our claim, we investigated Nielsen’s traditional heuristics and their applicability to the usability of smartphone applications by considering the potential problems and gaps with the method based on a review of 105 peer-reviewed papers.

Defining a set of heuristics that prove effective in the usability evaluation of smartphone applications could allow evaluators to discover a substantial number of usability issues prior to more expensive usability methods being used during the latter stages of the development life cycle. Ideally the set of usability heuristics developed for smartphone applications would maintain the quick-to-use, easy-to-learn, and inexpensive-to-conduct benefits of Nielsen’s traditional heuristics.

It is anticipated that this work might benefit HCI practitioners, educators and researchers in their quest to develop usability heuristics, and other usability evaluation paradigms, for smartphone applications. To take this work forward in developing a set of usability heuristics for smartphone applications, HCI experts might consider the information presented under each heuristic, subsequently modifying and removing heuristics, as well as covering gaps with the creation of new heuristics. A Heuristic Evaluation could be conducted to evaluate the effectiveness of the new set of heuristics against smartphone applications in sketch, prototype or final format.