Keywords

1 Introduction

Heuristic-based assessments remain a popular selection for practitioners when conducting usability testing due to the numerous benefits associated with the approach. In addition to being a relatively quick and inexpensive method, the approach is flexible across platforms, allows for utilization at various points during development, and requires a small group of experts to conduct [3], [1]. Despite these benefits, the method is associated with sub-par reliability across testers and lacks comprehensiveness in the analysis depending on implementation [1]. Additionally, reliance on human factors (HF) experts to conduct the analysis and provide feedback limits the type of design input produced by the method. As a consequence, reliance solely on heuristic-based assessment would mean a lack of end user input in the process. While this approach remains a vital tool during the usability lifecycle, resource constraints (e.g., budget, schedule, personnel access) often limit the variety of usability approaches that teams can implement.

The goal of the User Interface - Table for Evaluating & Analyzing Composite Heuristics (UI-TEACH) effort was to develop a validated heuristic approach based on a review of past literature and practice to provide an integrated system for heuristic analysis. In line with previous work [2], the method taken by the authors for the organization of the approach included defining heuristic categories and providing examples of upholding or violating the heuristics. This method takes advantage of the heuristic approach and provides qualifying information for less experienced practitioners.

This effort extended previous work in two primary ways. The first was to increase comprehensiveness. The Multiple Heuristic Evaluation Table (MHET) [2] was grounded in four paramount approaches: Nielsen’s ten usability heuristics [4], Shneiderman’s Eight Golden Rules of Interface Design [12], Tognazzini’s First Principles of Interaction Design [6], and a set of principles based on Edward Tufte’s visual display work [7]. The UI-TEACH expanded consideration of guidance. Specifically, the authors reviewed approximately 80 heuristic articles and identified almost 30 alterative heuristic approaches for consideration. The second difference was to conduct a content validation of the heuristics. While a qualitative analysis was the sole method underpinning the MHET, HF experts validated the heuristic categories through a card sort.

The long-term goal of this work is to leverage the UI-TEACH to develop an empirically validated heuristic-based tool to support expert-led usability assessment and generation of design guidance by end users. The concept is to rely on broad heuristics categories to organize the analysis, while providing practical examples to end-users, who likely will not be familiar or educated in HF or human computer interaction (HCI). These guidance examples should improve their capacity to identify important usability issues while helping communicate this information in a language that is understandable to system designers.

2 Heuristic Development Process

2.1 Qualitative Analysis of Literature

Revisiting the literature prior to an empirical validation allowed the authors to consider best practices, while taking advantage of progress made in other prominent usability guidance efforts. Expanded coverage supports early identification of issues and provides wider variety design guidance. Such guidance contributes to a user interface that enhances information processing and minimizes user workloads. A number of resources were identified and evaluated to ensure the UI-TEACH has included relevant, practical and comprehensive usability guidance to bolster and provide for a robust evaluation. The primary sources consulted include: The Evaluation Checklist [8], Research-Based Web Design & Usability Guidelines [3], Audience Centered Heuristics: Older Adults [10], Hedonomics: The Power of Positive and Pleasurable Ergonomics [11], Designing the user interface: Fourth edition: Strategies for effective human-computer interaction [12], User Experience Interaction Guidelines [13] and Principles of Accessible Design [14].

The aforementioned review of the literature provided the basis for our qualitative analysis. The analysis involved identifying overlap of conceptually related heuristics, and identification of unique aspects of approaches for consideration to expand and address gaps. The results of this analysis yielded 17 preliminary heuristic categories from a set of 250 pieces of design guidance. In order to validate the resulting heuristics and organize the design guidance, the authors’ used a card sort discussed in the next section.

2.2 Card Sort

Prior to conducting a closed card sort, the research team conducted a pilot open card sort to inform the number of categories expected beyond the qualitative input. The results of this pilot were evaluated using (N = 18 HF majors from a southeastern university) a hierarchical cluster analysis, which revealed a division of 10 categories (Range = 5 to 17, M = 11, Mode = 14). A follow-up qualitative analysis of those groupings resulted in the following general descriptors of the categories: Help, User Efficiency, User Control & Interaction, OS Properties, Data Management, Graphic Design & Aesthetics, Memorability, Cognitive Facilitation, Feedback, and Learnability.

Participants (N = 25) for the closed card sort were recruited from a Navy command and held positions in the fields of engineering (systems, industrial, human factors), psychology (human factors, engineering, industrial/organizational), instructional systems, or student interns in related fields. They were instructed to group design guidance within the 10 categories previously mentioned. Instructions also included a caveat that up to two additional categories could be formed if a participant felt it was necessary to provide a comprehensive set of heuristics. While this varies from a traditional closed card sort, the flexibility allowed participants to identify if a group of cards did not conceptually fit within an existing category, but they felt that placing it in a trash group would result in a loss of critical information.

Analysis. A co-occurrence matrix was completed for each participant to identify how cards were grouped. Next, a sum of the individual co-occurrence matrices results formed a collective co-occurrence matrix as a means for developing a probability matrix. While a hierarchical cluster analysis helped inform the development of categories, the result of an orthogonal Varimax rotation confirmatory factor analysis was the predominate method for interpreting results.Footnote 1 The team established low factor loadings (i.e., < 0.05) and loading on multiple factors as criteria for card removal, which reduced ambiguity in the data. Although statisticians typically reserve factor analyses for studies with large amounts of data, Capra [14] suggests that it is possible to use a factor analysis for card sort data when there are enough cards and participants because it generates the statistical power needed for factor analysis.Footnote 2

3 Results

Analyses resulted in the formation of 10 heuristics within the UI-TEACH approach:

  • Graphic design and aesthetics: Interface display elements (e.g., color, text, graphics) and layout support a positive user experience;

  • Error handling and feedback: System feedback on status and errors supports users’ understanding of how to interact with the system;

  • User interaction control: Mechanisms that allow the user to feel in control of actions and system preferences;

  • Learnability: System design and aids support users learning how to use the system;

  • Effectiveness of developmental characteristics: Characteristics of the hardware/software compatibility that affect the ability of the system to deliver the intended functionality and detect errors;

  • Memorability and cognitive facilitation: System design helps ease learning and memory load (short-term and long-term memory);

  • User efficiency: System design & functionality that supports completion of tasks with minimal time and effort;

  • Consistency: System information & actions are consistently located and formatted throughout the interface; and

  • Help: Readily accessible instructions or clarifying information that are easy to use and support task completion.

The number of heuristics is consistent with established methods (e.g., Nielsen [3]; Shneiderman [11]; Tognazzini [5] ), and leverages similar descriptors for consistency. The UI-TEACH approach differs from the majority of existing methods by establishing a list of examples to provide elaboration for less experienced usability testers. Specifically, each heuristic is supported by a set of examples that demonstrate adherence to (i.e., support) or violation of a heuristic (see Table 1). Based on the results of the card sort analysis, heuristics were backed with between 67 (Graphic Design & Aesthetics) and nine (Learnability) examples.

Table 1. Example Heuristic Category and Set of Examples Outlined in UI-TEACH Approach
Full size table

4 Discussion

The UI-TEACH provides an alternative heuristic-based usability approach meant to simplify access to usability concepts for a range of testers while increasing the comprehensiveness of previous approaches (e.g., Atkinson et al. [2] ). The authors acknowledge that the design guidance within UI-TEACH is predominately focused on graphical user interfaces and does not provide specific guidance related to multi-modal interfaces; additional examples within this area would increase the generalizability to more systems. Additionally, while the authors validated this approach through a card sort, additional validation through implementation of the method would further demonstrate how the UI-TEACH can facilitate the assessment process. One way the authors are continuing this aspect of testing is through the validation of the assessment tool that was developed in follow on phases of this overall research effort. At this time, preliminary testing of this tool – the Experience-based Questionnaire for Usability Assessments Targeting Elaborations (EQUATE) – demonstrates that end users identify a range of issues touching all heuristic categories [16].