Keywords

1 Introduction

Designing Graphic User Interfaces (GUIs) for complex applications such as those used to support naval operational and training systems is a challenging endeavor. The challenges arise from the balance between appeasing the functional needs of the end-user population with ensuring the usability of the system. Despite the challenges, the goals should always be to develop systems that are accessible, easy to use, make task completion more efficient, reduce the cognitive workload on the end user, and generally satisfy the user. Even the most seasoned computer programmers cannot develop such systems without early, frequent and prolonged end-user feedback. Such feedback has traditionally been solicited through a process of usability testing. While no single definition of usability exists in the literature, it is generally defined as the quantifiable characteristics of a system that provide information to developers about the ease of use and usefulness of the system [1, 2, 5]. Literature regarding usability evaluations suggests they are necessary for developing usable interactive systems [3, 4]. While the challenges of conducting usability testing for Navy systems can be unique (e.g., domain specificity, complex systems of systems), they are not immune to the benefits of such evaluations. The following report details an initial usability evaluation of the Post Mission Assessment for Tactical Training and Trend Analysis (PMATT-TA) Increment 1 web-based application using a novel, heuristic-based measure called the Experience-based Questionnaire for Usability Assessments Targeting Elaborations (EQUATE).

1.1 PMATT-TA System Purpose and Goals

PMATT-TA Increment 1 is an online web-based application and database designed for tracking important data points from operational and training events for the maritime patrol community. The post mission data captured includes a variety of missions (e.g., mission type, communications) and contextual information (e.g., weather). The ultimate goal for capturing this data is to facilitate debriefs with aircrews to identify strengths and areas for improvement. By centrally storing post mission information, PMATT-TA also supports data calls related to trends (e.g., number of events a crew has participated in). Additionally, data increments of PMATT-TA will address a need for a digitally-based system that streamlines and automates the process of data collection, analyses, and feedback. As a result, the system benefits a range of users (e.g., aircrew, instructors, squadron, or group leaders). While the demand for PMATT-TA within the Navy is apparent, it has not yet been subject to formal usability analysis. Usability analyses must be conducted to be confident that the system achieves its goals, reduces post mission reporting time and is operable by end-users.

2 Data Collection

2.1 Recruitment

Participants (N = 9) evaluated the PMATT-TA Increment 1, web-based application in a laboratory setting with a desktop computer. After attaining an informed consent from qualified participants, researchers provided a brief explanation of tasks and measures as they moved through the protocol.

2.2 Measure

After completing the PMATT-TA tasks, each participant was given the EQUATE. It was developed based on an extensive review of the extant literature regarding system design and heuristic evaluation [6]. The review identified a number of items (i.e., 250) that qualified as design guidance. Through further testing eight heuristic dimensions (i.e., Error Handling & Feedback, Graphic Design & Aesthetics, User Interaction Control, Memorability & Cognitive Facilitation, User Efficiency, Learnability, Consistency, and Help were included within the EQUATE). Validation in the form of internal consistency and discriminate validity was ongoing at the time of this study but preliminary analysis has demonstrated sound psychometric properties (i.e., average internal consistency for all dimensions < .85).

3 Results

The quantitative analysis is intended to provide a general overview of the usability of the PMATT-TA system and a localized evaluation of the usability of system and system components across the heuristic categories previously mentioned (e.g., Learnability, Help, and Consistency).

The analysis revealed both positive and negative elements within the PMATT-TA system (see Table 1). Usability was assessed on a 5-point scale (i.e., 1 = Strongly disagree, 5 = Strongly agree, 0 = Not applicable). Items on the EQUATE are framed in both positive and negative terms (e.g., The design provided a pleasant experience and There was too much clutter on the display). Negatively framed items were reverse coded prior to analysis. Higher averages imply better usability while lower averages imply potential usability issues exist.

Table 1. Descriptive statistics for Evaluator responses on the EQUATE (N = 9)
Full size table

The average visibility across all items of the EQUATE was acceptable (M = 3.41, SD = 0.24). While this average appears to indicate the system demonstrates adequate usability from a global perspective, a more detailed evaluation of EQUATE items, dimensions, and free-responses is necessary to validate and elaborate on this assertion. Two of the heuristic categories exhibited an average score below 3.0, indicating the need for redesign, adjustment or enhancement: Learnability (M = 2.89, SD = 0.56) and Help (M = 2.81, SD = 0.93). This demonstrates that participants (i.e., usability evaluators) felt they needed more or better help to learn the systemFootnote 1. The remaining dimensions all demonstrated adequate or above adequate average scores (i.e., ≤ 3)(see Table 1 for descriptive statistics).

The opportunity for participants to articulate system issues in a free-response format was exercised extensively. The vast array of free responses required the research team to qualitatively (i.e., summarize for system developer feedback) and quantitatively (i.e., coding into respective heuristic dimensions) analyze the data provided. The quantitative analyses validated EQUATE survey items; Learnability and Help were the most frequently mentioned dimensions. In addition to validating survey responses, free-responses lent participants the opportunity to elaborate on the specifics of usability issues through severity ratings of those issues and suggesting ideas for fixing them. Specific issues were identified in subject free-responses that would not have been discovered with an analysis of only survey results.

For example, one field for inputting a unit’s location was not working properly (i.e., would not accept letters, numbers or any combination of the two). Not only did this prevent participants from entering in the necessary information, it prevented them from adequately completing the overall task. Each of the nine subjects identified this as a highly severe issue. The information was communicated to the development team and addressed appropriately.

Aside from identifying obvious fixes, free responses embedded in the EQUATE survey allowed participants to elaborate on issues. For example, while less than half the participants reported subpar User Efficiency in survey items, an examination of their free-responses yielded important information for system usability. They indicated that the procedure for creating a training event in PMATT-TA was not sufficiently clear. They described this issue as severe and offered suggestions for fixing it (i.e., making the New Event button on the home page more conspicuous by bolding, adding color or increasing its size). Because creating a New Event is one of the system’s main purposes, this information was vital to the system development team to enhance usability. In sum, though tedious to analyze and interpret, the opportunity for subject free-response proved essential for improving and especially fixing the PMATT-TA system.

4 Discussion

It is the goal of any armed forces to maintain a competitive advantage over the enemy by having the most advanced capabilities. Such capabilities are the result of constantly evolving hardware and software technologies. Unfortunately, developing these technologies is only one half of the equation. If they cannot be used effectively, their development has limited impact or desired effect. As a result, inexpensive, efficient, adaptable and easily analyzable systems for evaluating the usability are essential. PMATT-TA was the first system to be evaluated using a novel usability assessment method, which was developed based on an extensive review of literature and best practices. This initial analysis indicates the EQUATE provides a reasonably comprehensive, inexpensive, efficient, adaptable, and easy to analyze and interpret method for capturing end user feedback. Because of the results of this analysis, the development team benefited from the identification of critical issues prior to product development. While both the PMATT-TA and the EQUATE are still receiving research attention for validation and general improvement, preliminary analyses demonstrate the utility of each system in advancing the capability of the U.S. Navy.