Keywords

1 Introduction

In software engineering, the term “Usability” is related to the ease of use of a software product [3]. Usability is a quality attribute that measures how easy the user interface is to use. It also includes methods to improve ease of use during the software design process [1]. Nowadays on the web, usability is a necessary condition for survival. If a website is difficult to use, people will stop using it. If the page does not clearly state what a company offers and what users can do on the site, people will stop using it. If users get lost on a website, they will stop using it. If the information on a website is difficult to read or does not answer the key questions of users, they will stop using it.

The first e-commerce law is that if users cannot find the product, they cannot buy it either [1]. In this paper we will evaluate the website of an airline using heuristic evaluation and a usability test with users as a complement [5]. The results obtained in both tests will be shown to compare them and provide some conclusions.

2 Related Work

Given the importance of usability, there are two types of methods to perform a usability assessment: inspection methods and test methods [4]. The difference between them lies in the person who applies them. In the first case the inspectors perform and in the second the users participate [2].

Heuristic evaluation is a well-known inspection technique that is widely used by usability specialists. It was developed by Nielsen as an alternative to user testing [3].

The usability test with users is a method of software products evaluation in which active users are representative of said products. The main purpose of this method is the identification of usability problems during the interaction of users and the system. This method allows collecting qualitative and quantitative data, as well as user satisfaction with the software product [7].

In the present work, a usability assessment is performed with the “Heuristic Evaluation” inspection method, which is based on the analysis of the ten heuristics defined by Dr. Jakob Nielsen, Latam Airlines web application www.latam.com.

We also perform a usability assessment with the user testing method to the same application. This evaluation is done as a complement to the heuristic evaluation previously performed by the same group. In [6] the advantages of performing heuristic and user evaluations as complementary studies are developed.

The present work has been developed under an academic context, during the months of June and July of the year 2016. All the participants have developed the tests with professionalism and ethical values.

3 Research Design

In order to test usability in the website latam.com, were used two methods that complement each other: heuristic evaluations and usability test users. The objective of this test and selection of the website were academicals.

3.1 Description of the Web Site

The website corresponds to the new brand of LATAM Airlines Group: LATAM, which is a product of the consolidation of the two brands of the group that preceded it: LAN and TAM. LATAM is an airline based in the Republic of Chile, which operates in South, Central and North America, as well as the Caribbean, Europe and Oceania; with different destinations in more than 26 countries [8].

Due to the nature of the company, the website has numerous sections, among them: Purchase of tickets, Purchase of packages, Check-in Service, Charge Service, Flight Information, Offers Section, Press Room, Section for Investors, among many others. However, the main emphasis is on the first three sections because it is frequently used by passengers (Fig. 1).

Fig. 1.
figure 1

Main page of the evaluated web site: latam.com

Full size image

The purpose of the application is to offer its customers a computer tool for its main services: purchase of tickets, flight check-in and flight status display. In this sense, the website has a design oriented to emphasize these services and to link them to each other.

3.2 Study Design

The purpose of this paper was to compare heuristic evaluations and user usability tests based on a web transactional system. This work was developed in two moments. First a heuristic evaluation was carried out and then a user usability test was developed.

4 Heuristic Evaluation

4.1 Participants

The heuristic evaluation was performed using the Nielsen’s methodology analyzing the ten usability principles “heuristics”. The evaluation was performed by four evaluators, computer engineers, three of them master’s students and one doctoral student.

4.2 Phases

This section describes the steps used to perform the heuristic evaluation. These are described below.

First phase: Each participant performed alone an evaluation of the product and recorded all the results found in their respective reports.

Second phase: A moderator, who was previously selected, facilitated a meeting where the evaluators were able to unify the results obtained by briefly explaining the problems they found. As a result, a clean and unified listing of the problems encountered was obtained.

Third phase: Each evaluator independently rated the severity and frequency of each one of the problems of the unified listing. With the values of severity and frequency was calculated the criticality: criticality = severity + frequency.

Fourth phase: A coordinator, who was previously selected, performed the activities of this phase, calculating the averages and standard deviations of the three previously calculated values: severity, frequency and criticality of each problem. With the results, was established a ranking of the problems found.

The severity was evaluated according to the rating proposed by Nielsen [9], in which 0 means “I don’t agree that this is a usability problem at all” and 4 means “Usability catastrophe: imperative to fix this before product can be released”.

The frequency was evaluated according the rating of the Table 1.

Table 1. Frequency ratings
Full size table

4.3 Data Analysis and Results

A total of thirty seven usability problems were identified, which were categorized by the participants who performed the heuristic evaluation. Only for the heuristic “Recognition rather than recall” wasn’t found non-compliance. In Table 2, it can be seen the times that each unfulfilled heuristic.

Table 2. Unfulfilled heuristics
Full size table

Table 3 shows the identified problems sorted descending by severity, without considering the frequency. Of the total of problems found by the evaluators, almost 50% of them resulted in a severity value greater than or equal to 2.50, that is, they tend to be greater or catastrophic.

Table 3. Ranking of the more severe problems
Full size table

Thus, the most severe problem identified by the evaluators is that the system displays a blank screen after log off a user, which could be perceived as unsafe for the end user. Also, the evaluators have considered severe that during the ticket purchase’s process, the system doesn’t provide the option to return to the previous step, moreover, it doesn’t allow to save the information already entered so that forcing a backward the information is lost.

Other severe problems are that the system displays blank screens before certain options, other options don’t even work, displays error messages that aren’t understand and have broken links. Almost all screens don’t have the option of help for the user.

On the other hand, the Table 4 shows the identified problems sorted descending by criticality. It’s observed that the maximum value of criticality is 6, this means that the problems encountered don’t drastically affect the functionality of the system.

Table 4. Ranking of the more critically problems
Full size table

The problem of greater criticality is that the system doesn’t provide the user with the option to return on any of the pages of the ticket purchase’s process. This means that this problem is the most severe and most frequent.

Other critical issues include broken links, lack of system-wide help and poorly worded information on several pages. One problem worth highlighting is that the system has different behaviors when using different browsers.

Finally, the problems P17 and P26, although they were evaluated as very severe, were not the most critical because they didn’t have a high frequency of occurrence.

5 Usability Testing

5.1 Test Purpose

The purpose of performing the usability test in the LATAM application is that the user may encounter problems when using it by performing certain previously defined tasks. The tasks have been established based on the result of the heuristic evaluation performed in the previous stage. Problems with severity greater than 3 (P17, P26, P3, P18) that are directly related to the acquisition of passages were selected, since this is the most important functionality of the application and for them, activities that could cover these problems were defined.

5.2 Test Design

From the heuristic inspection that was performed in the previous stage, the problems were taken with severity greater than 3. With them, the problem, task and context matrix is elaborated, as shown in Table 5. Of the list of problems, those that emphasize the transactional functionalities, that is, in the purchase of air passages, on other functionalities of the web site were chosen.

Table 5. Matrix of problems, task and context
Full size table

5.3 Participants

The four participants were students of the computer science masters of the PUCP, male, whose ages ranged from 28 to 47 years.

5.4 Materials

The following materials were developed for the Usability Test:

Confidentiality Agreement:

It is a consent document where the evaluated user manifests his voluntary intention to participate in the usability test at a certain place and date.

Previous Indications:

In order to help to participant and a brief description of the stages that will be followed in the test is made.

Pre-test Questionnaire:

It is a questionnaire of demographic type and serves to obtain information to classify the evaluated user. In this way, the responses of the evaluated users can be contextualized.

Post-test Questionnaire:

It allows to obtain additional information that complements the observation made during the execution of the tasks assigned.

Task List:

It is a document that describes in detail the activities that the user will perform for each task defined. In this list, the user must detail, when required, information that is requested as a backup of what was done in the application.

For this test, the tasks have been built on the following scenarios: In the first task the user need to look for alternative (double) tickets to Miami. The dates of the holidays are from August 1 to 14, 2016 and there is a budget of S/4,700 destined for the tickets. Since it is high season, the user must utilize some promotion with which you can purchase tickets with the allocated budget (in that sense, you must use economic rates, etc.). In the second task, the user must change the flight schedule: now you want to spend three days in Miami and the rest of the time in New York. For this, the option “Multiple destinations” must be used.

Task Compliance Observation Sheet:

It is used by the evaluator to detail the fulfillment of each activity of the task, the time spent and the pertinent observations.

5.5 Usability Testing Process

The test was performed individually, each user had at his disposal an evaluator who accompanied him in the process.

Each participant was presented the Confidentiality Agreement and a list of previous Indications. Each participant gave their consent and signed the indicated documents. Subsequently, each participant was given the pretest, which was filled immediately. Then, each participant was given the task list and some general inquiries were acquitted.

The recording of the interaction was started, and the user was left in front of the browser, and each evaluator took note of the Observations of Compliance of the Tasks on what he was observing.

Finally, each participant was given the post-test questionnaire, which ended the execution of the test.

5.6 Data Analysis and Results

Task 1 Results

Hits presented:

  • Users were able to select the options indicated.

  • Those who had previous knowledge of the application, were already more familiar, so they performed tasks faster compared to less experienced users.

  • Users can obtain the prices of the cheapest tickets.

  • Even when users experience complications, they did not have to seek help from the system.

Inconveniences presented:

  • The functionality to see the different exchange rates cannot be used correctly.

  • The system is not easy to use for people who are not experienced in this type of applications.

  • The system cannot complete the payment through the credit card option when the steps indicated in task 1 are performed.

Task 2 Results

Hits presented:

  • Users were able to select the options indicated, those who did not have much knowledge of the application were already more familiar.

Inconveniences presented:

  • Since the users had gone through another previous task, where the functionality was different, confused the new interface for the selection of flights.

  • Most of the error messages presented clearly indicate why they were presented, but users before proceeding with the process do not identify which information they lacked to complete.

  • In a particular case, an error occurred that wiped all data on the screen but did not tell the user the reason for it.

  • Since it is not possible to select route rates, users are not sure to continue with the purchase.

  • There is no help in the system, which is necessary for users who do not have much experience in the use of similar applications.

Data Analysis: Observations

In general, users did not use system help. In Task 1 most users had trouble getting the rate in other currencies when using Chrome. Most users had trouble getting to the VPOS payment window. In addition, in Task 2, most users had trouble selecting destinations. Finally, all had problems to select the flight with scales more economic since that information is not shown.

Data Analysis: Post-test Questionnaire

Table 6 shows that the general appreciation of users with respect to the page evaluated is positive. They emphasize the ease of navigation and the possibility of re-using the portal. The two points that received the lowest rating was the fact that they were able to complete the tasks and overall satisfaction with the portal. This could be considered complementary, since not being able to complete a task can influence the satisfaction of use. In addition, users 1 and 2 stand out for having extreme opinions regarding the page, the first with an extremely negative opinion and the second with an extremely positive opinion. What is recommended to validate these results is to increase the number of evaluated users.

Table 6. Results of Post-test questionnaire
Full size table

6 Conclusions

It emphasizes the heuristic evaluation as a tool for evaluation of usability of wide use, given its advantages in time and cost versus analysis with participation of end users. This evaluation is an expert analysis to determine if the elements of an interface comply with widely accepted principles such as Nielsen heuristics.

In the heuristic evaluation, a significant amount of usability problems have been found. It is important to mention that an expert has been used less than recommended by Nielsen. When evaluating large or multi-functional applications, it is advisable to pre-define a scope to focus the experts on what may be of interest in the evaluation. A heuristic evaluation will provide better results when focused on detecting relevant aspects for the client.

According to the above, the problems found, which concentrate on problems of lack of help, broken links and consistency errors, are detected in the heuristic evaluation, shows problems in the process of migration or changes to this LATAM platform.

With reference to the site as a whole, apart from the problems detected, in general, it is emphasized that the site meets the objectives of each functionality evaluated, problems in general are not blocking but can hinder the user’s tasks. Besides, the overall evaluation of the site is positive, since it meets most of the objectives for which it has been built, and improvement is pending on the basis of the problems detected, as well as the completion of the process of migration or change through which the indicated site is passing.

For those reasons, it concluded that two methods of usability evaluation which are heuristic evaluations and usability test users, are complement each other. When they are used together, let a better focus in user, a better diagnostic and analysis about the object of study and increase a better feedback, among other advantages.

It is suggested, for usability tests with users, to increase the number of users evaluated, with a more heterogeneous sample, in order to obtain more results that validate the conclusions.