Keywords

1 Introduction

In the past few years people have changed the way they read news, as they are moving from the traditional paper to online version of newspapers such as website or mobile apps; In 2011, surveys conducted by the Pew Project for Excellence in Journalism, showed that 34 % of respondents said they read news online within the past 24 h (as opposed to 31 % who favored physical newspapers); and a full 41 % said they get most of their news online, 10 % more than those who said they got most of their news from a physical newspaper [1, 2]. Therefore, the search of information in these sites becomes very important due to the amount of news that is posted everyday.

Good usability of the websites plays an important role because it can define if the user is going to develop fidelity to the site and use it often or, with new users, it determines whether they will ever use it again after a first try [3]. As with many other interactive products, with newspaper websites the evaluation of the user experience becomes necessary to find the characteristics that users seek.

Previous usability studies have been conducted to evaluate newspapers’ sites from around the world. Among them, in the UK [4], “The Guardian” was evaluated as the best among other major newspapers, and the same was for “La Crónica de Hoy” in a Mexican study conducted by a local usability consultancy [5].

To get in context, in Fig. 1 screenshots were taken from both websites. As seen, the general layouts are very similar and have similar information and options.

Fig. 1.
figure 1

“The Guardian” in the left and “La Crónica de Hoy” at the right

Full size image

In this work, first, a general evaluation comparing these both sites (“The Guardian” and “La Crónica de Hoy”) is reported to know and detect the usability differences and similarities between them. Then, an analysis was conducted to contrast the results of performing text vs. visual searches.

2 Method

This study follows a typical usability test approach where users were asked to do tasks for which standard metrics such as task success and duration were collected; after completing all of the tasks, users were asked some questions to evaluate qualitative aspects of the user experience. For the present study, the tasks were the following:

  • Task 1 - Search for an image of the tennis player Novak Djokovic, winner of the 2010 Wimbledon tennis tournament.

  • Task 2 - Search for information about the Strauss-Kahn’s case.

    • In this task, the participant was asked to write the date of the article.

  • Task 3 - Search for information in the “Business” section.

    • In this task, the participant was asked to write the title of the main article in the section.

In this study, 47 participants were randomly assigned to evaluate one of the previously mentioned websites. All the participants were bilingual with great experience reading and writing in both, English and Spanish. We did not detect any challenge related to interacting with an English-based web site despite the fact that all participants’ mother tongue was Spanish.

Participants were asked to do two different types of search tasks, which are usually done while visiting this kind of sites. First, to perform a search of textual information and, second, a search of visual content (a photograph). The metrics of the evaluation were: task success and task time. The participants also made an overall evaluation of the site, answering two questions (“Ease of finding information” and “Visual appealing”), with a 7-point Likert scale [6], and an open-ended question to measure qualitative aspects. Finally, the overall satisfaction was measured with a SUS questionnaire.

The time elapsed from the presentation of the task to the click of the “Next” button that showed the selection of the answer was recorded automatically.

The metrics captured for each task were: on one hand, self-reported task success that is whether the participant reported if they had definitely found the answer, or not, and, on the other hand, task time.

In the post-study self-reported data, the captured metrics were: ease of finding information and visual appeal (with the 7-point scale), challenging or frustrating aspects (open-ended questions), effective or intuitive aspects (open-ended questions) and SUS scores.

3 Results

The results shown in this section are derived from using the geometrical means, and tests such as Chi Square test, t-test and the Mann-Whitney U test, which were used to calculate the statistical significance depending on the type of data, with an alpha of 0.05. In general, the differences between both sites were not significant; both sites were rated with similar scores. However, we found that, although both of them are well designed and have an overall good user experience, there are some differences that make one of the sites score better than the other. The rest of this section will describe some of these differences.

3.1 Comparison Between the Websites

Task Success. After completing every task, the participants were asked whether or not they believed they were able to finish the task successfully. This was a self-reported assessment. If the participant was sure that the task was done it was marked as a success. If the user was not sure the task did not count, and it was marked unsuccessful. “The Guardian” users reported 76.12 % of success vs. an overall 81.69 % of success reported by “La Crónica” users. These results are shown in Fig. 2.

Fig. 2.
figure 2

Task Success

Full size image

As seen in Fig. 2, the difference between both results is minimal. With the purpose of evaluating if the difference was statistically significant, the Chi-squared test was performed, as the data was dichotomous, i.e., accomplish or not the task. The result was not statistically significant between both sites, c2(1, N = 140) = 0.83, p > 0.05.

Task Time. The second aspect evaluated was the time the participants had to take to finish a task. The geometrical mean of the “The Guardian” users was 55.7 s, while for “La Crónica” was 64.75 s. Despite that in Fig. 3 an apparent difference is noticeable it was not statistically significant, t(136) = -1.35, p > 0.05.

Fig. 3.
figure 3

Task Time

Full size image

Satisfaction Assessment. Following the execution of all tasks, the participants were asked to express comments regarding any challenge or frustrating experience they had while using the site.

On one hand, for the site of “The Guardian” (GU), 13 out of 23 participants (56.5 %) commented that they had negative experiences; for the site of “La Crónica” (CR), 11 out of 24 participants (45.8 %) indicated finding this kind of aspects.

The negative comments about the GU site given by at least two participants were:

  • Not being able to search for images (4)

  • Too much information, bad distribution of the information (9)

The negative comments about the CR site given by at least two participants were:

  • Color and content of the menu (2)

  • Problems with the search of images (3)

  • Presentation of the information, too much content (3)

On the other hand, for “The Guardian” site, 47.8 % of the participants (11 out of 23) indicated easy or intuitive aspects characterizing a good user experience; for “La Crónica” site, 62.5 % (15/24) indicated effective or intuitive aspects.

The effective or intuitive aspects about the GU site given by at least two participants were:

  • Good, intuitive and helpful menu (4)

  • Good headings that helps finding information (3)

  • The search tool is easy to use (2)

  • Overall good design (2)

The effective or intuitive aspects about the CR site given by at least two participants were:

  • Helpful and efficient search tool with good presentation of the results (9)

  • Overall good design (6)

Overall Assessment. Among the collected comments concerning negative comments, on one hand, “The Guardian” had 13 negative ones, which are all related to the overload and bad distribution of the information, and the difficulty of finding an image. On the other hand, “La Crónica” had only six comments related to these aspects and two more negative ones about the color and content of the menus. As to positive comments, the same amount of users (nine) said that both sites have good and easy to use search tools. Nevertheless, the main difference in the comments is present in the ones related to the overall design, in which, nine came from users of “La Crónica” vs. two from “The Guardian” users. In this work, it is argued that the main reason of this difference may lie in the amount of negative comments that the “The Guardian” received.

Given that the qualitative assessment favors “La Crónica”, it is worth to contrast this with the results of applying the SUS test. Following the open questions on the application, a SUS was asked to be completed and, as can be seen in Fig. 4, ten users rated with more than 60 points to “The Guardian” vs. 15 users to “La Crónica”, making the last one 31.9 % easier to use as assessed by the participants. However, a significant difference was not found, t(45) = -1.14, p > 0.05.

Fig. 4.
figure 4

SUS Scores

Full size image

Despite of this lack of significant difference, this leaded to the question of why there is a trend towards lower scores that the users give to “The Guardian” even when both are good sites, and from our results it was perceived that it seems that the main difference is between the visual and written search of content.

Consequently, a specific analysis to explore the differences between visual vs. written search was conducted. The aim was to find if the differences mentioned in the comments were visible in other tests and how they affected the general scores.

3.2 Comparison Between Visual and Written Search

Task Success. Figure 5 shows the task success by type of task. With a geometrical mean of 65.72 % for the visual task and 84.8 % for the written tasks, a statistical difference was found between both type of searches, c2(1, N = 140) = 4.62, p < 0.05. This difference is mainly due to the lower rate of success of “The Guardian” which was significant too, c2(1, N = 69) = 6.59, p < 0.01. These results show the first revealing finding, which leads to identify that the design of “The Guardian” makes it more difficult for the user to do the visual task.

Fig. 5.
figure 5

Task Success by type of search

Full size image

Task Time. The same pattern is observed in the task time results as shown in Fig. 6. There was a significant difference between the two types of search, U = 1,595, p < 0.05. On one hand, the times from “La Crónica” users does not vary, but “The Guardian” users needed more time to complete the visual search than the written ones, which is a significant difference, t(66) = 3.28, p < 0.05. On the other hand, the difference between newspapers in the written tasks was significant, U = 720.5, p < 0.01.

Fig. 6.
figure 6

Task Time by type of search

Full size image

As can be seen in the results of task success and task time, “The Guardian” shows results with significant differences between both types of tasks. This leads to the conclusion that this newspapers’ website is good for written searches, but it is not as good in the visual content searches. This is observed in the lower rate of success in these tasks and the longer time required to complete the tasks compared to “La Crónica” users.

3.3 Post-Study Self-Reported Data

After finishing their three tasks, the participants were asked to rate the site they used on two 7-point scales (higher numbers better): Ease of finding information and Visual appeal.

Ease of Finding Information. The difference between both websites was not significant, t(44) = -0.69, p > 0.05. The geometric mean was 5.07 for the CR website and 4.58 for the GU site, as shown in Fig. 7.

Fig. 7.
figure 7

Ease of Finding Information (Higher = Easier)

Full size image

Visual Appealing. As with the previous result, the difference between both websites was not significant, t(44) = -0.66, p > 0.05. The geometric mean was 4.55 for the CR website and 4.22 for the GU site, which can be seen in Fig. 8.

Fig. 8.
figure 8

Visual Appealing (Higher is better)

Full size image

The results shown in this section, although they are not significant, showed that in general, “La Crónica” was rated with better scores than “The Guardian”. These differences are important to understand the reason of the different results found in the SUS. This also shows the importance of how the information is presented to the users, in the main pages as well as the search tool results. A significant impact is seen in the general usability scores mainly because of a poor performance of this particular part of the interface.

4 Conclusions

The differences between the two search functions shows that, despite the fact that both sites were rated with good usability scores, slightly favoring “La Cronica”, the design of the interface and the search tools were a significant factor to evaluate the usability of a newspapers’ website. The finding is supported by the users’ comments, which shows that the overload of information, the ease of use of the menus and the search tool were important aspects to define a website with better usability. As seen in the metrics, the difficulty to do the visual searches can lower the overall usability perception of the user, as seen in the SUS score, and in the perception of how easy is to find information and the general visual appealing.

This generates important information for future development of this kind of websites, finding new ways to make different kind of searches more intuitively and to pay attention in how they present visual content. This is because the users tend to evaluate the visual content first, and they will find difficult to understand the content or to find the needed information if they do not understand it. The information found in these results can lead to new studies to derive a deeper understanding of how to improve the usability of the search tools in the newspapers’ websites.