Keywords

1 Introduction

There are different terms to talk about current society. The term Information Society emerged after the Industrial Society, where technological development was boosted to manage the information. Other authors use the Knowledge Society term as an alternative by some in academic circles to the Information Society [1]. According to UNESCO, the two concepts are complementary; Information Society is the building block for knowledge societies [2]. There is a third term used nowadays, Digital Society, in order to put relevance on how digital technologies have an impact of digitalization on today’s society, culture and politics.

The fast development of digital technologies during the last decades has influenced in all life aspects of the world population. The communication paradigm has undergone a major change; people can interact without temporal or spatial dependence; information and knowledge are available for most of the people at any time in any place. The way to do the shopping, watching TV, listening to music or finding a couple has changed. There are new jobs created by technological advances, such as data engineer, business intelligence analyst or security management specialist; and other jobs have evolved, such as delivery services, bankers or taxi drivers.

Although specific forms of technology uptake are highly diverse, a generation is growing up in an era where digital media are part of the taken-for-granted social and cultural fabric of learning, play, and social communication [3]. The young have emerged as a distinct social group, and with it, an understanding that they have a role in social change, as drivers of new behaviors and understandings [4], but half of them feel that their concerns are not taken into account.

In this context, WYRED project (https://wyredproject.eu) aims to provide a framework for research in which children and young people can express and explore their perspectives and interests concerning digital society, but also a Platform from which they can communicate their perspectives to other stakeholders effectively through innovative engagement processes [5]. WYRED [6] is a European project funded by Horizon2020 programme for three years, from November 2016 to October 2019, in which are involved institutions from Spain, Italy, United Kingdom, Turkey, Ireland, Belgium, Austria, and Israel.

From a technological point of view, WYRED provides a technological ecosystem to support the research framework and boost international conversations among young people from countries directly involved in the project. The ecosystem is composed of a set of Open Source tools and people involved in the project – partners, stakeholders and young people between 7 and 30 years –. The main component of the ecosystem in which the interaction among young people takes place is the WYRED Platform, a private and secure space in which young people can interact inside communities focused on different aspects of Digital Society [7]. Each community has a person or persons in charge of its management, which are the so-called facilitators, and also a set of members, mainly young people, interacting through discussion threads and creating research projects.

It is important than the ecosystem and, in particular, the Platform, being accepted by its final users (young people between 7 and 30 years, facilitators and stakeholders). Given this need, a preliminary study was performed to analyze the usability of the Platform from the point of view of young people [8, 9]. This study applied the System Usability Score [10] after a pilot experience in order to obtain insights about usability of the WYRED Platform. Some technical problems and usability issues detected by the participants during the study were solved before carried out the present work.

Although the main users are young people, facilitators have a crucial role in the ecosystem. They are responsible to engage young people in the international conversations about Digital Society inside the WYRED Platform. Also, they serve as a guide in the conversations taking place within communities, as well as in the research projects developed by the young people about different topics related to the Digital Society.

This work aims to describe the usability study targeting facilitators to reach insights about how these users value the WYRED Platform usability. This usability study was performed through a combination of two techniques. First, a heuristic analysis following the heuristics proposed by Nielsen [11] was executed with four experts with different profiles. This analysis was complemented with the results of the Computer System Usability Questionnaire (CSUQ) (version 3) [12] to collect the experience of real users. The combination of both techniques provides a comprehensive status of the WYRED Platform’s usability, to fix any issue and improve its features and engagement.

The work is set out as follow. The second section introduces the WYRED Platform. The third section describes the methodology used to study system’s usability. The fourth and fifth sections describes the heuristic evaluation and CSUQ results. The sixth section presents the discussion and the last section concludes the work with its more significant contributions.

2 The WYRED Platform

The Platform is organized in multicultural and interdisciplinary communities where young people can share their ideas and opinions about Digital Society and develop research projects with the support of facilitators from different European institutions and associations. The communities have different tools: forums to support conversations and coordinate research projects; a calendar to share dates and organize events or activities; and a form to give visibility to the research projects. Figure 1 shows a community example within WYRED Platform.

Fig. 1.
figure 1

Community to talk about influencers across Europe

Full size image

One of its main innovations is the strong commitment to user privacy and security; it is designed as a safe space in which children and young people can express themselves freely. Users need an invitation to register within the Platform and children under 14 years old need parental consent in order to have access after finishing the registration. Besides, privacy policies are established to ensure the anonymity of young people, while allow collecting demographic information.

The users with a facilitator role inside a community are their managers, they can manage user’s roles and subscriptions, moderate the conversations, publish research projects created by community members, and invite new members to the community both registered and non-registered users. Moreover, there are users that have the facilitator role in the whole Platform; these users can create new communities and invite new users to register in the Platform.

3 Methodology

3.1 Participants

The heuristic evaluation was carried out by four experts, two women and two men between 25 to 39 years old. Two experts had used the WYRED Platform as facilitators before analyzing it; the other two had access for two weeks before complete the analysis. The experts were selected according to their profiles:

  • A Ph.D. student whose doctoral dissertation deals with customizable dashboards to analyze and visualize any kind of data.

  • A web developer and researcher with ten years of experience whose main research line is the development of technological ecosystems for knowledge and learning processes management in heterogeneous contexts.

  • A researcher with more than ten years of experience in multimodal human-computer interaction.

  • A developer and researcher with more than ten years of experience focused on data visualization in different fields such as digital humanities or sports; and also with experience in teaching human-computer interaction in a Computer Science degree.

Regarding the second part of the usability study, a subset of facilitators was invited to answer the CSUQ. There are two profiles of facilitators in the Platform, young people that are involved in some communities as facilitators in order to coordinate, moderate and organize research projects; and members of the consortium of the project that is in charge to involve young people, create communities and manage them. Although there are more than 40 facilitators of both profiles, only those who used the Platform in the last six months were invited to answer the questionnaire. Finally, a total of 30 facilitators were invited to participate in the study and 28 answered the CSUQ. Most of the participants are female (64.3%), 21.4% are male, 3.6% selected another gender, and 10.7% decided not to provide this information. Regarding their profiles, 64.29% are young people (under 30 years old according to the European Union), and 35.71% are over 30 years old.

The WYRED Platform is an online tool so that it can be accessed from different environments. Most of the participants have used several browsers to access the Platform (Fig. 2), taken into account this; Chrome is the most used browser (89.29%). Regarding the device, 10.71% only through a smartphone, 39.29% only access through a computer, 42.86% used both (smartphone and computer), 3.57% access through tablet and computer, an finally 3.57% used all.

Fig. 2.
figure 2

Browsers used by participants

Full size image

3.2 Instrumentation

The same template was provided to the experts in order to get the reports to perform the heuristic evaluation. The template is composed of two fields to collect the name of the evaluator, the name of the tool evaluated and a table with one row per each of the heuristic rules proposed by Nielsen [11] and three columns – heuristic rule, points from 1 to 10, and problems detected –.

The chosen tool to measure the usability of the WYRED Platform was the Computer System Usability Questionnaire (CSUQ). This questionnaire is an adaptation of the Post-Study System Usability Questionnaire (PSSUQ), except that the wording of the items does not refer to a usability testing situation [13]. There is three versions of the PSSUQ, the first version had 18 items, the second version was 19 and in the same version, Lewis found that three items did not contribute to reliability of the scale so the version 3 has only 16 items [14].

The CSUQ version 3 contents 16 items rated on a 1 to 7 Likert scale (from “strongly disagree” to “strongly agree”, respectively) and a not applicable (N/A) option. The PSSUQ and CSUQ were originally designed to assess the perceived user satisfaction with IT systems; for this study, the term “system” or “computer system” was replaced by “application”.

In addition to the 16 items of the CSUQ, a set of variables were also collected by the instrument. First, an open field was provided at the end of the CSUQ to let the users remark any relevant experience during the use of the Platform. Second, a set of questions related to the user environment where included: the language used in the Platform, the role of the user, the devices used to access – computer, smartphone, tablet –, the operating systems – Android, Chrome OS, iOS, Linux, MacOS, Windows –, and the browsers (Chrome, Edge, Firefox, IE, Safari). Finally, a set of demographic variables: year of birth, gender, country, and any eye diseases of the user that could affect the experience.

The CSUQ was implemented using a customized version of LimeSurvey (https://www.limesurvey.org), an Open Source online statistical survey web application. The instrument was applied in English, but it is also available in Spanish [15].

3.3 Study Design and Data Collection

The heuristic evaluation was carried out by four experts. All experts had access as facilitators to the WYRED Platform in order to analyze all screens. Before starting the evaluation, a brief description of the project and the aim of the WYRED Platform was provided to the experts.

Each expert navigated through the application several times observing all the screens and detecting the usability problems. Each expert has assigned a value of 1 (serious problems) to 10 (no problems) to each heuristic and a brief description of the problems associated in order to justify this value. The heuristic rules used were those proposed by Nielsen [11]: (1) visibility of system status; (2) match between system and the real world; (3) user control and freedom; (4) consistency and standards; (5) error prevention; (6) recognition rather than recall; (7) flexibility and efficiency of use; (8) aesthetic and minimalist design; (9) helping users to recognize, diagnose, and recover from errors; (10) help and documentation. Data was collected through shared documents in Google Drive.

Regarding CSUQ, the participants involved in the study has experience using the Platform in real scenarios; it was not conducted scenario-based usability tests in a laboratory environment. To collect the data, a description of the study and the link to the questionnaire was sent by email to all the participants. Two reminders were sent before the deadline ended (one month after the first message).

4 Heuristic Evaluation

To show the results of the heuristic evaluation, each expert was identified by a number (E1, E2, E3, E4). Table 1 summarizes the values for each heuristic rule, where 1 indicates that the expert detected a huge amount of serious problems and 10 no problems were found. In order to get a final value for each heuristic, the average of each heuristic rule was calculated (Fig. 3).

Table 1. Assigned values to each heuristic by each expert
Full size table
Fig. 3.
figure 3

Final value for each heuristic rule

Full size image

Experts detected problems associated with all heuristic rules. The heuristic that presents the largest number of usability problems was HR4 (Consistency and standards) with 5.75 points. The lowest values assigned by E2, E3, and E4 are related to this heuristic; 19 different medium and serious problems were detected. It should be pointed out one of the main problems identified by E4: “there are two types of roles with a different set of permission using the same word: facilitator, one that is only moderator/community manager inside a community, and one that can create communities.” Also, E2 has detected a serious problem associated to accessibility standards, there are several problems related to color contrast in links (yellow over light grey) and menu (white over blue) according to Web Accessibility Guidelines, the standard proposed by W3C (https://www.w3.org/WAI/standards-guidelines/wcag/).

On the other hand, highlight the low values associated to help users, HR9 (Help users recognize, diagnose, and recover from errors) and HR10 (Help and documentation). The WYRED Platform has a help section with videos in English, but the application is multilingual, so the contents should be provided with subtitles. Moreover, the display of the content is not clear, the screen should be redesigned according to E1, and E3 indicates that the information is not accessible on demand in text format and there is not a FAQ to make the doubt resolution more straightforward. Also, there is a community for technical support but E2 and E4 comment that it is difficult to find and use it.

The lowest number of usability problems was detected in HR2 (Match between system and the real world) with an average of 8.75 points and 6 minor usability problems, one of them related to the concept facilitator, which needs more explanation according to E2 and E4. Regarding the other heuristics, most of the issues detected are minor and medium usability problems, but it is important to highlight several problems associated with search tools in community and projects sections. These problems were detected by E1, E2, and E3, although most of them have low priority to fix them, their combination generates one of the most significant problems. Some of them are:

  • “When searching for public projects or communities, there is no straightforward manner of knowing which filter is currently applied” by E3. Associated to HR1.

  • “The layout of the search tools is difficult to understand. The search button appears before the search field” by E2. Associated to HR4.

  • “When searching for public communities, there are no autocompletion options” by E3. Associated to HR5.

  • “It was difficult to find out how to add several tags to the different search bars shown in the Platform. The user has to manually input a comma after the first keyword to be able to insert the second one” by E1. Associated to HR7.

The number of problems identified by each expert is small but the combination of all of them provides an input to improve the WYRED Platform.

5 User Experience

5.1 CSUQ Questionnaire Results

A total of 28 persons answered the CSUQ questionnaire (version 3), which is an enough sample number for the purpose of this evaluation. The mean and a 95% confidence interval were computed for each CSUQ item answer, as well as for the overall and subscales’ scores. The results were compared to the PSSUQ norms (Fig. 4) as previous studies indicated that the CSUQ and PSSUQ scales are comparable [14]. To properly compare the results, the PSSUQ norms were reversed, as their original scale indicates that lower values denote higher satisfaction (i.e., 1 means “strongly agree” and 7, “strongly disagree”). The following results were obtained regarding the system quality, information quality, interface quality and overall score:

Fig. 4.
figure 4

Overall results regarding the CSUQ results compared to the PSSUQ norms identified in [16]. The overlapped lines represent the 95% interval.

Full size image

  • System quality: 4.87

  • Information quality: 5.02

  • Interface quality: 4.82

  • Overall score: 4.92

To gain better understanding of these overall scores, each item mean value was also individually analyzed (Fig. 5). This analysis gives hints about the potential causes of the questionnaire overall results. The detailed results are listed in Table 2.

Fig. 5.
figure 5

Individual results of each CSUQ items compared to the PSSUQ norms. The overlapped lines represent the 95% interval.

Full size image
Table 2. Summary of the CSUQ scores for the WYRED Platform (n = 28).
Full size table

The distribution of the given scores can be seen at Fig. 6. The figure shows the distribution of values of the CSUQ Likert scale. Through this representation it is possible to understand how the participants rated each item with less granularity.

Fig. 6.
figure 6

Distribution of individual Likert values regarding each CSUQ item.

Full size image

Finally, as previously mentioned, a set of demographic and technical variables were collected in addition to the CSUQ items. With this data, the main goal is to check whether the employed device or software to access the Platform, the language or other demographic variables were correlated to the finally given CSUQ score.

The Fig. 7 shows the correlation among these variables with the different CSUQ items; yellow cells represent no correlation and more blue cells represent higher correlation. As presented in this figure, there is no relevant correlation among the demographic and technical variables and the CSUQ items, except for the birthdate and the use of the iOS operating system. Younger people and people using the Platform through iOS seem to rate better the Platform. However, this correlation is not very strong (around the 0.3–0.5 interval). Gender has no correlation with the CSUQ items, as previously demonstrated in [14], as well as the language, device and browser employed during the experience with the WYRED Platform.

Fig. 7.
figure 7

Heat map showing the correlation among different variables. (Color figure online)

Full size image

These analyses can be consulted in detain at https://github.com/AndVazquez/wyred-csuq-analysis [17].

6 Discussion

The heuristic evaluation provides a set of potential usability issues. The problems detected can affect the WYRED Platform usability, but sometimes these problems are different from those found by final users of the system. There are some studies focused on what problems detected by usability experts are experienced by users in their interaction with the system. According to Khajouei, Ameri and Jahani [18], in some areas, the perception of evaluators in using this method is not consistent with the users’ experience with a system.

The results of the heuristic evaluation are useful to improve the WYRED Platform, but the results from the CSUQ questionnaire are required to get a full overview of the usability of the Platform. The usability issues detected by each expert are a small number of the existed usability problems, for this reason, several experts with different profiles were involved in the study.

Some of the identified problems were also detected by the users, not through the CSUQ questionnaire, but through informal comments provided during the activities that take place inside the WYRED Platform. In this sense, the priority to fix the problems will be defined according to final users, although the experts’ opinion will be taken into account.

Regarding the user experience, one of the identified potential issues of the CSUQ questionnaire is the positive nature of the items’ statements. The tone of every item elicits agreement (in contrast with the SUS questionnaire, in which positive items are intertwined with negative items [10, 19]), making the answering process more easy, but also being vulnerable to response bias, especially, acquiescence bias [20]. However, there were no evidence of response styles in the PSSUQ data [21], which is almost identical to the CSUQ items, only modifying the wording of the items given its non-lab nature [13].

The obtained results, compared to the PSSUQ norms, show acceptable values, although lower than the references. However, as indicated in [16], these norms are not specially suitable for direct assessment, given the fact that the data came from “a variety of sources that included different types of products at different stages of development and the performance of different types of tasks using systems that were available from the mid-1990s through the early 2000s”. Nevertheless, the PSSUQ norms provide a useful reference to compare the obtained scores, and to reach insights about potential weaknesses of the tested Platform.

On the other hand, as presented in Fig. 3, taking as a neutral reference the 4 value, the CSUQ questionnaire responses are mostly positive.

The information quality items are the best-rated items, being even slightly higher than the norms above [16]. These results are directly related to the low number of usability issues associated with the appropriate feedback provided to the users (HR1), the use of real-world conventions such as communities or conversations (HR2), the clear support to undo and redo actions inside the Platform (HR3) and the instructions provided to use the tools (HR6).

The weakest values of the CSUQ questionnaire are related to items 14 (“I like using the interface of this application”) and 15 (“This application has all the functions and capabilities I expect it to have”). The potential cause of these lower ratings is the fact that the Platform is still in development, explaining the lack of capabilities and interface maturity. Moreover, usability experts identified several problems associated with these items, in particular, the most significant number of usability issues are associated with the heuristic rule focused on consistency and standards (HR4). Also, during the heuristic evaluation, experts detected several problems related to the layout and the interface elements related to communities. Most of the activity inside the Platform takes place in the communities, so the problems associated with HR8 (Aesthetic and minimalist design) could be a potential cause of these lower ratings in the CSUQ.

No strong correlations were found regarding the employed devices or technologies to test the Platform, as well as regarding other demographic values. Only the birth date variable showed that younger people tend to rate better the Platform.

7 Conclusions

A usability evaluation of the WYRED Platform has been executed through two different methods. First, a heuristic evaluation was performed with four experts involved. To complete this information, an evaluation of the Platform using the CSUQ questionnaire was also performed to gain insights about the users’ perceived usability of the system.

Having these two points of view of the Platform’s usability helped to identify weaknesses of the WYRED interface and interaction methods by comparing them against a set of accepted usability principles (in this case, the Nielsen’s Heuristics were chosen), and to identify usability-related issues from the users’ perspective by asking a sample to complete the CSUQ questionnaire.

The experts involved in the heuristic evaluation were selected according to their problems in order to get different perspective although all of them has experience with usability issues. Each expert identified a small number of usability issues, but the combination of the results provides relevant information to develop a new version of the WYRED Platform in order to solve the different identified problems and provide a stable final version. It would be interesting to assess this new version through a user testing and a second heuristic evaluation.