Keywords

1 Introduction

Nowadays it is inevitable for any organization not having a website since websites are used as an information dissemination medium to the public. Universities also uses websites to provide information for their users and to promote themselves. A university website communicates information regarding their academic programs, teaching facilities, student affairs, research opportunities, campus facilities etc. [1] to its various kinds of users such as current and prospective students, faculty, staff, alumni, parents, researchers, etc. [2, 3]. Each of these user groups has their own requirements and expectations from the website but they all want to access the accurate information they required easily in a short time. Hence, usability becomes an essential issue for university websites as well [1, 4].

Usability in general is considered as one of the most important quality attributes for websites [5, 6]. In addition, Nielsen [7] states that usability is a prerequisite for websites. As in other domains, usability of university websites has been studied by researchers previously [8,9,10,11,12]. These studies generally reported usability problems of university web sites and some provided feedback for design. They have applied various usability evaluation methods. These studies individually are valuable for university website developers or researchers studying on this topic. However, in this study, it was aimed to gather all the usability of a university website related studies published for a decade and consolidate issues reported in those studies and reveal the trends by using a systematic review approach [13, 14]. Thus, any developer or a researcher who is interested in usability evaluation of a university website can benefit from previous researchers’ experiences.

Systematic review studies are based on an approach called evidence based software engineering, which is a method to gather best practices and results of the research studies [13, 14]. This approach enables evaluating the work done in a specific area, comparing the results and revealing the trends. Previously some literature reviews or survey studies on website usability evaluation methods regarding various domains have been conducted [6, 15,16,17]. Insfran and Fernandez [6] focused on the difficulty of developing more useful web applications and they first conducted a systematic review study on 51 papers regarding usability evaluation methods used for websites for the period of 1998 to 2008. Their results showed that the most commonly used method was user test and inspections next. They revealed that evaluations were generally conducted at the implementation phase rather than design or requirements phases. In 2011, Fernandez et al. [15] re-conducted another systematic review, covering period of 1996 to 2009. They investigated 206 studies and revealed similar results to the previous study. They emphasized in this review that studies generally applied existing usability evaluation methods and frequently they applied user tests as an evaluation method again. Nawaz and Clemmensen [16] conducted another systematic review study by just focusing on 60 papers published in Asia region between 2001 and 2011. In a more recent study, Ugras et al. [17] conducted a review covering 2005 to 2014 period on 199 papers. This study reported that the most commonly used usability problem was navigation, the majority of the studies used user-based usability evaluation methods, and the most commonly used user-based methods were survey and usability tests.

Since above mentioned studies were all related about general web site usability, there is not any systematic review study related specifically with usability of university websites. Therefore, the research question of this study is to reveal the general trends in the usability evaluation of university website research by determining methods used in the usability study, evaluation phase in which the study conducted, type of evaluation, profile distribution and level of participants, data analysis methods and frequently addressed usability issues.

2 Method

Systematic review studies are known as a method called software-based software engineering, which collects the best practices and results of research studies [13]. This approach makes it possible to assess the work done on a particular area, compare the results again and confirm trends. In other words, systematic reviews allow researchers to compare the result of studies on a particular study. Through systematic reviews, it is possible to see what the trend is, what the studies are generally focused on, and what they are missing.

In this study, a systematic review method based on Kitchenham’s theoretical framework [13] was used. The study was conducted in three-phases, which were planning, investigation and reporting. In the first phase, as a first step of planning, the research questions were determined as follows;

  1. 1.

    What is the distribution of the number of publications by years?

  2. 2.

    At which stage is the usability evaluation carried out in the studies?

  3. 3.

    What are the most commonly used methods in the studies?

  4. 4.

    What is the type of evaluation methods used in the studies?

  5. 5.

    What are the most commonly used data analysis methods used in the studies?

  6. 6.

    What is the user profile distribution in the studies?

  7. 7.

    What are the most commonly addressed usability issues in the studies?

  8. 8.

    What is the frequency of mentioning the issue “accessibility” in the studies?

  9. 9.

    What are the research institutions and countries of the studies?

A systematic review protocol was developed in relation to the research questions to classify the published articles. The protocol was formed based on the previous systematic literature review studies on general web site usability. Review protocol was formed of seven sub-sections including article general record, methods used in the study [6, 15, 17], the stage of evaluation [15], user profile [17], data analysis methods, frequently addressed usability issues [17], mentioning the issue of “accessibility”.

Afterwards, university website evaluation related articles published between years 2006 and 2016 were searched through the online library of Kocaeli University in 2016. The keyword string used in search was [(“usability” OR “usability evaluation”) AND (“educational websites” OR “higher education websites” OR “academic websites” OR “university websites”)]. Conference or journal publications accessed through this search criterion were downloaded. It was also noted that the articles that were written in English were considered in the scope.

At the end of the systematic search process, 54 articles were determined but 53 of them could be downloaded as full-paper and examined in the investigation phase of the study. Articles were reviewed by two authors independently first and then two came together and compared their results to reach a consensus. Finally, in the reporting phase, research trends were revealed.

3 Results

Detailed findings of this systematic review based on the research questions determined are presented in the following subsections.

3.1 What Is the Distribution of the Number of Publications by Years in University Website Usability Evaluation Studies?

The total number of investigated articles can be seen in Fig. 1. The year with the highest number of publications was 2013 with 10 articles. In 2014 and 2016, there were 8 studies conducted. The lowest year was 2007 with one article. However, the number of studies published in years was very close to each other (Avg = 5, SD = 3). In recent years, there has been an increase in the number of studies carried out on this subject according to the first years.

Fig. 1.
figure 1

Distribution of usability evaluation of university websites publications over years

Full size image

In the scope of the study, only conference and journal articles were investigated. The distribution among these two groups can be seen in Fig. 2. 68% of the investigated articles were journal articles while 32% of them were gathered from conference proceedings.

Fig. 2.
figure 2

Types of usability evaluation of university websites publications

Full size image

3.2 At Which Stage Is the Usability Evaluation Carried Out in University Website Usability Evaluation Studies?

Many of the usability evaluation of university website studies conducted can be considered as summative evaluation since 84% of them were conducted while the university websites were on use while very few of them were conducted during the deployment phase (2%). On the other hand, few studies conducted were considered as formative evaluation. Among the formative evaluation studies, 8% were conducted during the requirements phase while only 2% of them were conducted during the design phase of the website development. These can be seen in Fig. 3 in detail.

Fig. 3.
figure 3

Stage at which usability evaluation conducted

Full size image

3.3 What Are the Most Commonly Used Methods in University Website Usability Evaluation Studies?

Usability evaluation methods applied in university website studies were investigated. The methods were categorized in four main groups which were user-based, expert-based, data-driven and pure literature reviews [17]. Many of the studies were conducted as user-based or expert based while few of them were data-driven which depended on automated tools such as web analytics and log analysis. The details of this distribution can be seen in Fig. 4.

Fig. 4.
figure 4

Methods used in usability evaluation of university websites publications

Full size image

Tool and techniques that were used in user-based, expert-based and data driven methods were investigated in more detail. The distribution among the tools and techniques for each method can be seen in Fig. 5a–b. Questionnaire was frequently used user-based method since more than half of the studies applied this. Usability testing follow this method with 17%. The other methods such as think-aloud, eye tracking, interview, card-sorting, remote usability testing or observations were applied in few studies. In expert-based methods, 52% of the articles followed heuristic evaluation based on Nielsen’s heuristics [18] while 35% of them used other checklists specifically developed for those studies. Very few studies applied guideline reviews or cognitive walkthroughs.

Fig. 5.
figure 5

a. User-based methods, b. Expert-based methods

Full size image

3.4 What Is the Type of Evaluation Methods Used in University Website Usability Studies?

The studies were investigated whether they applied manual evaluation or automated methods. Many of the studies applied manual methods with 60% while only 15% of them applied automated methods and the rest applied both methods (15%). Some of the tools used for automated evaluation were HTML ToolBox, PageRankChecker, SEO PageRank, Web Accessibility checker, HERA and WAVE.

3.5 What Are the Most Commonly Used Data Analysis Methods Used in University Website Usability Studies?

Data analysis methods applied in the studies were investigated and they were grouped as descriptive, predictive and qualitative methods as can be seen in Fig. 6. Most of the studies have applied descriptive statistics and they reported their findings with frequencies, percentages, means and standard deviations. The others applied predictive statistics such as Anova, factor analysis, t-test and correlation. Few studies applied qualitative methods and that was content analysis.

Fig. 6.
figure 6

Data analysis methods used in university website usability studies

Full size image

3.6 What Is the User Profile Distribution in University Website Usability Studies?

There were 34 studies that applied user-based methods. These studies were investigated according to the users participated in them. Mostly non-special users participated in the studies. There were only one study that included a disabled users. Participant distribution can be seen in Fig. 7. Although there were different user profiles that formed the audience of these sites, many of the studies were conducted with undergraduate and graduate students and usability experts. On the other hand other possible audiences such as high-school students, parents, faculty or administrative personnel involved in few studies.

Fig. 7.
figure 7

User profile distribution in university website usability studies

Full size image

The number of participants in the usability testing, survey/questionnaire based studies and heuristic evaluation studies were examined in detail since it has been an issue in the HCI research. It can be seen in Table 1 that the number of participants were varied a lot. For instance, for usability testing studies the lowest number was 5 while the highest was 115 participants. Similar to that the lowest number was 6 while the highest was 864 participants in survey/questionnaire studies. The number of participants in usability studies were lower than questionnaire studies since to conduct user test with too many participants is hard in terms of accessing participants, time required to conduct tests, etc. On the other hand, in heuristic evaluation studies, the lowest number was 1 while the highest number was 30. The numbers were lowest in heuristic evaluation since it was hard to access many usability experts.

Table 1. The lowest and highest number of participants in the studies
Full size table

3.7 What Are the Most Commonly Addressed Usability Issues in University Website Usability Evaluation Studies?

The usability issues reported in the investigated studies were examined and their distribution can be seen in Fig. 8. General usability characteristics of efficiency, effectiveness and satisfaction were grouped as general usability issues and coded according to that and they were the mostly addressed issues in the studies. Navigation, UI design, content quality, accessibility and search related issues followed that.

Fig. 8.
figure 8

Usability issues in university website usability evaluation studies

Full size image

3.8 What Is the Frequency of Mentioning the Issue “Accessibility” in University Website Usability Researches?

Accessibility; can be defined as any work, service, technology or environment to be usable, accessible and understandable by everyone, including children, elderly people and people with disabilities [19]. The goal of accessibility is to enable its users, to access the information they are looking for, to use the site and to understand the content of the site. In addition, web sites must be designed and have the necessary content to address not only a specific user population but also different user groups such as disabled, elderly, children [20]. Since accessibility is an essential feature for web sites regarding the various user profile of university websites, studies were examined whether they mentioned this issue. Nearly half of the studies mentioned about this issue but only four of them examined this issue in depth.

3.9 What Are the Research Institutions and Countries of the Studies?

Contributing institutions and countries were also investigated. Only the first author’s affiliation and country was coded. Top five contributing countries were Turkey, Jordan, USA, Malaysia and Saudia Arabia. On the other hand, top five contributing institutions were Zarga University, Yarmouk University, Uludağ University, Hacettepe University and Telkom University as can be seen in Table 2.

Table 2. Contributing research institutions and countries
Full size table

4 Conclusion

This comprehensive systematic mapping study presents the research trends between 2006 and 2016 on the usability issues of university websites based on 53 papers. According to the results of this review, user-based usability evaluation methods was used to evaluate the usability of university websites. The most frequently used user-based methods were usability testing and questionnaires. Studies were conducted with manual processes. Many of the studies were conducted with university students rather than faculty or staff although they were also the internal users of these websites. Both qualitative and quantitative data analysis were common in all these studies. Many of the studies just reported the usability problems rather than providing recommendations for the reported issues. In addition to general usability issues, the frequently mentioned usability issues were navigation, UI design and information/content quality.

Although previous systematic review studies have provided important insights into web site usability in general, this focused mapping study provides useful insights for researchers and practitioners that develops websites for especially universities or academic institutions. One of the limitation to be considered in these kind of studies is the reliability of the results. Therefore, the study was conducted iteratively. Authors first mapped articles alone and then came together to form consensus about their findings so researcher bias was tried to be overcome.