Exploring the Relationship Between Web Presence and Web Usability in Peruvian Universities

Abstract

Currently the use of the internet is an essential requirement for any business that aspires to reach many customers, sharing the information of their products, their achievements, their day to day, etc. This leads to the generation of a quantity of information year after year by these companies, which allows them to gain presence in the network. This reality also applies to the education sector, so this work will focus on Peruvian universities. Universities generate a large amount of content, and therefore it is necessary to be easy to find and read. In this way they will not be at a disadvantage compared to other universities that compete to attract more applicants. Our motivation is based on a study conducted in Turkey whose results show that the universities with the greatest web presence in that country have better usability. With this motivation, it was decided to do this work that, repeating the Turkish experience, verifies if there is a relationship in the Peruvian universities with greater presence, according to a ranking of universities called Webometrics. This evaluation takes as reference other usability studies, among them those that use tests with users and questionnaires, to obtain a ranking of web usability and to be able to contrast it with the web presence ranking.

1 Introduction

In today’s competitive world there is a wide variety of solution options from many organizations to different needs. These organizations seek to be constantly in the sights of their potential stakeholders in order to have more possibilities to offer their services. For this reason, they are interested in having an Internet presence, since it is one of the main mass media.

This is not different in the education sector, where universities constantly seek to be an option for potential students. Using a web page, these educational centers show the information to the possible applicant and their parents as: university programs, the benefits of their campuses, extracurricular activities, quality of teaching, libraries, among many others.

The information they offer is abundant and quite diverse. However, people have an idea of the information they are looking for when accessing a university website, so it is necessary to make it easy to find for them. This requirement, as in any business, must be focused on the target audience, which consists mostly of young people who are in their last year of high school and their parents.

There are currently recommendations to help websites to better achieve the objectives for which they were created, a high level of usability. The term “usability” refers to “the extent to which a product can be used by specific users to achieve specific goals with effectiveness, efficiency and satisfaction in a context of specific use” [13]. The universities that apply these recommendations in the most appropriate way will have a competitive advantage.

In other countries, such as Turkey, studies have been conducted that show that universities with a greater web presence have better usability [15]. This seen in another way may mean that a university with less presence worries less about its usability. Studies have been sought, like that of Turkey, conducted here in Peru but to date they do not exist.

For this reason, it has been decided to carry out this work that includes the evaluation of web pages of Peruvian universities and a comparison of their position in the Webometrics ranking. The result of this research allowed us to see if in Peru there is also that relationship usability-presence web, as well as to obtain recommendations on the usability of the websites that were tested.

1.1 Usability

According to ISO 25010 [8], Usability is understood as the ability of the software product to be understood, learned, used and attractive to the user, when used under certain conditions. A usable system complies with the following sub-characteristics:

• Allows the user to understand if the software is suitable for their needs.

• Allows the user to learn their application.

• Allows the user to operate and control it easily.

• It can protect users from making mistakes.

• It has a nice user interface and satisfies the interaction with the user.

• Allows it to be used by users with certain characteristics and disabilities.

It is an essential feature that every website must have for the dissemination of information to the public [22]. This also includes university websites that communicate information about their academic programs, teaching facilities, research, etc. [15].

1.2 Webometrics

Ranking Web Webometrics is the largest academic ranking of Higher Education Institutions. Cybermetrics Lab (CSIC) provides multidimensional and updated information about the performance of universities around the world based on their presence and impact on the web every 6 months.

Its main objective is to promote the academic web presence, supporting the initiatives of Free Access to increase the transmission of knowledge. To achieve this goal, the publication of rankings is one of the most powerful and successful tools to initiate and consolidate the processes of change in academia, increasing the commitment of academics.

The information published is only the positioning of these institutions. It does not show the values obtained that result from the counts or formulas that apply. The positioning methodology is based on these 4 indicators [23] (Table 1):

Table 1. Webometrics indicators

1.3 Web Presence

According to Webometrics [23], Web Presence is understood as the size or number of web pages of the main web domain of the institution. It includes all subdomains that share the same web domain and all file types, including rich ones like PDF documents. Webometrics obtains this information from the Google search engine, as indicated on its website.

It is expected that this indicator reflects the activities of an organization because it is related to its volume of content.

1.4 Usability Testing with Users

It is a technique to evaluate the system performing tests with users and whose objective is to obtain direct information about how the real user uses the system [9].

When performing these tests, the following benefits will be sought [21]:

• Know if participants can complete the specified tasks successfully

• Identify how long it takes to complete specified tasks

• Know how satisfied the participants are with the website.

• Identify opportunities for improvement in performance and user satisfaction

• Analyze the performance of the website in relation to its usability objectives.

In order to perform a user test, the following steps are followed [17]:

• 1. Development of research questions or test objectives.

• 2. Use of a representative sample of end users.

• 3. Representation of the real work environment.

• 4. Observation of end users who use the product.

• 5. Interviews and surveys of the participants by the moderator of the test.

• 6. Collection of quantitative and qualitative performance and measures of preference.

• 7. Recommendation of improvements to the product design.

A user test does not guarantee the success of the product evaluated at 100%, even if it is carried out with complete rigor. The reasons are the following [17]:

• Tests are artificial situations due to elements that can affect the results.

• The results do not show that a product works even if the tests have significant statistical results.

• Participants do not usually fully represent the population

• The tests are not necessarily the best techniques to use and it depends on the time, cost, precision and the time to apply them.

1.5 Questionnaires

Questionnaires are widely used to measure users’ perception and satisfaction when using a system [7]. They are composed of several questions, each one of them seeks to cover a quality construct [12].

Their main advantage is that they are relatively inexpensive and can generate many responses, rather than observation-based tests that require a laboratory, observation personnel, recruitment participants and other logistics issues [12].

2 Background

An important part of all research is to be able to review what has been studied previously on the corresponding topic. For this reason, a systematic review of all studies and research related to usability studies was carried out taking into account the level of web presence of several universities, regardless of the country. This allowed us to know what methodology and tools they have used and what conclusions they have reached based on their results in order to use them for the study in Peru.

The process of searching, filtering and selecting articles was carried out on April 22, 2018. Initially, 276 articles were found and 13 were selected.

The results of the search for related studies are based on a selection of primary and secondary studies, which are grouped according to the research question to which they respond, as shown in the Table 2. In addition, the percentage of studies that offer classified answers according to research topic is shown.

Table 2. Systematic review results

2.1 Usability Studies on University Websites

The study by Peker, Kucukozer-Cavdar and Cagilta [15] establishes a correlation between web usability and web presence for universities in Turkey. To do this, it evaluates the usability of the 5 websites of universities with the greatest web presence, according to the Webometrics ranking. The evaluation consists of user tests, which measured the time and achievement of previously defined tasks, and satisfaction questionnaires. The universities were classified according to their web usability and web presence, and subsequently a correlation between both values was calculated.

In the study by Torres, Méndez and Orduna-Malea [20] an evaluation is made of applications or webs of libraries of 50 universities selected from the Webometrics ranking. In this evaluation, 14 criteria are considered based on content that a web or library application must contain, and a total score is applied. Finally, based on that score, they are given a position and compared to their position in the Webometrics ranking and a correlation is sought.

A study like the previous one is carried out by Aziz et al. [2], which seeks to raise awareness about the importance of usability and accessibility for universities and other educational centers in Malaysia. In his work, he carries out a usability analysis and includes metrics such as number of broken links, page size and speed according to whether it is a public or private university, polytechnics and community colleges.

The study by Panigrahi et al. [14] shows a relationship between the gross enrollment rate and the level of web usability for distance education. In this paper an analysis of 15 universities in the United States, Australia and India is carried out, checking the presence of 23 attributes that they should fulfill for distance learning. An expert analysis was done for the websites and questionnaires and interviews were conducted to find the real reasons for the low recruitment. Finally, with this data, a linear regression is performed to verify the relationship between enrollment and usability criteria.

The study by Huang and Huang [5] focuses on evaluating university websites from subjective and objective points of view. For this, the behavior of the users during the tasks (user tests) is evaluated, obtaining objective information and grouped in diffuse clusters (where an element can belong to more than one cluster). In addition, a questionnaire is made to users where they can place values between 0 and 1 as a response and obtain subjective information. Finally, both data are combined giving them weights and a relation is found.

2.2 Methods Used in the Web Evaluation

In the studies found, only in [15, 20] we seek to establish a statistical relationship between web presence and web usability. In addition, in [10] the web presence was used as a selection criterion for the sample, but it was not encouraged to establish a correlation with web usability. Also, the other works showed web usability analysis but did not use web presence.

On the evaluation methodology, only in [3, 5, 15] were user tests, which will be used in this work. In the case of expert analysis, in [4, 10, 14, 20] an analysis was performed reviewing usability attributes in each web by themselves to obtain results. Finally, in [1, 2, 6, 11, 16] automated tools were used to help perform performance tests, content evaluation and structure. However, they do not make an in-depth usability analysis because they cannot show the accomplishment of needs of the user.

In addition, [3, 5, 15, 18] web usability questionnaires were used to obtain additional information. Of these studies, the questions were shown only in [5] and the questionnaire used was Palmer. Instead, in [18] questions were asked based on Nielsen’s heuristics. In both cases, only multiple-choice questions were used.

In [3, 15], the questionnaire contained demographic information questions, multiple choice questions (satisfaction) and open questions (feedback). In [3] it is indicated that the CSUQ questionnaire is used, unlike [15] where they do not give more information.

Finally, only in [15, 20] it is sought to find a correlation between usability level and web presence by analyzing the collected data. In [14] a statistical correlation between web usability and recruitment problems is sought.

3 Methodology

The present research work focuses on conducting a web usability evaluation of Peruvian universities that are within the Webometrics ranking. The data collection consisted of a user test and a questionnaire. Subsequently, an analysis of the information was carried out to obtain a correlation between its positioning according to usability and web presence.

3.1 Selection of Websites

The universities considered for the evaluation were the first 10 in web presence in the Webometrics ranking. This ranking currently considers 182 universities or colleges. This work was carried out in the second semester of 2018. It has been decided to hide the identity of these universities and they are named as follows (Table 3):

Table 3. Selected universities

3.2 Participant Profile

The selection of participants focused on those who are potential users of universities.

• Age: ages between 18 and 34 will be considered because they represent the highest number of visitors to a university website.

• Experience: At least one basic level in the use of web pages will be considered.

• Previous use: Participants will not be able to participate in the evaluation of websites they have visited previously.

3.3 User Testing

The evaluation of the selected websites consisted in the observation of the participants and how they interact with it. In order to carry out this evaluation in a correct way and that does not negatively influence the execution of the same, this procedure has been designed considering the following points.

Participants

The number of participants was 20 people. Each participant evaluated 4 websites, having a total of 80 evaluations. That means that there were 8 evaluations per web, which is enough to find their deficiencies.

Tasks

The tasks defined for this test can be performed for any of the 10 websites chosen. They consist of very basic instructions for the participants to seek information and answer the questions. The tasks consist in looking for information about:

• University Programs: (4 questions)

• Admission process (5 questions)

• Research Projects (3 questions)

It will be verified if the user manages to complete the objective and the time it takes to execute the task will be measured.

The score was calculated as follows for each task:

• Percentage of compliance (Pc) for each task that is equal to the number of correct answers among the total of questions.

• Task score (Pt) equal to the compliance percentage will be divided by the task execution time in seconds. If the task was not fulfilled, the result will be zero.

• Maximum score (Pm) that will be equal to the highest task score (Pt) of all the universities in that task.

• Score per university (Pu) averaging the corresponding homework scores (Pt).

• Final score per university (PFu) is equal to the score per university (Pu) between the maximum score (Pm)

The final position of each university will be determined by the median of the positions in each task. Finally, the universities will be ordered according to their median. If a university has the lowest median, the first position corresponds to it and if it has the highest median it corresponds to the tenth position.

Questionnaires

The questionnaire to be used for the evaluation of the websites is WAMMI [24] and has been chosen for the following reasons:

• Specializes in websites, unlike other questionnaires that are software oriented in general.

• It is not complex or extensive for the participant.

Calculation of the Score of the WAMMI Questionnaire

If a question seeks to confirm any strength or positive point on the website, the following score will be assigned:

• 1. Strongly disagree (1 point)

• 2. Disagree (2 points)

• 3. Neither agree nor disagree (3 points)

• 4. Agree (4 points)

• 5. Strongly agree (5 points)

Otherwise, if you want to confirm a problem or negative point, the following score will be assigned:

• 1. Strongly disagree (5 points)

• 2. Disagree (4 points)

• 3. Neither agree nor disagree (3 points)

• 4. Agree (2 points)

• 5. Strongly agree (1 point)

The total score of the questionnaire consists of the sum of the individual scores of each question, being able to reach from 20 to 100 points. According to this score will be given a positioning to universities.

3.4 Statistical Analysis

Once obtained the positions for web presence, user tests and questionnaire; are compared:

• User test positions with web presence positions.

• The questionnaire positions with web presence positions.

However, it is necessary to define the most appropriate correlation coefficient for these positions. The most popular coefficients to compare rankings are Kendall’s Tau and Spearman’s Rho. For this experiment it has been decided to use Kendall’s Tau-b for the following reasons:

• Kendall’s Tau is best in small data set [25]

• Spearman’s Rho is more sensitive to differences between any pair and is more appropriate to use when a significant difference in a single pair is critical.

4 Results

The evaluation of the university websites was agreed with different users and they were programmed within a range of 10 weeks. Next, a summary of all the data obtained in the user tests and in the questionnaires will be displayed.

20 people were contacted, and the corresponding user tests were carried out. The shortest duration of the test was approximately 40 min and the longest time was approximately 1 h and 30 min. There were users of different ages in a range between 19 and 31 years old, and 40% of them were between 25 and 28 years old. In addition, 25% of the users were women.

About the Internet experience, in Fig. 1 a large majority (85%) prefer the Google Chrome browser, 10% prefer Mozilla FireFox and 5% prefer Safari. In addition, 40% dedicate more than 8 h a day to surf the Internet and 25% dedicate between 4 and 7 h a day. The rest of the users invest between 1 and 3 h a day.

Fig. 1.

Internet experience

4.1 Task 1: University Programs

The first task was for users to search for information about any university program they want. They were asked to write the name of the program they chose, the number of semesters in length, the number of courses in the fifth level and whether they found the number of credits per course.

As noted in Fig. 2, University E has the first place in the university careers section. In second place is the University A that has a close score, next to the University F and the University J.

Fig. 2.

Task 1

The University E, the direct access to a university program is easier in the main menu but the information about its courses should be looked for in an embedded PDF that is not friendly. University A has a large list of all the programs, and it makes you invest more time searching. However, the information within each program is well distributed and easy to find. This is reflected in the measurements made because both have almost the same effectiveness, but University E has better times in the achievement of the task.

4.2 Task 2: Admission Process

The second task was for users to seek information about the admission process. They were asked to indicate if they found the following information: date and cost of the exam; contents that will be evaluated; detailed steps of the process; and scholarships.

As noted in Fig. 3, University J has the first place in the Admission section. The difference in score with respect to the University G and the University H is not very big.

Fig. 3.

Task 2

The University J has elements in its design that confuse the user at the beginning but is sufficiently ordered to find it quickly. In contrast, the University G does not have an ordered content to help the user; and University H has a more modern design with respect to the other universities, but the information is a little less orderly.

4.3 Task 3: Research Projects

The last task consisted in the search of information about research projects of the university. Users were asked to name a project, the leader or principal responsible and the budget assigned to that research.

As observed in the Fig. 4, University A is the university that has better usability in the Research section, with a great difference compared to other universities. The information is concise and well structured, allowing a shorter time to find it.

Fig. 4.

Task 3

The other universities have a more complicated navigation. Users often cannot find the information they were asked for, and users who found it have delayed doing so.

4.4 Questionnaire

After using each website, users answered a questionnaire with questions about positive aspects, negative aspects and suggestions they would make to the portal.

The questionnaire used was WAMMI and consists of 20 multiple-choice questions.

As observed in Fig. 5, University A has the first place in user satisfaction due to the orderly structure of its website and easy access to information. University D has the second highest score according to the opinion of the participants even though some of its pages could not be seen at times.

Fig. 5.

Results of WAMMI

5 Data Analysis

As described in the previous section, two methods of usability measurement have been carried out: user tests and satisfaction questionnaires; and the data have been obtained to be able to assign positions to each of the 10 universities. These positions have been defined for each of the 3 tasks and for the questionnaire.

The hypothesis for this research was:

• H0: There is no correlation between usability and web presence positions

• H1: There is a correlation between usability and web presence positions

where H0 is the null hypothesis that will be rejected.

When defining the correlation coefficient to be used for the analysis, this null hypothesis will be considered based on the chosen correlation coefficient.

• H0: τ = 0

• H1: τ <> 0

where τ is Kendall’s Tau-b correlation coefficient between web presence and the usability of Peruvian universities, with a significance level of 0.05 in a bilateral contrast. That is, the p-value must be less than 0.025. This value is the most used for experiments whose cost per error is low.

5.1 User Testing of All Universities

First, the comparison will be made with the positioning obtained in the user tests, so a summary of the information obtained in Table 4 is shown.

Table 4. Web presence and test user positions

The partial positions of each task must be converted into a single final position, so the median will be applied to these values because they are of the ordinal type. The mean is not used on an ordinal scale because the successive intervals on the scale are of unequal size [19]. It is not possible to apply mode in these cases because several universities have different positions in each task there is no value that is the most frequent.

From the median, the final position is calculated by university. That final position, shown in Table 5, will be compared with the web presence position. To do this, both sets of positions are processed and the bivariate correlation coefficient “Kendall’s Tau-b” is calculated. Once the analysis is executed, the result shown in Fig. 6 is obtained.

Table 5. Final positions

Fig. 6.

Correlations: web presence – user test (All Universities)

The correlation coefficient is almost zero (−0.092) and the level of significance (0.717) is much higher than the p value defined above. Therefore, a correlation with the obtained data cannot be established.

Due to this, it has been decided to look for it in subsets of universities. In this work the universities were grouped into state and private universities, and a correlation was sought between the universities within these groups.

5.2 User Testing with Private Universities

Once again, we carried out a positioning of the universities according to their scores in the user tests, but this time only the private universities are chosen (Table 6).

Table 6. Private universities positions

Figure 7 shows the results obtained. Again, the correlation coefficient is almost zero (0.072) and the level of significance (0.845) is much greater than the p value. Therefore, a correlation with the data obtained with this group cannot be established.

Fig. 7.

Correlations: web presence – user test (Private Universities)

5.3 User Testing with State Universities

A final positioning of the universities was made according to their scores in the user tests, but this time the state universities will be included.

Figure 8 shows the results obtained. Both positions have a perfect negative correlation. That is, the greater the web presence (content) of a public university, the lower its usability level. The level of significance obtained is 0, so it is a valid correlation (Table 7).

Fig. 8.

Correlations: web presence – user test (Public Universities)

Table 7. Private universities positions

5.4 Satisfaction Questionnaires for All Universities

A correlation with the results obtained from the satisfaction questionnaire will also be sought. To do this, first a summary of the scores obtained and their position according to these scores is shown. In addition, the positions according to the web presence according to Webometrics are also displayed in Table 8.

Table 8. Scores and positions – WAMMI (All Universities)

Figure 9 shows the results obtained. The coefficient is practically zero and the level of significance is very high, greater than the p-value, and this result cannot be considered. Therefore, no correlation can be established with this data and the next steps would be to look for the correlation for state and private universities separately.

Fig. 9.

Correlations: web presence – WAMMI (All Universities)

5.5 Satisfaction Questionnaires for Private Universities

We again take the subset of private universities and assign the web presence orders and questionnaire scores for these. The new order can be seen in the Table 9.

Table 9. Scores and positions – WAMMI (Private Universities)

Figure 10 shows the results obtained. We see a weak correlation between both orders and with a level of significance greater than the p value. Therefore, it can not be concluded that there is a correlation.

Fig. 10.

Correlations: web presence – WAMMI (Private Universities)

5.6 Satisfaction Questionnaires for Public Universities

Once again, we take the subset of public universities and assign them the new web presence orders and questionnaire scores. The new positions can be seen in the Table 10.

Table 10. Scores and positions – WAMMI (Public Universities)

Figure 11 shows the results obtained. A strong correlation is seen between both orders but with a level of significance greater than the defined p-value. Therefore, it cannot be concluded that there is a correlation.

Fig. 11.

Correlations: web presence – WAMMI (Public Universities)

6 Conclusions

According to the results, a zero relation was obtained for the user tests and a weak relation in the case of the questionnaire. However, p-value was lower than the level of significance and the null hypothesis could not be rejected. Therefore, it was decided to look for a correlation considering separately the private universities and the state universities.

In the case of private universities, a strong relationship was obtained for the user tests and a weak relation was obtained for the questionnaire. However, p-value was lower than the level of significance and again the null hypothesis could not be rejected. It can be interpreted that private universities update their website according to priorities different from the amount of content they publish, and that usability is affected for that reason. These universities are self-sustaining, so starting from their priorities is to make a profit.

In the case of state universities, a perfect inverse relationship was obtained in user tests, so the hypothesis can be rejected in the case of these universities. According to this, it can be interpreted that a web with a greater amount of content is more difficult to find something specific if there is not a design that really helps to do it. The estates universities always have several applicants much greater than the number of vacancies, so they do not need the web to be a tool to attract applicants. For that reason, they could be neglecting its usability. In the case of the questionnaires, a strong inverse relationship was found but again the p value was lower than the level of significance and the null hypothesis could not be rejected.

However, only an inverse relationship between usability and the web presence of Peruvian public universities has been found, but only in user tests. This result is quite different from that obtained in the Peker study, where a strong direct relationship was found in user tests and questionnaires for the 5 universities. Being different countries, it is possible that the way in which usability is prioritized is different.

It is considered necessary to carry out an additional study to confirm this relationship between usability and web presence in Peruvian state universities. This new study should have a methodology designed especially for these universities. The reasons are as follows:

• The difference between two public universities was almost nil, so it would be better to corroborate this in a new study. If the universities had different positions, the correlation obtained would not have statistical significance.

• The user tests had to include workable tasks in the 10 private and state universities that were not homogeneous with each other. The new study should include tasks designed especially for Peruvian state universities.

• The state universities considered in this study were only 4, so the sample is small to conclude a sufficiently convincing correlation.

Finally, the users mentioned more frequently negative aspects related to the location of the content they seek and the information that is published. This result was obtained from the open questions of the questionnaire. This would show that in many of these websites the information that the university considers necessary is published, but without considering an adequate maintenance of the same.