1 Introduction

About ten years ago, students still prefer to print out the materials than the digital version, especially in academic reading because they prefer to make highlights, underlines, and write some notes on the materials [6]. However, with the development of technology, students are used to read digitalized materials on digital devices, such as computers and tablets, instead of read the printed papers [5]. In 2014, Chen and Chen’s [3] study shows that no matter students prefer to read printed version or digital version, their reading attitudes have no difference.

Even students using digital devices to read learning materials, their behaviors on digital devices might still be similar to what they did on the printed papers. When students read materials assigned by teachers, they usually take notes and highlight important words/sentences [3]. Students have their prefer ways to take annotations while reading; they may annotate words in different ways (e.g., highlighting, underlining, or double-underlining). For example, when John, Andrew, and Mary read a text – “Every year in the U.S. factories release over 3 million tons of toxic chemicals into the land, air and water” – in the “Pollution” articleFootnote 1, they will use different annotations ways and have different focus. John only circles the word “air”; Andrew underlines the whole sentence; and, Mary highlights the three words – “air”, “water”, and “land”.

While reading an article, students might not want to annotate some important keywords intentionally or simply overlook those words accidentally. When they preparing exams and doing homework with the annotated article, they may skip those un-annotated words because they believe that all important words or concepts already have been annotated by them earlier. For example, while answering a question of “Environment Pollution” in the mid-term exam, John may only mention the toxic chemicals released to air because he didn’t annotated “land” and “water” earlier. The incomplete answer for the mid-term question may make him lose marks.

To avoid themselves missing important thing, students always try to borrow friends’ textbooks and notes before the exam comes. While preparing the forthcoming written exam and quiz, they take their friends’ annotations as reference. In the previous example, John may borrow Mary’s annotated text before the final exam comes. If he does so for the final exam preparation, he might probably notice the “land” and “water” highlighted by Mary are missed in his text copy. He might also circles the words on his copy so it has more complete annotations and he would probably have better answer for the question when he writes the exam.

2 Literature Review

2.1 Reading Activities in Digital Materials

Some studies show that people are getting used to read on the screen. In 2005, Liu has asked participants to think about their reading habits in the past ten years [6]. 83% of participants report that their electronically reading is increased. In Chrzastowski and Wiley’s research [2], students can choose to use digital books or get the printed books for their reading activities. When they choose the printed books, the research team sends the hard copies to students; on the other hand, when they prefer using digital books, they can read the materials online. The result shows that students prefer the digital books more than the printed ones.

Bounie et al. [1] have discovered that Amazon sells more digital books than printed books. Especially in higher education, digital materials are popular in undergraduate and graduate students [11]. Lopatovska and colleagues’ research [5] shows the reasons that people choose digital books include (1) digital books are convenient (58%); (2) study for school’s need (55%); (3) there is no printed version available (49%); (4) it is easy to use (48%); (5) digital books costs less (46%); (6) the feature of searching in the text (43%); (7) the interactive features (36%); and (8) they just want to use digital books (80%).

Because people’s reading preference has changed to digital devices, Tashman and Edwards’ research [11] focuses on analyzing people’s reading behaviour son digital materials. The participants in their study write reading diaries when they do Active Reading activities. There are approximately 25% of the diaries shows that participants use both papers and computers for finishing their Active Reading tasks. They also find that 63% of diaries in doing their reading activities are performed on computers only.

Above studies show that people’s reading hobbits are changed to digital reading materials and they feel comfortable to read on the screen. If people get used to annotate on the paper-based reading materials, they might also have the same habit and need when they read on the digital devices. The next section discusses the research related to annotations on digital devices.

2.2 Annotations on Digital Material

In traditional learning, teachers always give students reading assignments that ask students to read pieces of articles on papers or in a text. Reading and annotating articles are students’ routine jobs of study. Chen and Chen’s research [3] discovers that when students use paper-based way to study, students are frequently highlighting or underlining words, phrases, or passages, writing short comments in blank space, between lines, or near figures. Before an exam, students can find some annotations that may important to themselves by reviewing other students’ annotated text.

Nowadays, digitalized materials are common to be used and students may use digital devices like computers and tablets for their studying. Hoff et al. [4] classify functions provided by existing annotation systems into four categories: media formats support (e.g., support web document, office documents, PDF, or multimedia.), annotation functions (e.g., user can annotate on articles), interactions management (e.g., is this annotation private, group, or public shared? Do I want to get notifications when others make new annotations), and repository implementation (e.g., the repository is local, global, or client-server). The four categories help researchers understand the gaps between the annotation features that students need and current widely accepted annotation systems have.

Some other research provide annotation service to help users read and annotate articles on their computers. Yang et al. [12] have developed a web-based annotation platform – Personal Annotation Management System (PAMS) – where users can highlight, underline, attach notes and do voice recordings to the text in an article. Su and colleagues [9] and Yang and colleagues [13] improve the PAMS system, i.e., PAMS 2.0, to know students’ perceptions toward the collaborative annotation system and how the collaborative annotation system helps students improve their reading competence.

Another research provides a system which combines annotation service and collaborative learning together. Pearson et al. [8] aim to provide students an annotation system to help students learn better. They provide a collaborative system, BuddyBooks, to students and ask students to read article as teams. While students read articles, members in the same group have to stay at the same place and discuss. When students read and annotate on the article, the actions will be sent to other group member’s pad. Every members can see others’ annotations in different highlighted colors. If one student wants others to look at a particular paragraph in the article, he or she can just point it out on his or her pad and others will receive notification at the side bar and can easily follow. The result shows that students believe sharing members’ annotations is useful and enjoy the feature while doing reading activity.

In Pearson, Buchanan and Thimbleby’s research [8], they allow students to review other students’ annotations only in small group. The research team aims to provide students not only an online annotation system but also an annotation recommender so they can receive annotation suggestions to find useful ones instead of reviewing a small group of classmates’ annotations on their own. To find useful annotation suggestions for students, the relations between students’ annotations and their annotation behaviours are needed to be analyzed.

Above mentioned research allow students to use different ways to annotate their reading materials. The common annotation ways are underline, highlight, and note-taking. Other functions provided by annotation systems include, for example, students can attach multimedia resources (such as audio and video) to a word or sentence and students can collaborate with others in a group (such as point out an annotation). Melenhorst [7] records students’ annotation behaviors in the annotation tool to identify the relationship between reading phases and annotation ways (e.g. highlighting words and sentences, taking notes, copying passages to notepad, etc.). With showing the relations between reading phases and annotation ways on two-dimensional plane, Melenhorst has found that students use different annotation ways in different reading phases.

3 Evaluation Plan

The perceived usability toward the proposed system from both students and teachers are necessary to be known. This section talks about the experience and process that the research team had in a Tunisian university to execute an evaluation study. We start from introducing how to recruit participants include teacher(s) and students to explaining the stages that both teacher(s) and students would be done.

First of all, the teachers and students who are teaching and studying in Higher Institute of Computer Science and Management of Kairouan (ISIGK), Kairouan, Tunisia, are the potential participants of the evaluation study. The courses that teachers teach include educational games or languages in first and second year master degree’s students. The research team first approaches to the teachers and ask for their willingness of adopting the proposed system in their courses so students need to use the online annotation system for reading activities and the student clustering results would be provided for them.

As soon as a teacher agrees to adopt the system in his or her course, he or she is required to prepare few reading activities for students. As Fig. 1 shows below, six reading activities are created for students in the 3-month study and each of them has its own starting and end date so students can only do the particular reading activities within the predefined periods. The reading activities can be created in any language. In the case, the teacher has reading activities in both of English and French. He or she can also edit the reading activities, check out the cluster results, and review students’ annotations.

Fig. 1.
figure 1

List of reading activities

Full size image

Before students can start using the system for doing the reading activities, they need to first fill out a questionnaire about their experience of using any kind of e-readers. Further details and descriptions of the questionnaire can be found at Table 4 in Sect. 4.2 below. For accessing the reading activities, they also need to have accounts to sign in the system. The teacher can import a student list to create accounts for his or her students in batch.

With the account created by their teacher, students can see all the reading activities the course(s) they enrolled has, the reading time periods, and the forthcoming reading activities. When they want to do particular reading activity, they can simply click the correspondent “Reading” link to start as Fig. 2 shows. However, they cannot enter to do a reading activity prior its start date. Although they can no longer do any further annotation on the reading material after the end date of a reading activity, they can still review the reading activity and the annotations they made earlier.

Fig. 2.
figure 2

Student’s reading activities in all enrolled course(s)

Full size image

When students start a reading activity, as Fig. 3 shows, on the top of the window is the annotation function panel, students can choose their preferred annotation ways for their annotations; on the left-hand side of the window is the reading material; and, on the right-hand side of the window is student’s sidebar notes. Even students they accidently close the window, the system will still recover their annotations because the system is implemented in AJAX technique – the system sends and retrieved data to and from backend services asynchronously.

Fig. 3.
figure 3

Reading and note taking in the proposed system

Full size image

At the end of the 3-month study, students are asked to fill out another questionnaire which asks for their experience of using the Online Annotation System. The details and explanation of the questionnaire can be found at Table 5 in Sect. 4.2. At the same time, the research team also asks the teacher to review the clustering results. If the teacher doesn’t agree with the results, he or she can edit the results. After that, the research team interviews the teacher and asks him or her the questions to get idea of whether or not the system can help him or her to teach better, whether or not the clustering results meet his or her expectation, and how he or she thinks about the system. The detailed interview questions are listed in Table 1.

Table 1. Interview questions for teacher
Full size table

4 Research Model and Questionnaire

4.1 Research Model and Hypothesis

Providing a system for students to use, the research team wants to know the students’ perceptions toward it. We assume that there are three factors may affect the perceived usability of the system: (1) student’s experience of using e-readers, (2) student’s demographic information, and (3) student’s experience of using the system. The research model shown in Fig. 4 is built based on the seven research questions (listed in Table 2) we have.

Fig. 4.
figure 4

Research model

Full size image
Table 2. Research questions
Full size table

According to the research model, we have the following seven hypotheses made and would like to test them via the analysis of data collected with two questionnaires (Table 3).

Table 3. Hypotheses
Full size table

4.2 The Questionnaire

This evaluation plan includes two five-point Likert-scale questionnaires. The first questionnaire is Diffusion of Innovation questionnaire (DoI) and this questionnaire has 24 items. The purpose of DoI questionnaire is to get idea of students’ experiences and thoughts of using any kind of e-reader applications so the connection between their perceptions toward the system and their attitudes toward e-reader applications can be found. For instance, will students who think the use of e-reader application can help them learn better make them perceived more positive toward the Online Annotation System? The factors and correspondent items that DoI questionnaire has can be found in Table 4.

Table 4. Diffusion of Innovation questionnaire
Full size table
Table 5. Usability questionnaire
Full size table

After students use the Online Annotation System doing their reading activities and before the semester is end, the research team asks students to fill out the usability questionnaire. This questionnaire asks students whether or not they think the system is useful for them as well as their willingness of using the system later. The questionnaire has forty-one five-point Likert point items for four higher level factors: Like, Effectiveness, Efficiency, and Satisfaction. Each higher level factor has sub-factors. There are 5 items for Like factor; 16 items for the three sub-factors of Effectiveness factor; 10 items for the two sub-factors of Efficiency factor; and 10 items for the two sub-factors of Satisfaction factor.

5 Conclusion

This paper explains the details of the evaluation plan and procedure, research questions and hypotheses and the two questionnaires used for data collection. The evaluation plan wants to know whether or not the proposed system can help teacher to know students learning problems as well as the students’ perceptions toward the usability of the system. The study also collects students’ experiences and thoughts about the use of e-reader application before they start using the Online Annotation System. The research team chooses to use both of Diffusion of Innovation and Usability questionnaire. With the analysis of the collected responses of the two questionnaire, we are not only can verify the effectiveness of the system and students’ willingness of the system, but also can find whether or not students’ pre-experiences and attitudes toward e-reader applications will affect their perceived usability of the system and intention of keeping to use the system in the future.

This evaluation plan has already got approval from Athabasca University’s Research Ethic Board. The research team plans to do the 3-month study from April to June, 2017, in Tunisia. We have already found a teacher who is interested in adopting the system in his class in computer science in ISIGK, Tunisia. We also would like to find another teacher who teaches English or French course so we could know whether or not students in different disciplines perceived the usability of the Online Annotation System differently.