Keywords

1 Introduction

Grading is still an important and non-trivial part of modern university life, since it heavily influences student behaviour [1]. The assessment process “can be used as a means of channelling students’ energies, and the feedback that it generates can provide students with an opportunity for reflection” [2]. Various approaches to assessment offer students the possibility to demonstrate their competences using a full range of examination modalities for different types of knowledge.

Student and teacher involvement in assessment, including digitally-enhanced assessment, is a crucial aspect of 21st century learning [3]. However, despite the large number of possible assessment strategies, the most common method in higher education is summative assessment, which evaluates factual knowledge with (mostly) standardized, semi-standardized and/or open-text questions, leading to a huge correction workload if conducted with paper-and-pencil exams. As a result, lecturers must allocate a large amount of time to assessing tests, a task that could be carried out more efficiently by automated systems.

Next to the noticeable time and money savings [4] due to automatic delivery, storage and the (semi-)automated correction of (semi-)standardized question types, along with the improved readability, structure and clarity of typed open-text answers, eExams offer various advantages and improvements for assessment. Online exams increase student engagement due to their relative novelty and provide greater flexibility if compared to traditional testing methods [4]. The greater efficiency of online exams provides students with instant grading and feedback [5]. Moreover, today’s students are not used to extensive handwriting anymore [6], leading to hand pain and bad handwriting when taking paper-and-pencil exams. Furthermore, with eExams different handwriting styles do not influence the lecturer when grading. It is also easier, for example, to evaluate all the answers for a single question at once by switching between the different students’ answers, thus enabling each question to be evaluated on its merits without being influenced by other answers provided by the student. Hence, online exams enhance objectivity in that they are not steeped in subjective construction processes. Additionally, eExams can be saved and shared between lecturers easily, offering the opportunity for (mobile) synchronous correction. Although the creation of online questions might require greater initial effort, the establishment of a question pool leads to a reduction in effort over time, thus guaranteeing the sustainability of online testing. Moreover, the digital format offers opportunities for statistical analysis of questions, improving the quality of questions over time. Finally, the shuffling of questions and answers decreases chances for cheating. Another and very promising advantage of online exams is the opportunity to include additional software and multimedia into the examination environment. Biggs and Tang [7] postulate that a well-founded lecture design includes assessment. Their concept of “constructive alignment” emphasizes the necessity of establishing coherence between all phases and elements of the learning process. Intended learning outcomes, teaching/learning activities, assessment tasks as well as grading should support one another [7, 8]. Although Biggs and Tang’s [7] approach has received criticism, we are of the opinion that it is particularly useful in highlighting the power of assessment to shape students’ experiences [9]. Thus, in order to ensure coherence, the software tools used for teaching and learning should be part of the examination process. This means that mathematical calculations and analysis, spreadsheets, literature essays etc., can be easily processed online using appropriate software. Being able to use specific software and multimedia in electronic exam environments facilitates the way to promising (hands-on) performance assessments too.

Despite all the positive aspects of eExams mentioned so far, we found a lack of technical solutions to conduct secure online exams for larger audiences. The problems we encountered were too small computer rooms as well as a lack of consideration of the security requirements which inevitably arise in the context of (electronic) exams: confidentiality, privacy, integrity, authenticity, and availability. The first four aspects are commonly addressed through cryptography (e.g. encryption of transmitted and stored data, and authentication of messages and users), whereas the last one is provided by redundancy and continuous monitoring of the IT system.

To overcome the existing shortcomings, we implemented the Secure Exam Environment (SEE).

2 The Secure Exam Environment (SEE)

The Alpen-Adria-Universität Klagenfurt (AAU) launched the Secure Exam Environment (SEE) for online testing in 2011 [10] with the aim of supporting large class sizes and modern teaching as well as testing strategies, while working within budgetary and organisational constraints. In particular, the SEE makes use of the students’ existing technical resources, specifically their personal computers (laptops). The efficiency of allowing students to use their own devices is complemented by an effectiveness factor since they are presumably familiar with the hardware.

The SEE disables access to students’ own files and data as well as to other internet sites. Loan devices are offered for those who do not own a laptop. As a result, institutional asset requirements as well as the associated maintenance costs are minimized. We are currently able to test up to 220 students simultaneously.

Furthermore, courses are becoming increasingly based on or supported by different software tools and programmes, for example a statistics programme or special mathematical software packages. Traditional testing methods do not allow testing related to the use and application of such software programmes; the SEE, on the other hand, offers this possibility which is consistent with pedagogical coherence [8].

The actual exams are presented as quizzes, a key component of the Moodle learning management system (LMS) utilised by the AAU.

2.1 How Do We Secure the SEE?

As with traditional exams, cheating is a major problem. On the one hand, the issue of impersonation must be addressed, on the other hand, access to (electronic) materials such as notes, books, (external) devices, or local or online resources which are not allowed during an exam must be prevented.

We discourage impersonation by requiring students to present a valid student ID card. In addition, we utilize the photo stored in each student’s Moodle profile to verify that they are indeed using the correct login credentials. During the exam we compare the registration data with the (number of) students who started the exam in the Moodle course. Starting the exam from outside a lecture hall is not possible due to automated grouping of registered students in Moodle after registration, since solely the members of this grouping are able to open the exam (students leave this grouping by checking out with their student card after completing the exam). In addition, only devices with an IP address within the IP-range of the specific lecture hall are allowed to open the exam and the settings of the exam in Moodle only allow access via the Safe Exam Browser: students are only permitted to attempt the exam once, and the exam is available in the Moodle course only at the scheduled time and, once started, only for a certain period.

Preventing students from accessing materials which are not allowed during the exam is more complex. In contrast to other electronic exam environments (e.g. [11]), we avoid the use of special equipment and encourage students to use their own device. However, accessing the Moodle server directly via a web browser running on the student’s OS is an inefficient approach. In this case, blocking connections to Wikipedia or other online resources may be simple, but cheating by using materials stored on the local hard drive is rather easy. Since we do not want to force students to install additional software (such as lockdown modules) on their personal laptops, we have to use our own operating system (OS) in order to restrict the access to the local resources and programmes that are prohibited during the exam. We decided to boot this OS via the Preboot eXecution Environment (PXE) protocol over a local area network (LAN), since the handling of USB sticks or DVDs is very error-prone, time-consuming and inflexible, especially when additional software is needed, and the usage of WLANs is too insecure and interference-prone. Clearly, this requires that the client is able to boot via the network.

In order to support a very broad range of (private) laptops and prevent students from accessing local resources and programmes, our solution is designed as a minimal Linux system. At the moment, this OS is realized using Fedora and Knoppix, which enables us to boot Legacy or UEFI devices (both Apple and PC). In order to restrict the access to external resources, we implemented corresponding firewall rules. Since Moodle as an LMS not only provides exam features but also chatting capabilities and course related material, a solution was needed to prevent the access to such resources and activities during exams. Running an ordinary web browser in centOS – even when restricted with firewall rules – would not have completely solved the cheating problem. Fortunately, the Safe Exam Browser (SEB – [12]) is fully supported by Moodle-core. The SEB is much more secure than an ordinary browser, since it prevents students from opening other programmes or additional web browser windows during the exam and ignores certain key combinations or clicks. So by restricting the access to the exam page only, cheating by exploiting Moodle’s features is not possible anymore. However, the SEB is only available for Windows XP, Windows 7 and MAC OS X. Therefore, we are forced to boot a minimized Windows 7 as a virtual machine on the minimized Linux system via VirtualBox [13] (see Fig. 1). Additionally, proprietary software which only runs on Windows systems is still widespread in the educational sector. On the one hand, the reliance on Windows 7 is a drawback in terms of performance, on the other hand, it adds flexibility regarding the management of the virtual machine image. Furthermore, hardware driver management is done completely in Linux, which is known for its broad hardware support especially for older hardware without the need to install specific drivers. The selection of the allowed programmes (beside the SEB) during the exam is set via a configuration file, which is retrieved from an Intranet Service. In the GUI of this service, administrators are able to configure the exam (e.g. only Calculator or Calculator and Excel or GeoGebra or Eclipse and PDFs allowed).

Fig. 1.
figure 1

The operating principle of the Secure Exam Environment (SEE)

Full size image

Starting an online exam using the SEE includes booting a minimized Linux from the LAN, then the minimized Linux automatically starts the Windows 7 virtual machine (VM), Window 7 automatically starts the SEB, the SEB automatically connects to the homepage of the AAU’s learning management system Moodle, and finally users have to log in to Moodle and select the exam.

2.2 How Do We Ensure the SEE is Reliable?

Reliability for examiner and examinee is a critical issue and depends on the availability of (information) technology - e.g. computers and computer networking technologies - during the exam [4]. At the time of writing, the SEE depends on the online connection between the SEB and the Moodle Server. In case of network failure, none of the users can save current results or proceed to the next question. Thus, the temporary storage of the answers (during network failures) remains a problem. Fortunately, Moodle saves the last answer received and the progress of each examinee. Therefore, the examinee is allowed to continue the exam from the point where the error occurred after potential network problems are solved. In the worst case scenario, the last answer of the examinee is lost. Hardware failures of laptops are not a problem because all answers provided up to the failure would have been stored on the server and the student can simply continue his or her exam on one of our loan devices.

In the context of testing, archiving exams is another important aspect. According to Austrian legislation [14], documents related to written exams have to be archived for at least six months, whereas protocols of oral exams have to be archived for at least one year . Moodle, however, offers a practical solution as it automatically archives exams, which dramatically reduces the physical storage requirements and, as a positive environmental side-effect, the amount of paper needed (especially in the case of no-shows).

2.3 How Do We Maximize the Availability of the SEE?

The availability of the SEE can be affected by hardware failure, network drop outs or service outages.

In order to provide maximum availability of the network connection, we only support wired LAN connections at this point in time. Despite recent developments, WLAN remains too error prone and, additionally, a malicious user could perform a denial-of-service (DoS) attack on the WLAN access points quite easily and hence prevent all users from taking the exam. To achieve the DoS attack, a battery-powered pocket-sized WiFi jammer could be mounted close to or in the room where the eExams take place.

To ensure the maximum stability of the network system, the network department of our university provides high redundancy within the network-core, distribution-switches, firewalls and the border-router as well as load sharing with the Border Gateway Protocol (BGP) in a multihomed environment and redundant cables. The load is separated among different virtual networks. The equipment in use in the core and distribution layer are high-end-components. Furthermore, an uninterruptible power supply is guaranteed during eExams through a diesel generator.

In order to reach the high availability targets of our SEE, we started to monitor all hardware components and services involved in the eExam process. Drop outs of components or services or deviations from thresholds within defined time intervals result in alerts, allowing support staff to react to and resolve issues immediately, leading to crucial time-savings within the identification process of failures, a very high level of availability and continuous system optimization. Monitored components and services include availability of the SEE-servers (implemented with centOS) including CPU, storage as well as DHCP, NFS, TFTP and HTTP services; availability of the administration backend of the SEE including the corresponding HTTP service; availability of Moodle including HTTP-access, as well as end-to-end-tests in the lecture halls with minimal computers (Raspberry Pi); availability of the network (connection between SEE-server, clients and Moodle), and end-to-end performance tests within the network with probes (Raspberry Pi).

2.4 How Do We Protect the Privacy of Examinees?

The answers provided by the students as well as the grades they receive are private and must be protected as stipulated by the Austrian Data Protection Act [15]. Subsequently, we use encrypted and authenticated transmission lines between the SEB and the Moodle server (HTTPS) and the Moodle server uses login/password authentication for students and lecturers to ensure that access is only provided to authorized persons.

2.5 How Do We Support Flexibility for Examinees?

One service for students, which followed from the development of the SEE, are so called slotted eExams. For the execution of eExams with the SEE, we developed an online-process to register for an eExam some time before the test takes place as well as an online-registration process right before the exam in the lecture hall. Thus, exams, registration data as well as access rights are available online. These processes enabled us to offer several time-slots for an eExam within a week, from which students can freely choose when they want or are able to take an exam. Especially for students who are employed next to their studies, who need to foster children or relatives or whose mobility is restricted, this service is very helpful.

The decision as to whether an eExam is conducted in a traditional way on a fixed examination date or as an slotted eExam is made by the lecturer: slotted eExams can only work if a suitably large question pool is available, such that on different days randomly generated questions and/or exams are sufficiently dissimilar from each other.

3 Technical Obstacles and Challenges

One of the current restrictions of online exams is the required network connection. WLAN is still prone to failure, especially for larger groups of students. This results into another challenging aspect, namely that lecture halls require LAN and power sockets at least every second seat. Unfortunately, not all lecture halls fulfill these requirements and retrofitting is extremely expensive. The obstacle with the LAN sockets could be overcome with access points. However, using a device only with battery during exams is too risky.

A persistent challenge are new generations of students’ laptops, requiring continuous adaptation of the SEE. For example, UEFI as a new interface between hardware and OS was rather complex. Moreover, some manufacturers started to disable the possibility of PXE-, resp. Net-Boots on their devices, forcing us to find workarounds. Furthermore, since a lot of new laptops come without Ethernet-sockets, we needed to integrate booting via adapters into the SEE.

4 Organizational Obstacles and Challenges

When it comes to eExams, organization is crucial: on the one hand, it is important for users to easily get through the process with the guarantee that everything will run smoothly, and, on the other hand, technological gaps need to be filled. Moreover, only a limited number of lecture halls is suitable for online exams for reasons mentioned above. Thus, lecturers have to book them early. Furthermore, it cannot be assumed that all students have a portable device. Loan devices must be ready for those students and in the case of technical problems or breakdowns of the students’ own devices during the exam. The AAU currently has approximately 100 laptops serving as loan devices for students.

4.1 Organizational Challenges Before an Exam

Students can come to information days, where we change the boot-order of their devices to allow PXE-, resp. NetBoot. Thus, students get a briefing on how an eExam works before the actual exam preventing unnecessary stress.

Lecturers need support when executing online exams as well. In the preparation phase lecturers (in our case) can rely on the support of specially trained eTutors, who help with setting up a test or in choosing pedagogically suitable examination designs.

4.2 Organizational Challenges in the Lecture Hall

Right before the eExam, the registration equipment including computers and card readers, loan devices and LAN-cables as well as adapters have to be physically taken in the specific lecture hall. At the registration desk, students get a LAN-cable as well as a adapter and a loan device if needed and register for the eExam by scanning their student card with a card reader. Other possibilities for the technical identification of students like iris recognition, retina scanning or fingerprint identification would be available, but they would infringe privacy. Therefore, at the moment users are identified by comparing their face with the photo on their student card.

Technical support is provided for students before and throughout the examination. In case of a hardware failure, students get a preinstalled loan device and can continue the eExam immediately. Right after the exam it is a great challenge for the organizational staff to get back sound LAN-cables and loan devices. Students sometimes tend to remove the cables with not enough care, ending up damaging them.

4.3 Support After an eExam

Lecturers get support from the eLearning department after an online exam with the evaluating process or with a statistical analysis of questions.

5 Experiences with eExams at the AAU and Further Developments

In June 2011 we began offering online exams with the SEE. Table 1 shows the growth of eExams conducted with the SEE at the AAU over the last six years.

Table 1. The progression of eExams with the Secure Exam Environment (SEE), * in process
Full size table

5.1 (Dis-)Advantages for Students

In 2012 we surveyed students about the benefits of eExams as well as any obstacles, and technical problems encountered. We received 308 usable questionnaires from a total of 1,075 students. Results showed a positive attitude towards online exams across all four faculties of the AAU.

The majority of the students who took the survey claimed that the key benefits of online assessment include: rapid exam results, improved readability of free-text answers, improved structure and ability to revise answers, and noticeable time savings. Students also found eExams particularly interesting, convenient and in general better than paper-and-pencil exams. Some students did not see any difference compared with conventional testing methods. Students also appreciated the novelty associated with online testing, including its environmentally friendly nature (less paper and printing costs) and the fact that their hands got less tired than during a handwritten exam.

Still, students encountered quite few obstacles. Problems include the additional time that some students require for example to boot their device or to connect cables and difficulties with the structure of the exam or the types of questions. Some students also had troubles with typing and others were not familiar with the loan devices they were provided with. It is clear, however, that students reported more benefits than obstacles.

In addition to the survey, we noticed that students at our university usually do not make use of their right to personally contact the lecturer to ask for more valuable feedback on the exam results [2, 5]. This hurdle can be overcome with automated or individual feedback in eExams. The Moodle login-data shows that all students seize the chance to view feedback without the need for personal consultation instead. Moreover, eExams at the AAU are also used for self-testing to give students the opportunity to receive feedback on their performance (see also [5]).

5.2 (Dis-)Advantages for Lecturers

Lecturers commonly mention the automated evaluation of standardized questions and the convenience of avoiding hand-written exams as clear advantages of the SEE. Furthermore, questions can easily be created, stored, extended, organized, and reused for future exams, fostering sustainability and decreasing work load. Thus, the time savings can be devoted to new topics or in-depth discussion [5]. Since lecturers at the AAU can freely choose which type of exam to offer to their students, solely those lecturers who see more advantages than disadvantages in eExams use them.

5.3 Further Developments

Currently, we are only able to execute one eExam with specific settings, e.g. additional software, at the same time. Therefore, we are developing a boot environment which enables us to run eExams with various additional software simultaneously by recognizing the identity of the student and transmitting the proper exam environment, even in different lecture halls at the same time. Furthermore, the support of newer devices without LAN ports is in development. However, for the future it is planned to provide a WLAN access point. Finally, like every software solution, the SEE needs constant security and compatibility updates.

6 Conclusion

eExams extend the possibilities for assessment in terms of quality and efficiency. Better alignment in teaching, the opportunity to use additional software, reduced possibilities for cheating, and reduced effort for correcting exams are some of the many advantages. Nevertheless, they are only one of a variety of methods for assessing performance and giving students feedback on their learning progress.

However, bring your own device (BYOD) solutions are crucial for larger exams. Next to a broader use of existing methods for testing (online), such as pre-tests, self-assessment, peer-assessment or problem sets, to name a few, further research is needed to extend the flexibility of online assessment, for example, by finding solutions to utilize WLAN for secure eExams, and to optimize promising techniques such as adaptive testing.