1 Introduction

The recent focus on cybersecurity education has led to the development of cybersecurity learning materials across various academic institutions and organizations in the United States [19]. Among such initiatives are web-based learning modules developed by the Security Injections @Towson project [5, 13, 17]. These learning modules target key secure coding concepts including integer error, buffer overflow, and input validation in various programming languages, for Computer Science 0 (CS0), Computer Science 1 (CS1), and Computer Science 2 (CS2); and general security concepts, such as phishing, passwords, and cryptography for use in Computer Literacy courses. The Security Injections @Towson modules have been found to be highly effective at improving security awareness and the ability to apply secure coding concepts [13]. In over six years of dissemination to over 150 institutions, the following issues have been observed: 1) instructors noted that students tended to skip content and proceed directly to lab exercises, and 2) large-scale adoption by instructors remains a challenge.

The modules originally developed in the project are web-based learning modules that follow a traditional linear format (we refer to these as 1.0 modules hereafter). This type of module design, that presents a large amount of content at one time, is common in many disciplines, but can lead to issues like skimming and skipping [11, 16]. In this paper, we discuss the key issues that lead to skipping and skimming, including the amount of content presented on a screen and decreased student engagement. Presenting less content allows students to process information more easily and improves learning; increasing student engagement motivates students to learn [1, 2, 9, 11, 12, 16, 18]. We describe enhanced modules, Security Injections 2.0, that address these issues by utilizing segmentation and interactivity, and a study that tests the engagement of the new modules. In addition, the enhanced modules include an auto-grading functionality to facilitate easier instructor adoption.

To assess the effectiveness of enhanced learning modules towards student engagement and instructor adoption, we compare the Security Injections 1.0 modules to 2.0 and explore the following questions:

Q1::

How can we improve the traditional web-based learning modules to reduce content skipping, increase engagement and instructor adoption?

Q2::

Can the use of 2.0 modules with segmentation and interactivity increase student engagement compared to the 1.0 modules?

Q3::

Can the use of 2.0 modules with auto-grading increase instructor adoption compared to the 1.0 modules?

2 Literature Review

Web-based learning modules present content using hypertext. Individuals reading hypertext have a tendency to skip or skim the content [14]. In this section, we discuss: 1) why hypertext readers skip or skim the content, and 2) how can we improve interactivity in learning modules to engage learners.

2.1 Hypertext Reading and Content Skipping

Hypertext is text that contains links to text, audio, video, or graphics in other documents or media resources, giving flexibility to readers to click any of the links to gain knowledge [3]. Hypertext readers often adopt a reading strategy which determines what to read and what to skim [3, 7]. This may lead more selective reading for lengthy documents. Readers who skim have a tendency to read the first half of the paragraph and if they determine that the gain of information is low, they may skip the other half of the paragraph and jump to the next paragraph [4]. This skipping may lead readers to lose important information [4, 11]. Selective reading may also lead to less in-depth reading, less concentration and attention towards the content [8]. In addition, skipping of the content leads to poor learning [12].

Research suggests that instead of presenting large amounts of hypertext content at once, the content should be broken into smaller chunks and presented one idea at a time on a single screen [1, 2, 9]. This concept is referred to as segmentation. Segmentation improves processing of information in the working memory and makes recalling and retention of concepts easier [9].

2.2 Improving Interactivity in Learning Modules

Interactivity in e-learning is the “responsiveness to the learner’s actions during learning” [9, 18]. Interactivity improves engagement and motivates students to learn [18]. The types of interactivity in e-learning environments include: dialoguing, controlling, manipulating, searching and navigating [9]. In this paper, we focus on dialoguing and controlling.

Increasing Interactivity with Dialoguing - Dialoguing occurs when the learner answers questions and receives feedback to his/her input. Dialoguing has been considered beneficial, as learners can relate the feedback to the current content [15]. In e-learning, dialoguing can be implemented using assessments (like formative questions) with appropriate feedback [15]. This assessment can be conducted using multiple-choice, fill-in-the-blank, short answer and essays formats. The feedback provided on the answers can be either immediate or delayed. The feedback provided can also be classified based on the amount of detail provided on the answers [6, 15] and is categorized as: 1) Knowledge of Results (KR), 2) Knowledge of Correct Response (KCR) and, 3) Elaborate Feedback (EF).

Knowledge of results (KR) informs the learner if their answer is correct or incorrect; knowledge of correct response (KCR) informs if the answer is correct or incorrect and includes the correct answer; and elaborate feedback (EF) informs if the answer is correct or incorrect, includes the correct answer, and also includes a concise explanation of the correct answer.

We plan to implement dialoguing in 2.0 modules using Multiple Choice Questions (MCQs), true or false and constructed response (CR) type of assessments with immediate knowledge of results (KR) and immediate elaborate feedback (EF).

Increasing Interactivity with controlling - Controlling implies that the learner can determine the pace of the presentation. Controlling helps students learn better by allowing them to process information at their own pace [2, 9]. We plan to implement controlling in 2.0 modules using answer-until-correct [15] with immediate knowledge of results (KR) and immediate elaborate feedback (EF) to MCQs, true or false and constructed response (CR) type of assessments.

Segmentation breaks large content into smaller chunks and presents them one at a time which may result in less reading and less skipping of content. Less skipping of content leads to increased learning [12]. Interactivity (dialoging and controlling) on segmented chunks leads to engagement and enforces learning (see Fig. 1.).

Fig. 1.
figure 1figure 1

Literature suggests that segmentation and interactivity in modules may increase learning

Full size image

3 Incorporating Segmentation, Instant-Feedback and Auto-Grading in Security Injection Modules

The Security Injection learning modules 1.0 were developed on the cognitive learning principles of Bloom’s taxonomy and adopt a uniform structure. Each module begins with a background section to describe the problem with examples, followed by a “Code Responsibly” section (that includes methods to avoid security issues), a laboratory assignment with a security checklist, and discussion questions. The module structure is designed to help students to first understand the problem through the background and code responsibly sections, remember it through the laboratory assignments, evaluate it through checklists, and apply the concepts learned through discussion questions.

In the 2.0 modules, we enhanced the 1.0 modules by applying segmentation to reduce skipping of content, and interactivity, using dialoging and controlling, to improve student engagement. We implemented segmentation by breaking up the module content per section (background, code responsibly, laboratory assignment, discussion questions) and presenting each section, one at a time, on the screen (see Fig. 2.). We implemented dialoguing using formative assessment including true or false, multiple choice and constructed response that include immediate knowledge of results (KR) or elaborate feedback (EF). We implemented controlling using answer-until-correct giving the students control over the number of attempts. Students receive immediate knowledge of results (KR) feedback until third attempt and immediate elaborate feedback (EF) thereafter.

Fig. 2.
figure 2figure 2

Modules broken into sections

Full size image

Module Design - In the background and code responsibly sections, students are required to go through the content and answer a set of checkpoint questions. Each question provides immediate feedback on submit (see Figs. 3 and 4.). The student cannot advance to the next section until all questions are answered correctly. In the laboratory assignment and discussion question, students answer text-based, multiple choice questions, and identify vulnerabilities based on a security checklist (see Fig. 5.). These are also auto-graded.

Fig. 3.
figure 3figure 3

Instant-feedback on incorrect answer

Full size image
Fig. 4.
figure 4figure 4

Instant-feedback on correct answer

Full size image
Fig. 5.
figure 5figure 5

Auto-graded security checklist

Full size image

System Implementation - To implement the enhanced modules, several solutions were considered (including writing the system from scratch) before determining that a modified version of Stanford University’s class2go web-based application (https://github.com/Stanford-Online/class2go/) was most appropriate. Class2go is built using the Django framework. Class2go is an open-source framework that provides core functionality, including user registration, course creation, test administration, and some components for auto-grading. We wrote code to auto-grade the security checklist, and regular expressions that match keywords to verify answers for short-answer questions (constructed response). In addition, we developed an instructor dashboard with a progress grade-book to monitor student progress (see Fig. 6.).

Fig. 6.
figure 6figure 6

Instructor grade-book

Full size image

4 Study

4.1 Methodology

A quasi-experimental study was conducted in fall 2014 across four sections of a CS0 course (using C ++) at a large public university, using a posttest only control group design. Two of the sections used the 1.0 version (control group) and the other used 2.0 (treatment group). The study was conducted during the laboratory sessions which were at different times for each section. Three modules - integer error, input validation and buffer overflow - were introduced, in that order, with approximately four weeks between the interventions. Both groups were administered a student engagement survey at the end of the semester.

The survey instrument used in this study measured student demographics and student engagement. While questions related to student demographics were derived from previous security injection studies [13], which measured students’ gender, age-group, ethnicity and major, we adapted a set of eight item questions from a well-tested User Engagement Scale (UES) [10] to measure student engagement. The eight item student engagement questions were recorded on a five point Likert scale. (See Table 1 for sample survey questions.)

Table 1. Survey questions
Full size table

Based on the survey scores, we proposed the following hypothesis to compare Security Injections 1.0 and Security Injections 2.0 (treatment group) on the following dependent variable: student engagement score.

H1: The mean of survey scores for student engagement in the treatment group will be significantly higher than the mean of the survey scores for student engagement in the control group.

4.2 Initial Results

A total of 116 students participated in the study. After filtering missing data and outliers, 80 (42 in the treatment group and 38 in the control group) students including 54 (29 in the treatment group and 25 in the control group) males and 26 (13 in the treatment group and 13 in the control group) females completed the survey. For student engagement, each response was assigned codes from 1 to 5, on a five point Likert scale, 1 representing ‘strongly disagree’ and 5 representing ‘strongly agree’. The engagement score for each respondent were calculated as the mean of codes for eight questions. The Cronbach’s alpha, for eight-item engagement questions, was found to be 0.74, which suggested good internal consistency. The hypothesis was tested using independent samples t-test. We picked independent samples t-test because Shapiro-Wilk test showed that scores for student engagement in both the groups (treatment n = 42, control n = 38) satisfied the conditions of normal distribution (treatment p = .593, control p = .187) and homogeneity of variance (F = 2.554, p = .114 > 0.05).

Comparison of survey scores for student engagement in treatment and control groups.

In the survey results, the mean score for the treatment group (n = 42, mean = 3.43) was found to be significantly higher at 95 % level (t = -2.265, p = 0.026) than the mean score for the control group (n = 38, mean = 3.19). This implies that students found Security Injections 2.0 more engaging than Security Injections 1.0 (see Fig. 6a.). This leads us to accept H1 and supports research question Q2. In addition, higher engagement persisted across gender (see Fig. 6b.) and race (see Fig. 6c.).

Fig. 6.
figure 7figure 7

(a) Average student engagement score in treatment and control group (b) Average student engagement score between males and females in treatment and control group (c) Average student engagement score between ethnic groups in treatment and control group

Full size image

Comparing results for individual survey questions between control and treatment groups.

The student engagement mean score for the treatment group was found to be higher than the control group (see Table 2). In particular, the scores for Q6 and Q8 were found to be statistically significant at 95 % level (refer to Table 1 for survey questions).

Table 2. Results of individual survey questions
Full size table

5 Instructor Feedback

Security Injections 2.0 modules were introduced to over 50 faculty members in workshops conducted last year. In the past year, five instructors have used the beta system for both in-class and online instruction to approximately 180 students. Introduction of the modules at the workshops has led to overwhelmingly positive response among instructors who look forward to incorporating the 2.0 modules in their curriculum, primarily due to the auto-grading functionality. One workshop attendee indicated, “I am excited about the interactive modules (injection 2.0). I will be more likely to incorporate these modules.” An instructor who used both versions of Security Injections modules in a class of 30 students in summer 2014 commented, “The students were able to access the materials easily, and seemed to enjoy the interactive modules more. They were also able to get feedback faster, and seemed more comfortable with it compared to the other version.” In spring 2015, approximately 20 instructors are using 2.0 modules. Security Injections 2.0 modules are developed on the principles of learning theory and with the intention of facilitating wide-spread instructor adoption. Initial feedback indicates that segmented and instant-feedback based modules are likely to increase instructor adoption and student interest, which supports our initial result and third research question.

6 Conclusion

In this study, we designed a system of enhanced secure coding learning modules that implement the principles of segmentation; by breaking content from 1.0 into individual sections per screen with checkpoint questions, and interactivity; by including instant-feedback. We conducted a study to test their effectiveness towards increasing student engagement across four sections of CS0 using the post only control group design. In addition, we collected instructor feedback.

Results from the study showed the segmented and interactive 2.0 modules led to significant increase in student engagement. Additionally, higher engagement persisted across gender and race. Feedback from instructors indicates both higher student and instructor interest. In addition, the increase in use of enhanced modules by instructors indicated higher instructor adoption.

Due to the limited instruments available to assess whether segmentation leads to less skipping and skimming, we plan an observational study in fall 2015 to record student behavior using segmented and interactive modules.

In addition, in future work, we plan a full scale experiment across multiple sections and courses, to examine the effectiveness of the enhanced modules, towards our goal of increasing secure coding knowledge among computer science students.