Keywords

1 Introduction

Learning by teaching has been generally accepted one of the best ways for students to learn for centuries. As of this writing, Google Scholar lists over 7,030 entries for a search on this topic, which is 1,390 more than in two years ago when our first article about Flip-Flop was published [1]. A yearly increase of almost 10% in scholarly publications clearly shows that the interest in this arena continues unabated and there is and will be increased demand for practical applications that support this idea. Many sources [2,3,4,5,6] discuss the various pros and cons of learning by teaching: students benefit not only in quantity of learning in terms of the amount of knowledge gained, but also in its quality, i.e. the degree to which they understand the material, when they are required to learn from teaching and being taught by their peers.

As the theory of learning by teaching proposed the concepts of Active Learning [7, 8] and Constructivist Learning [9, 10], these approaches have also seen a widespread acceptance and integration into curricula. Both of these methodologies prioritize actively involving students in their own education rather than passively absorbing lectures. These concepts in general and our approach in particular satisfy most of the levels in all the dimensions of the Bloom’s Taxonomy of Educational Objectives [11].

In the traditional model of instruction, students attend lectures in person while working on assignments at home. The Flipped Classroom [8, 12] builds on these concepts and reverses this model: students watch lecture videos before class sessions, then work on exercises and ask questions in class where the instructor is available to assist them. The use of video makes the flipped model particularly useful for Massive Open Online Courses (MOOC) [13,14,15,16,17] as this allows for students to watch and listen to lectures from around the globe. Our Flip-Flop methodology builds on and expands the theory of learning methodology and provides the underlying software framework and online tools that support both the learners and the instructors.

As our earlier article [1] described in detail, the Flip-Flop approach lets learners construct quizzes synchronized with educational screencasts and take peer-constructed quizzes in order to gain a deeper understanding of the topics explained in the screencast and more objectively assess whether they understand the content. The quiz tasks consist of questions, correct and incorrect answers, feedback to each of the answers, hints, and hint links. In order to create such a quiz task, the student needs to construct a question that is related to the specific portion of the video and with answers whose correctness is not too easy to guess or too hard to understand. While the feedbacks for answers are optional, we encourage the quiz authors to explain why the correct answer is indeed correct and why the wrong answers are incorrect, typically indicating the likely misconceptions. In addition, formulating a concise hint is also challenging as the author should only point the quiz taker in the right direction without giving away the entire correct answer. The hint link also challenges the author to find the most appropriate web page online that explains the topic well enough to answer the corresponding question.

Since we offer a variety of modes to synchronize tasks with the video, the author can simply opt for limiting the response time by the duration of the corresponding video segment, or choose to pause the video and give the quiz taker a specific time limit to choose an answer. He or she can also choose whether to show the correct answer even if an incorrect answer was selected.

In addition to multiple-choice questions, we support poll tasks – where there is no distinction between correct and incorrect answers – as well as “pinboard tasks” where the author displays text or an image and optionally a link to an external web page and the quiz-taker does not need to take any action except possibly to click on the link to view such a page. Since the author of the quiz can specify the maximum points per task and whether the number of possible points decreases with time, correctly answered questions increase the total score for this quiz and consequently motivates and rewards students for positive performance. In the context of inverted learning it is important that the students are likely to become more involved with the topics presented in the screencast while creating and taking peer quizzes, and to understand the topics covered in the screencast better at home before they come to the classroom.

According to the most recent Fall 2017 report by Phil Hill [18], 87% of institutions of higher learning and 91% of student enrollments rely upon one of the learning management systems (LMS): Blackboard [19], Canvas [20], Moodle [21], or D2L Brightspace [22]. None of these popular school-wide ‘big four’ LMS’s offers quiz editing features for the students, records the authored quizzes, or allows taking peer’s quizzes. While the name Quizlet [23] - another recent commercial LMS - seems to indicate that quizzes are its core essence, it mainly supports creating study plans, scheduling study sessions, or taking short quizzes to measure progress, and does not for creating peer quizzes. Moreover, Quizlet does not target educational institutions, and is used almost exclusively in the language learning community. Quiz It!, another interesting framework that recently won “Best Education Hack” as well as “Best Google Cloud Platform Hack” prizes [24] attempts to create quizzes automatically using artificial intelligence to analyze the underlying resources. While quiz-making is offered by some based on video lectures – e.g. Kahoot [25], none of these platforms features the all the feedback, hints and hint links components as our platform. In particular, the recently added Peer Improvement component discussed below is not available in any of the aforementioned environments.

Since the tightly structured and systematic approach of Flip-Flop does not explicitly fit other well-documented and researched educational methodologies, we propose the term constructive education for approaches and technologies that require the students to construct teaching materials based on and synchronized with recordings of educational lessons.

Since we are convinced the recent additions to the Flip-Flop methodology, its concepts, and tools deserve most of the attention from the reader, let us take the unusual approach of highlighting these improvements first and then follow up with the general description of the preexisting concepts and tools in Sect. 3. This way, the readers familiar with our introductory article [1] do not need to reread the descriptions they are already familiar with. If the generic Flip-Flop approach is new for you, however, you may opt to defer reading the next section after you read Sect. 3.

2 New Developments and New Methodology

In addition to the approach discussed in our previous article [1], we have added several enhancements. Please refer to Sect. 3 for the preexisting Flip-Flop concepts and tools.

2.1 Peer Improvement

Most importantly, we have now augmented the Flip-Flop technology with another methodology component that we call Peer Improvement. Once the student has finished taking the quiz, he or she enters an editing suite where he or she can page through every task of the quiz just taken and suggest an improvement to every task component. For instance, clicking on the question for the task displays an editing field where the student can change the wording or type an entirely new question text. Similarly, the student can change each of the answers and modify the feedback for each of the answers, the task’s hint text, and hint link. Figure 1 shows the editing field where the question has been reworded. Notice that the differences between the original and the edited text of the question are depicted in red and green colors.

Fig. 1.
figure 1

Peer improvement: suggested improvements (Color figure online)

Full size image

Naturally, the editing facilities do not stop at just editing the texts. Students can suggest that another answer is correct rather than the one that the author selected simply by clicking on its red “x” sign. (Obviously, a green check mark will now indicate the answer deemed correct.) In its final form, our framework will allow the student to add another answer, suggest that an answer gets removed, and add a missing feedback, missing hint, or missing hint link.

Notice that once one or more students take the quiz and suggest improvements, the next student can select any of the previous improvements (as well as the original item) and improve it again. Alternately, the student can just click on the check mark to indicate that he or she approves of that particular item. Such an approval earns this suggestion item a ‘+’. As Fig. 2 shows, the ‘+’ stars appear next to the ID of the student who suggested the improvement.

Fig. 2.
figure 2

Peer improvement: approval

Full size image

Peer Improvement has numerous advantages. The learner who just took the quiz and has its tasks fresh in memory can immediately change all the items he or she found problematic or even judges as incorrect. Note also that only one feedback - the one associated with the selected answer - is shown during the quiz taking session. In contrast, all the feedbacks are shown during the Peer Improvement session. Since we encourage the authors to provide additional information within the feedbacks such as an indication of what misconception might have led to an incorrect answer, the student can even learn more in-depth aspects of the subject. Similarly, the Peer Improvement framework presents all the hints and the hint links which were not shown during the quiz taking time unless the student clicked on the “Hint” button. Now the student can even follow the hint links and learn from these additional resources.

Peer Improvement is also likely to increase the motivation of learners. After all, improving and even suggesting additional improvements to already posted ones can be perceived as a challenging opportunity to showcase mastery of the subject. Collecting ‘+’ from the peers only adds to the motivation of the student.

Last but not least, the instructor naturally benefits from the Peer Improvement methodology as the improvements can be showcased in the order of most ‘+’ scores earned. This will make it easy to add the elements of Flip-Flop methodology to grading, to build a prioritized database of quiz tasks that can be used in future tests, and to promote the applications of online tasks discussed in the Conclusions section below.

2.2 Channels

In the previous version of our software technology, the students had to download a template of their quiz from the web site for the course, integrate it manually within their own web pages, and after they finished authoring the quiz based on this template they had to return to the web page of the course to choose an item from a menu that sent an email to the peers who needed to take this new quiz.

In the current version, each student has his or her own channel on the Flip-Flop web site. A template is now inserted for each new quiz to be authored based on the schedule of the course. Once the student has created the quiz he or she simply clicks a button that sends an email to all the peers announcing that the quiz is ready.

2.3 Peer Evaluation

Moreover, a more efficient peer evaluation component has simplified the Flip-Flop quiz making and taking component. Instead of relying on the quiz authors to include a series of peer evaluation tasks at the end of each quiz, we now automatically feature an evaluation page where the quiz taker clicks on the star rating for each of the essential quiz components as depicted in Fig. 3.

Fig. 3.
figure 3

Peer evaluation: star ratings

Full size image

3 Preexisting Flip-Flop Tools and Generic Concepts

3.1 Taking Quizzes

To take a quiz, a student may either navigate directly to a quiz through a notification email or find the quiz in the list of quizzes that peers have created. A flip-flop quiz itself looks like this (Fig. 4):

Fig. 4.
figure 4

Taking a quiz: question and answers

Full size image

If the quiz taker selects the correct answer, he or she sees feedback indicating that the choice was correct (Fig. 5):

Fig. 5.
figure 5

Taking a quiz: selecting an correct answer

Full size image

If the selected answer is incorrect, the feedback will indicate this as well, hopefully identifying the misconception that the student has made and suggesting how to correct this mistake (Fig. 6):

Fig. 6.
figure 6

Taking a quiz: selecting an incorrect answer

Full size image

Some questions may also have hints for the quiz taker to look at (Fig. 7):

Fig. 7.
figure 7

Taking a quiz: viewing a hint

Full size image

Once completed, the quiz taker may then provide his or her evaluation of the quiz.

3.2 Creating Quizzes

In order to create a quiz, a student must first log in to her channel - currently on slippah.com - and select the quiz she needs to create. The quiz authoring page looks like this (Fig. 8):

Fig. 8.
figure 8

Creating a quiz: authoring tool

Full size image

The left half of the page contains the video and segments for the quiz. The gray anchors below the video indicate the start and end points of the quiz; this particular quiz covers the second quarter of the lecture. Between those anchors, green and yellow rectangles represent individual question segments: the question will be displayed on the screen during that portion of the video. The student may change the quantity, placement, and duration of these segments to ensure that the application displays quiz questions during relevant portions of the video.

The right hand side of the page shows the currently selected question (highlighted in green below the video). In addition to the question, up to five answers, and a radio button to indicate the correct choice, the student may provide hints (the light bulb icon to the right of the question) (Fig. 9):

Fig. 9.
figure 9

Creating a quiz: editing hints

Full size image

As well as feedback for each response (the text bubble to the right of an answer) (Fig. 10):

Fig. 10.
figure 10

Creating a quiz: editing feedback

Full size image

We also provide the student with a list of symbols; the student may click on any one of these symbols to insert the symbol into his or her text (Fig. 11):

Fig. 11.
figure 11

Creating a quiz: symbols

Full size image

Any suggested improvements for the current element are also available for reference (Fig. 12):

Fig. 12.
figure 12

Creating a quiz: suggested improvements

Full size image

3.3 Instructor Perspective

Simplifying the chores of the instructors is one of the main objectives of a learning management system. Given that the Flip-Flop method needs specific scheduling facilities that are essential for assigning students quizzes to make and take based on screencasts, a substantial portion of the underlying technology had to be devoted to providing an instructor with (1) a tool for easy review of the work from students, and (2) with the scheduling tool that subdivides the students into groups and assigns the quiz authors portions of the corresponding screencasts.

After students create quizzes, their instructor can easily look through those quizzes for grading. For example, in Fig. 13 we can see the contents of the one quiz that Samantha Lewis wrote:

Fig. 13.
figure 13

Instructor perspective: viewing quizzes

Full size image

The scheduling tool helps instructors to define the student groups, determine the who is the quiz author for a particular segment of a screencast and when to generate the corresponding quiz templates, add them to the channel of each student, and remind the author once the template is up as well when his or her quiz is due (Fig. 14):

Fig. 14.
figure 14

Instructor perspective: creating quizzes

Full size image

4 Conclusions and Future Work

4.1 Can It Take Less Time?

In our experience, while the students typically see and appreciate the benefits of the Flip-Flop approach, their main objection is that they need to invest more time than in other classes. Indeed, creating a quiz is - at least initially - a challenging task in particular when the student is confronted with it for the first time in his or her career. Since we are now collecting data about how long it takes an author to create a quiz, we will be able to determine how much time it took at the beginning and compare it with the times at the end of the semester. (Naturally, we expect that the latter times will be shorter not only because an increased familiarity with the technology itself, but mainly because of better quiz-making skills). If the Flip-Flop approach is applied across courses during several consecutive courses, we would also expect fewer objections and more appreciation of the acquired benefits.

Our technology, however, can help shorten the time it takes the student to take a quiz. For instance, we can provide a button that skips the remaining portion of the screencast segment once an answer was selected and start playing the video at the beginning of the next task segment. Alternately, we could add a scale that allows the quiz taker to speed up the screencast. Another feasible approach would be to let the author who creates a quiz from a segment of one screencast to take only quizzes from built from other screencasts so that he or she does not need to view any portion of a particular screencast twice. While such speedups may please the students, they may be less appreciated by the instructors - after all, viewing a lecture twice certainly reinforces the understanding of the subject. Therefore we are inclined to make them optional within the scheduling tool and let the instructor decide whether they will be available to the students in his or her course.

4.2 Additional Question Types

Our quizzes are currently limited to multiple-choice questions, polls, and pinboard tasks that consist solely of text or images without user input. Although this has been sufficient for the quizzes until now, we envision having a wider range of questions for students to choose from.

Since the questions for Flip-Flop quizzes have been predominantly multiple-choice thus far, one next step might be to support the selection of multiple answers. For example, the author of a quiz might instruct his or her peers to select all of the options that are correct or to draw lines between the pairs of matching answers.

Multiple-choice questions limit students to a finite number of answers, and while this makes grading the quiz easier it also makes answering the question a matter of recognizing the correct choice rather than recalling or deducing the correct answer. Allowing quiz authors to write short response or essay questions would compel students to generate their own answers rather than selecting from a list of options. Furthermore, the author of the quiz would have to grade the responses to these types of questions, providing the student with the perspectives of his or her peers and requiring the author to discern whether each written answer is correct or incorrect.

Many domains are more graphically oriented and an answer or feedback purely in text would take too much space on the screen. Our application currently supports images in answers, but only as a thumbnail that is displayed next to the text. Improving support for graphical answers would allow students to click on pictures of answers rather than text, increasing the versatility of the software.

On the other hand, some fields are heavily text-driven or do not have lecture videos readily available for use. Although these Flip-Flop quizzes are primarily designed for video quizzes, this system could be adapted to display a document on the left side of the window while questions and answers appear on the right as students scroll through the text.

4.3 Improvements to Peer Improvement

Our software currently allows quiz takers to suggest modifications to the text of questions, answers, feedback, and hints. However, reviewers should also have the option to propose adding or removing quiz components as well. For example, if the author did not write feedback for an answer, the student who just took the quiz should be able to provide the author with some text that the author could eventually use when updating or revising the quiz.

Furthermore, quiz takers should be able to write comments accompanying their suggested modifications. This will allow reviewers to not just propose changes but explain why they would make those changes.

4.4 Agile Tooltip

We plan to use another novel technology to assist the students while they are constructing and taking quizzes so that they can easily seek targeted help as well as provide feedback about the Flip-Flop technology itself, the quizzes they take or make, and the screencasts themselves. We call this technology “Agile Tooltip”, as its roots are in one of the main concepts that the Agile Methodology in Software Engineering has pioneered: involving the customer in continuously defining and perfecting the software product rather than trying to make exact requirement specifications up front. The Agile Tooltip concept goes one step further in involving the end user in this process. It adds two buttons to every tooltip: A “help” button, typically represented by a “?” question mark icon, and a “feedback” button, typically depicted as a “thought bubble” icon. Both of these buttons either lead to a page or displays a dialogue where the user can find help information that is directly related to the widget: for instance, an explanation of how this particular widget is used within a typical workflow.

Naturally, in the context of Flip-Flop the “help” button of the Agile Tooltip will lead to the corresponding help pages. For example, the tooltip related to the “Link” entry in the authoring system will lead to the help page that describes how the student should search for resources on the web that are related to a question, how to select and copy the web page link from the address bar of the browser, and how to paste it into the entry field. The advantage of this approach is not just that the help is targeted to the purpose of the widget, but also that the help pages can provide additional semantic information: for instance, suggesting that the linked webpage should not completely answer the question but only help with deriving the correct answer.

The “feedback” button of the Flip-Flop’s Agile Tooltip will display a suggestion page or dialogue where the student will be able to choose the type of feedback he or she is providing, the entry field for the suggestion text along with a drag and drop pane where he or she can submit a screenshot, and an entry field for his or her email. We intend to support at least the following types of feedbacks:

  1. 1.

    Problems, errors, and suggestions how to improve the functionality and appearance of the Flip-Flop user experience,

  2. 2.

    Problems understanding the help pages and suggestions how to improve them,

  3. 3.

    Problems with the quiz the student is currently taking, and

  4. 4.

    Problems understanding the screencast itself and suggestions how it could be improved.

Once the student submits a feedback we will record and categorize the suggestions. The student will receive an automatic “thank you” email. Furthermore, the Agile Tooltip system will allow the instructor and/or teaching assistant to view and respond to the specific suggestion, for instance, promising to alert the student once the problem was corrected or the suggestion addressed. We also consider sending such replies to all the students in the course to demonstrate that the instructors and developers do care and thus encourage them to suggest more improvements on their own.

The detailed description of the Agile Tooltip methodology, the corresponding software framework and the support app, as well as its general applications will be subject of subsequent research article we plan to submit to a software engineering journal.

5 The Vision

The Flip-Flop quizzes have several important aspects:

  • They are largely subject-independent,

  • They are also grade-independent,

  • They are language independent, and

  • They can be easily translated into various other languages

Most importantly, a vast number of such quizzes can be stored online, associated and linked to other online resources. If Flip-Flop and future similar technologies were widely adopted and enhanced, a vast reservoir of testing resources will become available.

In particular, an abundance of online quizzes can potentially change how we perceive testing.

As most educators know, when taking a test the primary motivation for students is to get the minimum sufficient number of points to achieve a desired grade. They expect that their tests will be corrected by a knowledgeable - if not infallible - instructor and that once they took the test they will not be bothered with taking another test where this knowledge will be required - except maybe on a final test that covers an entire course. So for most students, taking a test is mainly about the question “Did I get enough points?” rather than “Did I learn and understand the topic?” In an ideal world, we are all learners who should be easily able to find out whether we understand a subject in a book, an article in a journal, or even a topic explained in a Wikipedia web page. If each article of an encyclopedia topic had an accompanying online quiz that could be taken online without any adverse consequences - such as a low score or time limit - then our attitude towards tests could be dramatically changed and we could start viewing tests positively as instruments that help us to objectively assess our knowledge and learning.

There are other advantages that are likely to result from the availability of online quizzes for an arbitrary subject at an arbitrary time. If there is no time limit on how long a learner can take to complete a quiz, then the lack of stress is likely to result in better scores and more satisfactory experience. On the other hand, if a learner can opt for time limitation, the increased challenge might also prove to benefit the learning effect. Moreover, online quizzes can become a competitive adventure where learners attempt to achieve better and better scores and compete. After all, the abundance of online trivia quizzes (currently 5,350,000 search results on Google) proves that testing can be very entertaining.

Furthermore, online quizzes can point the learner to resources that are the most appropriate for his or her current level of knowledge and learning style. For instance, if a learner performs miserably on a quiz, he or she can be guided to a learning resource that is less advanced or covers the topic in simpler terms. Moreover, quizzes may be constructed in different ways, such as using pictures rather than text. If a student performs better on the pictorial type of quizzes, he or she can be guided towards the visually rich presentations of a topic. Similarly, students who answer textual questions fast may learn better from a more abstract description of the subject.

Ideally, learners will perceive a test not as a threat but as a positive or even entertaining experience that helps them to learn a subject and even assess and improve their own learning progress, learning speed, and learning skills.

Abundance of online quizzes does not benefit only the learners. Making quizzes is one of the most tedious and time consuming chores of an instructor. Coming up with a new set of quizzes with new tasks every semester is not easy. Selecting from numerous quizzes even if they are not constructed by experts but by students and possibly just improving on their wording is bound to substantially shorten the time necessary for the instructor to create quizzes. Furthermore, since such improvements are stored and classified as employed or improved by an expert, the quality of the quizzes and tasks will undoubtedly improve gradually.

Last but not least, an abundance of online quizzes might solve one of the increasingly pressing educational quandaries: how to address copying and prevent plagiarism in a time when most of the resources, exercises, problems and their solutions are accessible online. If our students were willing to take dozens of quizzes to prepare for a test, we should be confident that they have learned the subject well enough. Given this perspective, we can argue that the more quiz tasks and problem solutions are uploaded to the internet the less important it will be to penalize students for copying. Plagiarism may become a historically interesting misdemeanor rather than an educational felony.