Dynamic Analytics for the Detection of Quality and Alignment Issues in an Online Hybrid Adaptive Instructional and Assessment System

Abstract

In a system that combines elements of adaptive instruction and assessment, misalignment at the intersection of instruction and assessment and assessment quality issues can result in poor learner experience and faulty system recommendations and prescriptions. This paper contains elements relevant for the detection and remediation of such issues in an online hybrid instructional and assessment system. The first section presents motivation of this research, and the advantage we can expect when we address misalignment and quality issues. The second section describes an exemplar hybrid adaptive instruction and assessment system which includes the following components: adaptive instruction, practice, mastery quiz, progress check, and adaptive diagnostic assessments. The potential for and impact of alignment and quality issues in the progress check, which measures students’ skill mastery for an adaptive loop, is described. The third section proposes psychometric methods that may be appropriate for the detection of misalignment and quality. The fourth section includes an empirical example of the proposed method with data from a production hybrid system, comments on the perceived utility of the procedure to date, outlines a workflow used to implement the processes from data analysis to dynamic result reporting, and presents a view of a dashboard used to increase collaboration among psychometric and content experts.