Is MOOC Learning Different for Dropouts? A Visually-Driven, Multi-granularity Explanatory ML Approach
- 205 Downloads
Millions of people have enrolled and enrol (especially in the Covid-19 pandemic world) in MOOCs. However, the retention rate of learners is notoriously low. The majority of the research work on this issue focuses on predicting the dropout rate, but very few use explainable learning patterns as part of this analysis. However, visual representation of learning patterns could provide deeper insights into learners’ behaviour across different courses, whilst numerical analyses can – and arguably, should – be used to confirm the latter. Thus, this paper proposes and compares different granularity visualisations for learning patterns (based on clickstream data) for both course completers and non-completers. In the large-scale MOOCs we analysed, across various domains, our fine-grained, fish-eye visualisation approach showed that non-completers are more likely to jump forward in their learning sessions, often on a ‘catch-up’ path, whilst completers exhibit linear behaviour. For coarser, bird-eye granularity visualisation, we observed learners’ transition between types of learning activity, obtaining typed transition graphs. The results, backed up by statistical significance analysis and machine learning, provide insights for course instructors to maintain engagement of learners by adapting the course design to not just ‘dry’ predicted values, but explainable, visually viable paths extracted.
KeywordsLearning analytics Visualisation MOOCs Behavioural pattern Machine learning
How can learning paths be visualised to differentiate between completers’ and non-completers’ behaviour (to inform teachers for early interventions)?
Are there (significant) differences in the learning paths of completers and non-completers and can they be deduced from visualisation early on in the course?
What kind of level of granularity is necessary for the visualisation of significant differences between completers and non-completers?
Providing new insights into early learning behaviour exhibited by course completers and non-completers through bird-eye and fish-eye visualisation of partial or full learning graphs, with different levels of information disclosure.
Proposing visual graph analysis as a pre-step to machine learning and prediction, here illustrated by discovering linear or catch-up behaviours, which then can be reliable predicted.
Showing that theme-based visualisation (which also can be at bird-eye or fish-eye level) detects other relations in the course, e.g. the effect of forums.
2 Related Work
Existing studies on MOOCs analytics mostly focus on finding reliable completion indicators from learner behaviour patterns, using data of forum activities , clickstreams , participants’ time spent and number of accesses , assignment activities  to name a few. Other predictive studies [12, 13, 14] attempt to forecast the performance, including final grade and pass/fail in exams. Overall, existing research usually does not disregard the potential in using visualisation as an initial step before prediction, nor do they consider the granularity of visualisation as a factor.
The explanatory power of visualisation has been stated to be crucial to provide more insights for module instructors and designers, next to completion prediction . A learning path is defined as the learning trajectory through a course by learners; according to , participants generally study web courses in a non-linear manner. We are specifically interested in comparing the transition behaviour between completers and non-completers, as suggested by .  investigated the typical learning activity sequences across two MOOC datasets and mined the difference in learning themes among groups with different grades, by visualising the theme distribution. However, they mainly focused on which topics were more popular, instead of visualising the entire learning paths of their four groups of learners (none, fail, pass and distinction). Later,  visualised log traces of learners across four edX MOOC datasets, using discrete-time Markov Models and observed that learners were more likely to jump forward than backwards from video content. Additionally, they found that non-passing learners were more likely to exhibit binge video watching, i.e., transit from one video to another without answering questions, deviating from the designed linear learning path. However, they only visualised the video interaction activity of passing and non-passing learners, instead of the whole learning sessions, like in our work. Recently,  used visualisation software, Gephi, to visualise clickstream-based learning paths. They observed that learners are more likely to skip the quizzes at the end of each chapter, by watching the beginning videos of the next chapter, but learn linearly within chapters. However, neither did such previous studies explore the phenomenon in-depth, nor provided a convincing explanation. This paper validates that completers behave differently from dropout learners, by visualising the entire learning paths of participants. We also implement machine learning models, to statistically analyse students’ movements, by combining courses’ themes and content.
2.2 Statistical Analysis
According to , statistical analysis can be divided into descriptive statistics, to summarise the demographics information of learners and inference statistics, to explore behaviours exhibited by participants. For instance,  firstly examined if there is any impact on the behaviour of learners, after they reached the passing state. They examine weekly quiz score distribution for all learners by K-means clustering, which showed that early passers obtained relatively lower scores immediately after passing, compared with their previous performance. Later,  investigated the motivation of two groups of completers: university students and general participants, using Mann–Whitney U test, and concluded that as participants’ ages increased, earning a certificate was less significant. Recently,  explored different behaviours of course completers and non-completers from a discourse perspective on course content review, via a chi-square test. They found that completers were more likely to post original and less negative opinions, whilst non-completers were more willing to reply to and criticise others’ posts. Inspired by their work, we implement a Wilcoxon signed-rank test for completers and non–completers to learn their transition behaviours among different learning themes, separately. Additionally, we also apply two machine learning algorithms for early completion prediction, based on our analysis of two groups of student activities during the first week.
To explore if completers and non-completers behave differently, we apply visualisation analytics firstly, to identify the different learning paths executed by these groups and then implements statistical modelling to analyse their learning behaviour (transition from themes such as video, quiz, discussion, review and article) across different courses. Additionally, by comparing dropout learners’ transition from different learning activities across different courses, the paper offers insights into the impact of the designed course learning path on the learning behaviour of participants.
The dataset of learner activities
Course Title & Run
Accessed 1 step
Babies in Mind
Shakespeare and His World
The Mind is Flat
Of all participants, 50230 learners have accessed at least one step of the course, but over half, 53%, have never accessed the course after registration. The 42024 learners who accessed less than 80% are defined as dropout learners. The 8206 learners who accessed 80% or more of the materials in one run are defined as completers. The 80% threshold (as opposed to, e.g., 100% completion) is based on prior literature discussing different ways of computing completion . Completers represent 7% of the total and 16% of the learners who accessed at least once.
3.2 Visualisation of High and Low Granularity Levels
The size of the circle represents the number of learners who accessed the course content and the thickness of the arrow shows the percentage of learners’ movements.
3.3 Statistical Analysis and Machine Learning
We applied the Shapiro–Wilk test for normality check. For non-parametric data, we applied the Wilcoxon signed-rank test for significance measurement of differences between groups. We report results as percentages, instead of a total number of learners, to eliminate the effects of having different numbers of learners for each course. To explore more on learners’ engagement with different learning themes, we also build an early dropout prediction model based on the time spent on each activity, by ensemble machine learning methods: Gradient Boosting  and XGBoost .
4 Results and Discussion
P-values of linear and catch-up learning activities
Babies in Mind Run 1
1.13E−13 (p < 0.001)
2.46E−85 (p < 0.001)
Babies in Mind Run 2
7.74E−14 (p < 0.001)
2.32E−62 (p < 0.001)
Big Data Run 1
2.66E−018 (p < 0.001)
5.97E−110 (p < 0.001)
Big Data Run 2
1.35E−68 (p < 0.001)
2.66E−18 (p < 0.001)
Shakespeare Run 1
1.130E−13 (p < 0.001)
2.09E−23 (p < 0.001)
Shakespeare Run 2
7.73E−14 (p < 0.001)
1.87E−09 (p < 0.001)
Mind is Flat Run 1
2.66E−018 (p < 0.001)
5.21E−74 (p < 0.001)
Mind is Flat Run 2
6.62E−63 (p < 0.001)
2.51E−16 (p < 0.001)
Early prediction (in first ten percentage of course) of dropout
The mind is flat
Babies in Mind
The most significant component, 25.9% and 24.12% of the participants continue from videos to videos and switch from videos to articles in the Babies in Mind and Big Data courses, respectively. The transition from videos to articles is 24.12% in the Big Data course, i.e., nearly one-quarter of the dropout learners lose interests after reading papers. Therefore, course designers should consider replacing some readings with interactive activities, to improve engagement. Interestingly, 17.1% dropouts transfer from quizzes to quizzes, which could be inefficient attempts to finish quickly. Such behaviour can inform instructors to intervene early.
5 Conclusion and Limitations
The paper visualises and compares different learning paths of completers and non-completers across four MOOCs, and explores from which learning theme learners tend to drop out. We have shown how different granularity visualisation (fish eye, bird eye) allows both researchers, and potentially teachers, to understand where issues occur and where patterns emerge, backed up by statistical analysis. Specifically, we have shown that course completers are more likely to learn linearly, while the dropout learners are more likely to jump forward to a later activity, which we dubbed here as “catch-up” learning pattern. Additionally, we find that video learning pattern is a powerful indicator for early prediction of dropout for learners, validated by two machine learning models, XG Boosting and Gradient Boosting. Our research has shown the benefits of theme-based visualisation, based on which we have identified that non-completers are more likely to watch videos and skip quizzes, shown by their transition from other themes, to videos. Moreover, we show how this type of analysis can generate fine-grained ideas for instructors and course designers; e.g., to improve retention, instructors (and online course designers) should introduce more discussion support mechanisms.
Limitations include clickstream analysis not entirely reflecting the true engagement of learners with learning activities (as learners may, e.g., click the complete button too soon). Additionally, we only consider the first access, due to our data limitations, which may underestimate jumping patterns of dropout learners. Other learning features could be considered, e.g., length of activity accessed.
- 1.Jordan, K.: MOOC Completion Rates: The Data (2013). Retrieved 27th July 2013, http://www.katyjordan.com/MOOCproject.html. Accessed 01 Feb 2020
- 2.Onah, D.F., Sinclair, J., Boyatt, R.: Dropout rates of massive open online courses: behavioural patterns. In: EDULEARN 2014 Proceedings, vol. 1, pp. 5825–5834 (2014)Google Scholar
- 3.Fei, M., Yeung, D.Y.: Temporal models for predicting student dropout in massive open online courses. In: 2015 IEEE International Conference on Data Mining Workshop (ICDMW), pp. 256–263. IEEE (2015)Google Scholar
- 4.Crossley, S., Paquette, L., Dascalu, M., McNamara, D.S., Baker, R.S.: Combining click-stream data with NLP tools to better understand MOOC completion. In: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, pp. 6–14 (2016)Google Scholar
- 6.Nagrecha, S., Dillon, J.Z., Chawla, N.V.: MOOC dropout prediction: lessons learned from making pipelines interpretable. In: Proceedings of the 26th International Conference on World Wide Web Companion, pp. 351–359 (2017)Google Scholar
- 7.Davis, D., Chen, G., Hauff, C., Houben, G.J.: Gauging MOOC Learners’ Adherence to the Designed Learning Path. International Educational Data Mining Society (2016)Google Scholar
- 8.Santos, J.L., Klerkx, J., Duval, E., Gago, D., Rodríguez, L.: Success, activity and drop-outs in MOOCs an exploratory study on the UNED COMA courses. In: Proceedings of the Fourth International Conference on Learning Analytics and Knowledge, pp. 98–102. ACM (2014)Google Scholar
- 9.Kloft, M., Stiehler, F., Zheng, Z., Pinkwart, N.: Predicting MOOC dropout over weeks using machine learning methods. In: Proceedings of the EMNLP 2014 Workshop on Analysis of Large Scale Social Interaction in MOOCs, pp. 60–65 (2014)Google Scholar
- 11.Coffrin, C., Corrin, L., de Barba, P., Kennedy, G.: Visualizing patterns of student engagement and performance in MOOCs. In: Proceedings of the Fourth International Conference on Learning Analytics and Knowledge, pp. 83–92. ACM (2014)Google Scholar
- 14.Li, X., Wang, T., Wang, H.: Exploring N-gram features in clickstream data for MOOC learning achievement prediction. In: Bao, Z., Trajcevski, G., Chang, L., Hua, W. (eds.) DASFAA 2017. LNCS, vol. 10179, pp. 328–339. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-55705-2_26CrossRefGoogle Scholar
- 15.Guo, P.J., Reinecke, K.: Demographic differences in how students navigate through MOOCs. In: Proceedings of the First ACM Conference on Learning@ Scale Conference, pp. 21–30. ACM (2014)Google Scholar
- 16.Jiang, S., Williams, A., Schenke, K., Warschauer, M., O’dowd, D.: Predicting MOOC performance with week 1 behaviour. In: Educational Data Mining 2014 (2014)Google Scholar
- 17.Wen, M., Rosé, C.P.: Identifying latent study habits by mining learner behaviour patterns in massive open online courses. In: Proceedings of the 23rd ACM International Conference on Information and Knowledge Management, pp. 1983–1986. ACM (2014)Google Scholar
- 18.Gao, M., Zhang, J.J., Sun, D., Zhang, J.: Using clickstream to understand learning paths and the network structure of learning resources: using MOOC as an example (2019)Google Scholar
- 19.Zheng, Y., Yin, B.: Big data analytics in MOOCs. In: 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing, pp. 681–686. IEEE (2015)Google Scholar
- 20.Zhao, Y., Davis, D., Chen, G., Lofi, C., Hauff, C., Houben, G.J.: Certificate achievement unlocked: how does MOOC learners’ behaviour change? In: Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization, pp. 83–88 (2017)Google Scholar
- 23.Cristea, A.I., Alamri, A., Kayama, M., Stewart, C., Alsheri, M., Shi, L.: Earliest predictor of dropout in MOOCs: a longitudinal study of FutureLearn courses. Association for Information Systems (2018)Google Scholar
- 24.Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat., 1189–1232 (2001)Google Scholar
- 25.Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794. ACM (2016)Google Scholar
- 26.Warren, J., Rixner, S., Greiner, J., Wong, S.: Facilitating human interaction in an online programming course. In: Proceedings of SIGCSE 2014, pp. 665–670. ACM Press, New York (2014)Google Scholar