Designing for Quality?
There are significant complexities in interpreting and measuring quality in MOOCs. In this chapter, we examine experts’ perceptions of how to measure quality in MOOCs, using empirical data we gathered through conversations with MOOC specialists. In their experience, while data can be helpful in understanding quality, the metrics measured are shaped by underpinning assumptions and biases. In conventional education, it is assumed that the learner wants to follow a course pathway and complete a course. However, this assumption may not be valid in a MOOC. Quality data might not capture the underlying goals and intentions of MOOC learners. Therefore, it is difficult to measure whether or not a learner has achieved his or her goals. We stress the need to explore quality metrics from the learner’s point of view and to encompass the variability in motivations, needs and backgrounds, which shape conceptions of quality for individuals.
The authors wish to thank Vasudha Chaudhari of The Open University for comments and for proofing this chapter.
- Adamopoulos, A. (2013). What makes a great MOOC? An interdisciplinary analysis of student retention in online courses. Paper presented at the Thirty-Fourth International Conference on Information Systems, Milan, Italy. Retrieved from http://pages.stern.nyu.edu/~padamopo/What%20makes%20a%20great%20MOOC.pdf.
- Alario-Hoyos, C., Perez-Sanagustin, M., Cormier, D., & Delgado-Kloos, C. (2014). Proposal for a conceptual framework for educators to describe and design MOOCs. Journal of Universal Computer Science, 20(1), 6–23.Google Scholar
- Amo, D. (2013, November). MOOCs: Experimental approaches for quality in pedagogical and design fundamentals. Paper presented at TEEM’13, Salamanca, Spain.Google Scholar
- Bayne, S., & Ross, J. (2014). MOOC pedagogy. In P. Kim (Ed.), Massive open online courses: The MOOC revolution (pp. 23–45). New York, NY: Routledge.Google Scholar
- Chandrasekaran, M., Ragupathi, K., Kan, M., & Tan, B. (2015, December). Towards feasible instructor intervention in MOOC discussion forums. Paper presented at the Thirty-Sixth International Conference on Information Systems, Fort Worth, TX.Google Scholar
- Coetzee, D., Lim, S., Fox, A., Hartmann, B., & Hearst, M. A. (2015). Structuring interactions for large-scale synchronous peer learning. In Proceedings of the 18th ACM Conference on Computer- Supported Cooperative Work and Social Computing (CSCW), Vancouver, Canada (pp. 1139–1152). New York, NY: ACM.Google Scholar
- Conole, G. (2013). MOOCs as disruptive technologies: Strategies for enhancing the learner experience and quality of MOOCs. RED—Revista de Educación a Distancia, 39. Retrieved from http://www.um.es/ead/red/39/conole.pdf.
- Daradoumis, T., Bassi, R., Xhafa, F., & Caballé, S. (2013, October). A review on massive e-learning (MOOC) design, delivery and assessment. In 2013 Eighth International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC) (pp. 208–213). Piscataway, NJ: IEEE.Google Scholar
- Downes, S. (2013, April 24). The quality of massive open online courses. Retrieved from http://mooc.efquel.org/files/2013/05/week2-The-quality-of-massive-open-online-courses-StephenDownes.pdf.
- Fenwick, T. (2015). Sociomateriality and learning: A critical approach. In D. Scott & E. Hargreaves (Eds.), The SAGE handbook of learning (online). London: SAGE.Google Scholar
- Gibbs, G. (2010). Dimensions of quality. York, UK: The Higher Education Academy.Google Scholar
- Grover, S., Franz, P., Schneider, E., & Pea, R. (2013). The MOOC as distributed intelligence: Dimension of a framework an evaluation of MOOCs. Paper presented at the 10th Annual International Conference on Computer Supported Collaborative Learning, Madison, WI. Retrieved from http://life-slc.org/docs/LSLC_rp_A194_Grover-etal_CSCL2013_MOOCsand-DI_Volume%202_CSCL2013.pdf.
- Grunewald, F., Meinel, C., Totschnig, M., & Willems, C. (2013). Designing MOOCs for the support of multiple learning styles. In Conference Proceedings from EC-TEL 2013, LNCS (pp. 371–382). Berlin, Germany: Springer.Google Scholar
- Guardia, L., Maina, M., & Sangra, A. (2013). MOOC design principles: A pedagogical approach from the learner’s perspective. eLearning Papers, 33, 1–5.Google Scholar
- Guo, P., Kim, J., & Rubin, R. (2014). How video production affects student engagement: An empirical study of MOOC videos. In Proceedings of the First ACM conference on Learning @ Scale Conference (pp. 41–50). New York, NY: ACM.Google Scholar
- Hood, N., & Littlejohn, A. (2016a). MOOC Quality: A call for new quality measures. Journal of Learning for Development, 3(3), 28–42 https://t.co/EePAUPWnDb.
- Hood, N., & Littlejohn, A. (2016b). Quality in MOOCs, surveying the terrain (Commonwealth for Learning Report). http://oasis.col.org/handle/11599/2352.
- Howley, I., Mayfield, E., & Rosé, C. P. (2013). Linguistic analysis methods for studying small groups. In C. Hmelo-Silver, A. O’Donnell, C. Chan, & C. Chin (Eds.), The international handbook of collaborative learning (pp. 184–202). New York, NY: Routledge.Google Scholar
- Illeris, K. (2007). How we learn: Learning and non-learning in school and beyond. London: Routledge.Google Scholar
- Istrate, O., & Kestens, A. (2015, April). Developing and monitoring a MOOC: The IFRC experience. Paper presented at the 11th International Scientific Conference eLearning and Software for Education, Bucharest, Romania. Retrieved from http://www.academia.edu/14707457/DEVELOPING_AND_MONITORING_A_MOOC_THE_IFRC_EXPERIENCE.
- Kanwar, A. (2013, October 16). Quality vs. quantity: Can technology help? Opening keynote presentation at the 25th ICDE World Conference, Tianjin, China.Google Scholar
- Kizilcec, R., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. LAK’13 Leuven, Belgium.Google Scholar
- Kolowich, S. (2013, March 18). The professors behind the MOOC hype. The Chronicle of Higher Education. Retrieved from http://chronicle.com/article/The-Professors-Behind-the-MOOC/137905/.
- Lackner, E., Ebner, M., & Khalil, M. (2015). MOOCs as granular systems: Design patterns to foster participant activity. eLearning Papers, 42, 28–37.Google Scholar
- Mak, S., Williams, R., & Mackness, J. (2010). Blogs and forums as communication and learning tools in a MOOC. In L. Dirckinck-Holmfeld, V. Hodgson, C. Jones, M. de Laat, D. McConnell, & T. Ryberg (Eds.), In Proceedings of the 7th International Conference on Networked Learning 2010 (pp. 275–284). Lancaster, UK: Lancaster University. Retrieved from https:// www.lancaster.ac.uk/fss/organisations/netlc/past/nlc2010/abstracts/PDFs/Mak.pdf.
- Mamgain, N., Sharma, A., & Goyal, P. (2014). Learner’s perspective on video-viewing features offered by MOOC providers: Coursera and edX. Paper presented at the 2014 IEEE International Conference on MOOC, Innovation and Technology in Education (MITE). https://doi.org/10.1109/mite.2014.7020298.
- Morozov, E. (2014). The planning machine. The New Yorker, 13 October (www.newyorker.com/magazine/2014/10/13/planning-machine.
- Morrison, D. (2014, January 18). Need-to-know-news: MOOC mentors for hire, Coursera’s MOC$s, edX shares MOOC data and more. Online Learning Insights. Retrieved from https://onlinelearninginsights.wordpress.com/2014/01/28/need-to-know-news-mooc-mentors-forhire-courseras-mocs-edx-shares-mooc-data-and-more/.
- Rodriguez, C. (2012). MOOCs and the AI-Stanford like courses: Two successful and distinct course formats for massive open online courses. European Journal of Open, Distance and E-Learning. Retrieved from http://files.eric.ed.gov/fulltext/EJ982976.pdf.
- Ross, J., Sinclair, C., Knox, J., & Macleod, H. (2014). Teacher experiences and academic identity: The missing components of MOOC pedagogy. Journal of Online Learning and Teaching, 10(1), 57.Google Scholar
- Scagnoli, N. (2012). Thoughts on instructional design for MOOCs. Retrieved from https://www.ideals.illinois.edu/bitstream/handle/2142/44835/Instructional%20Design%20of%20a%20MOOC.pdf.
- Selwyn, N. (2014). Distrusting educational technology. London and New York: Routledge.Google Scholar
- Selwyn, N. (2016). Is technology good for education. Cambridge, UK: Polity Books.Google Scholar
- Shah, D. (2016). By the numbers: MOOCS in 2016. [Online]. Retrieved from: https://www.class-central.com/report/mooc-stats-2016/.
- Sinha, T., Li, N., Jermann, P., & Dillenbourg, P. (2014). Capturing “attrition intensifying” structural traits from didactic interaction sequences of MOOC learners. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Workshop on Modeling Large Scale Social Interaction in Massively Open Online Courses (pp. 42–49). Retrieved from https://www.aclweb.org/anthology/W/W14/W14-41.pdf.
- Tabba, Y., & Medouri, A. (2013). LASyM: A learning analytics system for MOOCs. International Journal of Advanced Computer Science and Applications, 4(5), 113–119.Google Scholar
- Wen, M., Yang, D., & Rosé, C. P. (2014a). Linguistic reflections of student engagement in massive open online courses. In Proceedings of the International Conference on Weblogs and Social Media. Retrieved from http://www.cs.cmu.edu/~mwen/papers/icwsm2014-camera-ready.pdf.
- Wen, M., Yang, D., & Rosé, C. P. (2014b). Sentiment analysis in MOOC discussion forums: What does it tell us? In Proceedings of Educational Data Mining. Retrieved from http://www.cs.cmu.edu/~mwen/papers/edm2014-camera-ready.pdf.
- Yang, D., Wen, M., Kumar, A., Xing, E., & Rosé, C. (2014). Towards an integration of text and graph clustering methods as a lens for studying social interaction in MOOCs. International Review of Research in Open and Distributed Learning, 15(5). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1853/3083.