Students, Teachers, Exams and MOOCs: Predicting and Optimizing Attainment in Web-Based Education Using a Probabilistic Graphical Model

  • Bar Shalem
  • Yoram Bachrach
  • John Guiver
  • Christopher M. Bishop
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8726)

Abstract

We propose a probabilistic graphical model for predicting student attainment in web-based education. We empirically evaluate our model on a crowdsourced dataset with students and teachers; Teachers prepared lessons on various topics. Students read lessons by various teachers and then solved a multiple choice exam. Our model gets input data regarding past interactions between students and teachers and past student attainment. It then estimates abilities of students, competence of teachers and difficulty of questions, and predicts future student outcomes. We show that our model’s predictions are more accurate than heuristic approaches. We also show how demographic profiles and personality traits correlate with student performance in this task. Finally, given a limited pool of teachers, we propose an approach for using information from our model to maximize the number of students passing an exam of a given difficulty, by optimally assigning teachers to students. We evaluate the potential impact of our optimization approach using a simulation based on our dataset, showing an improvement in the overall performance.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Anastasi, A., Urbina, S., et al.: Psychological testing. Prentice Hall, Upper Saddle River (1997)Google Scholar
  2. 2.
    Anderson, T.: The theory and practice of online learning. Au Press (2008)Google Scholar
  3. 3.
    Archak, N.: Money, glory and cheap talk: analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on topcoder. com. In: Proceedings of the 19th International Conference on World Wide Web, pp. 21–30. ACM (2010)Google Scholar
  4. 4.
    Bachrach, Y., Graepel, T., Minka, T., Guiver, J.: How to grade a test without knowing the answers—a bayesian graphical model for adaptive crowdsourcing and aptitude testing. In: ICML (2012)Google Scholar
  5. 5.
    Bachrach, Y., Graepel, T., Kasneci, G., Kosinski, M., Van Gael, J.: Crowd iq: aggregating opinions to boost performance. In: AAMAS (2012)Google Scholar
  6. 6.
    Bachrach, Y., Graepel, T., Kohli, P., Kosinski, M., Stillwell, D.: Your digital image: factors behind demographic and psychometric predictions from social network profiles. In: AAMAS (2014)Google Scholar
  7. 7.
    Bachrach, Y., Kosinski, M., Graepel, T., Kohli, P., Stillwell, D.: Personality and patterns of facebook usage. In: ACM WebSci (2012)Google Scholar
  8. 8.
    Bachrach, Y., Rosenschein, J.S.: Distributed multiagent resource allocation in diminishing marginal return domains. In: AAMAS (2008)Google Scholar
  9. 9.
    Barrick, M.R., Mount, M.K.: The big five personality dimensions and job performance: a meta-analysis. Personnel Psychology 44(1), 1–26 (2006)CrossRefGoogle Scholar
  10. 10.
    Brabazon, T.: Digital hemlock: Internet education and the poisoning of teaching. University of New South Wales Press (2002)Google Scholar
  11. 11.
    Brusilovsky, P., et al.: Adaptive and intelligent technologies for web-based eduction. KI 13(4), 19–25 (1999)Google Scholar
  12. 12.
    Clearwater, S.H.: Market-based control: A paradigm for distributed resource allocation (1996)Google Scholar
  13. 13.
    Costa Jr., P.T., McCrae, R.R.: Neo personality inventory–revised (neo-pi-r) and neo five-factor inventory (neo-ffi) professional manual. Psychological Assessment Resources, Odessa (1992)Google Scholar
  14. 14.
    Darling-Hammond, L.: Evaluating teacher effectiveness: How teacher performance assessments can measure and improve teaching (2010)Google Scholar
  15. 15.
    De Raad, B., Schouwenburg, H.C.: Personality in learning and education: A review. European Journal of Personality 10(5), 303–336 (1998)CrossRefGoogle Scholar
  16. 16.
    Downes, S.: The rise of moocs. Stephens Web (2012)Google Scholar
  17. 17.
    Dumais, S.A.: Cultural capital, gender, and school success: The role of habitus. Sociology of Education, 44–68 (2002)Google Scholar
  18. 18.
    Eichler, H.-G., Kong, S.X., Gerth, W.C., Mavros, P., Jönsson, B.: Use of cost-effectiveness analysis in health-care resource allocation decision-making: How are cost-effectiveness thresholds expected to emerge? Value in health 7(5), 518–528 (2004)CrossRefGoogle Scholar
  19. 19.
    Engle, J., Tinto, V.: Moving beyond access: College success for low-income, first-generation students. Pell Institute for the Study of Opportunity in Higher Education (2008)Google Scholar
  20. 20.
    Gao, X.A., Bachrach, Y., Key, P., Graepel, T.: Quality expectation-variance tradeoffs in crowdsourcing contests. In: AAAI (2012)Google Scholar
  21. 21.
    Getoor, L., Friedman, N., Koller, D., Pfeffer, A., Taskar, B.: 5 probabilistic relational models. Statistical Relational Learning, 129 (2007)Google Scholar
  22. 22.
    Gosling, S.D., Rentfrow, P.J., Swann, W.B.: A very brief measure of the big-five personality domains. Journal of Research in Personality 37(6), 504–528 (2003)CrossRefGoogle Scholar
  23. 23.
    Glazerman, S., Loeb, S., Goldhaber, D., Staiger, D., Raudenbush, S., Whitehurst, G.: Evaluating teachers: The important role of value-added. Brookings Institution (2010)Google Scholar
  24. 24.
    Hambleton, R.K., Swaminathan, H., Rogers, H.J.: Fundamentals of item response theory, vol. 2 (1991)Google Scholar
  25. 25.
    Hara, N.: Student distress in a web-based distance education course. Information, Communication & Society 3(4), 557–579 (2000)CrossRefMathSciNetGoogle Scholar
  26. 26.
    Harasim, L.: Shift happens: Online education as a new paradigm in learning. The Internet and Higher Education 3(1), 41–61 (2000)CrossRefGoogle Scholar
  27. 27.
    Howe, J.: The rise of crowdsourcing. Wired Magazine 14(6), 1–4 (2006)MathSciNetGoogle Scholar
  28. 28.
    Khan, B.H.: Web-based instruction. Prentice Hall (1997)Google Scholar
  29. 29.
    Koller, D., Friedman, N.: Probabilistic Graphical Models: Principles and Techniques (2009)Google Scholar
  30. 30.
    Kosinski, M., Bachrach, Y., Kasneci, G., Van-Gael, J., Graepel, T.: Crowd iq: Measuring the intelligence of crowdsourcing platforms. In: ACM WebSci (2012)Google Scholar
  31. 31.
    Kosinski, M., Bachrach, Y., Kohli, P., Stillwell, D., Graepel, T.: Manifestations of user personality in website choice and behaviour on online social networks. Machine Learning 95(3), 357–380 (2014)CrossRefMATHMathSciNetGoogle Scholar
  32. 32.
    Kosinski, M., Stillwell, D., Graepel, T.: Private traits and attributes are predictable from digital records of human behavior. PNAS (2013)Google Scholar
  33. 33.
    Kosinski, M., Stillwell, D., Kohli, P., Bachrach, Y., Graepel, T.: Personality and website choice (2012)Google Scholar
  34. 34.
    Lavy, V.: Evaluating the effect of teachers group performance incentives on pupil achievement. Journal of Political Economy 110(6), 1286–1317 (2002)CrossRefGoogle Scholar
  35. 35.
    Lev, O., Polukarov, M., Bachrach, Y., Rosenschein, J.S.: Mergers and collusion in all-pay auctions and crowdsourcing contests. In: AAMAS (2013)Google Scholar
  36. 36.
    Mackness, J., Mak, S., Williams, R.: The ideals and reality of participating in a MOOC. In: Networked Learing Conference (2010)Google Scholar
  37. 37.
    Minka, T., Winn, J.M., Guiver, J.P., Knowles, D.A.: Infer.NET 2.4 (2010)Google Scholar
  38. 38.
    Minka, T., Winn, J.: Gates. In: NIPS, vol. 21 (2008)Google Scholar
  39. 39.
    Minka, T.P.: A family of algorithms for approximate Bayesian inference. PhD thesis (2001)Google Scholar
  40. 40.
    Moore, M.G., Kearsley, G.: Distance education: A systems view of online learning. Wadsworth Publishing Company (2011)Google Scholar
  41. 41.
    Palloff, R.M., Pratt, K.: Lessons from the cyberspace classroom: The realities of online teaching. Wiley. Com (2002)Google Scholar
  42. 42.
    Pearl, J.: Probabilistic reasoning in intelligent systems: networks of plausible inference (1988)Google Scholar
  43. 43.
    Picciano, A.G.: Beyond student perceptions: Issues of interaction, presence, and performance in an online course. Journal of Asynchronous Learning Networks 6(1), 21–40 (2002)Google Scholar
  44. 44.
    Ridgell, S.D., Lounsbury, J.W.: Predicting academic success: General intelligence,big five personality traits, and work drive. College Student Journal 38(4), 607–618 (2004)Google Scholar
  45. 45.
    Russell, M.T., Karol, D.L.: Institute for Personality, and Ability Testing. In: The 16PF Fifth Edition Administrator’s Manual, Institute for Personality and Ability Testing, Champaign (1994)Google Scholar
  46. 46.
    Salek, M., Bachrach, Y., Key, P.: Hotspotting – a probabilistic graphical model for image object localization through crowdsourcing. In: AAAI (2013)Google Scholar
  47. 47.
    Schochet, P.Z., Chiang, H.S.: Error rates in measuring teacher and school performance based on student test score gains. ncee 2010-4004. National Center for Education Evaluation and Regional Assistance (2010)Google Scholar
  48. 48.
    Scott, J.: Family, gender, and educational attainment in britain: A longitudinal study. Journal of Comparative Family Studies 35(4), 565–590 (2004)Google Scholar
  49. 49.
    Sprinthall, R.C., Fisk, S.T.: Basic statistical analysis. Prentice Hall, Englewood Cliffs (1990)Google Scholar
  50. 50.
    Welinder, P., Branson, S., Belongie, S., Perona, P.: The multidimensional wisdom of crowds. In: NIPS (2010)Google Scholar
  51. 51.
    West, D.B., et al.: Introduction to graph theory, vol. 2. Prentice Hall, Upper Saddle River (2001)Google Scholar
  52. 52.
    Witkowski, J., Bachrach, Y., Key, P., Parkes, D.C.: Dwelling on the negative: Incentivizing effort in peer prediction. In: HCOMP (2013)Google Scholar
  53. 53.
    Yorke, M., Thomas, L.: Improving the retention of students from lower socio-economic groups. Journal of Higher Education Policy and Management 25(1), 63–74 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Bar Shalem
    • 1
  • Yoram Bachrach
    • 2
  • John Guiver
    • 2
  • Christopher M. Bishop
    • 2
  1. 1.Bar-Ilan UniversityRamat GanIsrael
  2. 2.Microsoft ResearchCambridgeUK

Personalised recommendations