Abstract
Learner models are representations of the learner’s knowledge, skills and other attributes used by Adaptive Instructional Systems (AISs) to personalize their interactions with the learners (e.g., by implementing adaptive feedback, and recommending tasks/activities). Top-down and bottom-up approaches to learner modeling provide various affordances and challenges in terms of the need for interpretable learner models, the amount of data available, the complexity of the model, and the amount of human effort needed to implement and validate learner models. Research shows that hybrid approaches involving both top-down and bottom-up approaches are needed to effectively deal with the challenges of learner modeling in AISs. This paper describes several learner modeling approaches for integrating top-down and bottom-up approaches to gather additional evidence for supporting assessment claims and implementing personalization approaches. We elaborate on several learner modeling issues, including (a) evidence identification and aggregation in assessment systems; (b) making sense of process data aimed at supporting assessment claims related to learner cognition; and (c) approaches for improving interpretability and explainability of student models with some implications for validity and fairness of AISs.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abyaa, A., Khalidi Idrissi, M., Bennani, S.: Learner modelling: systematic review of the literature from the last 5 years. Educ. Tech. Research Dev. 67(5), 1105–1143 (2019). https://doi.org/10.1007/s11423-018-09644-1
Almond, R.G., Zapata-Rivera, J.-D.: Bayesian Networks. In: von Davier, M., Lee, Y.-S. (eds.) Handbook of diagnostic classification models. MEMA, pp. 81–106. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05584-4_4
Anderson, J.R.: How can the human mind occur in the physical universe? Oxford University Press, New York (2007). https://doi.org/10.1093/acprof:oso/9780195324259.001.0001
Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive tutors: lessons learned. J. Learn. Sci. 4, 167–207 (1995)
Anderson, J.R., Lebiere, C.J.: The Atomic Components of Thought. Erlbaum, Mahwah (1998)
Arslan, B., Jiang, Y., Keehner, M., Gong, T., Katz, I.R., Yan, F.: The effect of drag-and-drop item features on test-taker performance and response strategies. Educ. Measur. Issues Pract. 39, 96–106 (2020)
Bauer, M., Zapata-Rivera, D.: Cognitive foundations of automated scoring. In: Yan, D., Rupp, A.A., Foltz, P.W. (eds.) Handbook of Automated Scoring: Theory into Practice, pp. 13–28. Taylor and Francis Group, New York (2020)
Bejar, I.I.: Threats to score meaning in automated scoring. In: Ercikan, K., Pellegrino, J.W. (eds.) Validation of Score Meaning for the Next Generation of Assessments, pp. 75–84. Routledge, New York (2017)
Bodily, R., et al.: Open learner models and learning analytics dashboards: a systematic review. In: Proceedings of the 8th International Conference on Learning Analytics and Knowledge, pp. 41–50 (2018)
Brown, J.S., Burton, R.: Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Sci. 2, 155–192 (1978)
Brown, J.S., et al.: Steps towards a theoretical foundation for complex, knowledge-based CAI. Bolt, Beranek and Newman, Cambridge (1975)
Brusilovsky, P., Karagiannidis, C., Sampson, D.: Layered evaluation of adaptive learning systems. Int. J. Continuing Eng. Educ. Lifelong Learn. 14(4–5), 402–421 (2004)
Bull, S., Brna, P., Pain, H.: Extending the scope of the student model. User Model. User-Adap. Inter. 5, 45–65 (1995)
Bull, S., Kay, J.: SMILI☺: a framework for interfaces to learning data in open learner models, learning analytics and related fields. Int. J. Artif. Intell. Educ. 26(1), 293–331 (2016)
Carr, B., Goldstein, I.: Overlays: a theory of modeling for computer-aided instruction, Technical Report, AI Lab Memo 406. MIT (1977)
Chen, H., Tan, E., Lee, Y., Praharaj, S., Specht, M., Zhao, G.: Developing AI into explanatory supporting models: An explanation-visualized deep learning prototype. In: The International Conference of Learning Science (ICLS) (2020)
Chrysafiadi, K., Virvou, M.: Student modeling approaches: a literature review for the last decade. Expert Syst. Appl. 40(11), 4715–4729 (2013)
Conati, C., Porayska-Pomsta, K., Mavrikis, M.: AI in education needs interpretable machine learning: lessons from open learner modelling. Arxiv. http://arxiv.org/abs/1807.00154 (2018)
Conati, C., Gertner, A., VanLehn, K.: Using Bayesian networks to manage uncertainly in student modeling. User Model. User-Adap. Inter. 12(4), 371–417 (2002)
Confalonieri, R., Coba, L., Wagner, B., Besold, T.R.: A historical perspective of explainable artificial intelligence. WIREs Data Min. Knowl. Disc. 11, e1391 (2021). https://doi.org/10.1002/widm.1391
Corbett, A.T., Anderson, J.R.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Model. User-Adap. Inter. 4, 253–278 (1995)
D’Mello, S., Graesser, A.: Dynamics of affective states during complex learning. Learn. Instr. 22(2), 145–157 (2012)
Durlach, P.J., Ray, J.M.: Designing adaptive instructional environments: insights from empirical evidence. Technical Report 1297. U. S. Army Research Institute for the Behavioral Social Sciences, Arlington, VA (2011)
Falmagne, J.C., Koppen, M., Villano, M., Doignon, J.P., Johannesen, L.: Introduction to knowledge spaces: how to build, test, and search them. Psychol. Rev. 97(2), 201–224 (1990)
Falmagne, J.C., Albert, D., Doble, C., Eppstein, D., Hu, X. (ed.): Knowledge Spaces: Applications in Education. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-35329-1
Forsyth, C.M., Andrews-Todd, J., Steinberg, J.: Are you really a team player? Profiles of collaborative problem solvers in an online environment. In: Rafferty, A.N., Whitehill, J., Cavalli-Sforza, V., Romero, C. (Eds.). Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020), pp. 403–408 (2020)
Geden, M., Emerson, A., Carpenter, D., Rowe, J., Azevedo, R., Lester, J.: Predictive student modeling in game-based learning environments with word embedding representations of reflection. Int. J. Artif. Intell. Educ. (2020). https://doi.org/10.1007/s40593-020-00220-4
Gisolfi, A., Dattolo, A., Balzano, W.: A fuzzy approach to student modeling. Comput. Educ. 19(4), 329–334 (1992)
Graesser, A.: Emotions are the experiential glue of learning environments in the 21st century. Learn. Instr. 70, 101212 (2020)
Greer, J., McCalla, G. (eds.): Student Models: The Key to Individualized Educational Systems. Springer, New York (1994)
Grubišić, A., Stankov, S., Žitko, B.: Stereotype student model for an adaptive e-learning system. Int. J. Comput. Electr. Autom. Control Inf. Eng. 7(4), 440–447 (2013)
Gunning, D.: “Explainable artificial intelligence (XAI)”. Defense Advanced Research Projects Agency (DARPA) (2017)
Guzmán, E., Conejo, R.: Measuring misconceptions through item response theory. In: International Conference on Artificial Intelligence in Education, pp. 608–611 (2015)
Hambleton, R.K., Swaminathan, H., Rogers, H.J.: Measurement Methods for the Social Sciences Series, Vol. 2. Fundamentals of Item Response Theory. Sage Publications, Inc. (1991)
Johnson, W., Soloway, E.: Intention-based diagnosis of programming errors. Paper presented at the AAAI (1984)
Käser, T., Klingler, S., Schwing, A.G., Gross, M.: Dynamic Bayesian networks for student modeling. IEEE Trans. Learn. Technol. 10(4), 450–462 (2017). https://doi.org/10.1109/TLT.2017.2689017
Katz, I.R., LaMar, M.M., Spain, R., Zapata-Rivera, D., Baird, J., Greiff, S.: Validity issues and concerns for technology-based performance assessments. In: Sottilare, R., Graesser, A., Hu, X., Goodwin, G. (Eds.) Design Recommendations for Intelligent Tutoring Systems: vol. 5 - Assessment Methods. U.S. Army Research Laboratory, Orlando (2017). ISBN 978–0–9893923–9–6. 209–224
Kay, J.: Stereotypes, student models and scrutability. In: International Conference on Intelligent Tutoring Systems, pp. 19–30 (2000)
Kay, J., Zapata-Rivera, D., Conati, C.: The GIFT of scrutable learner models: why and how. In: Sinatra, R.A.M., Graesser, A.C., Hu, X., Goldberg, B., Hampton, A.J. (Eds.) Design Recommendations for Intelligent Tutoring Systems: vol. 8, pp. 25–40. – Data Visualization. U.S. Army CCDC - Soldier Center, Orlando (2020)
Kerr, D., Andrews, J.J., Mislevy, R.J.: The in-task assessment framework for behavioral data. Handbook of Cognition and Assessment, pp. 472–507 (2016)
Khajah, M., Lindsey, R.V., Mozer, M.C.: How deep is knowledge tracing? In: Proceedings of Educational Data Mining, pp. 94–101 (2016)
Koedinger, K.R., McLaughlin, E.A., Stamper, J.C.: Automated student model improvement. In: Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., Stamper, J. (Eds.) Proceedings of the 5th International Conference on Educational Data Mining, Chania, Greece, pp. 17–24 (2012)
Koedinger, K.R., Stamper, J.C., McLaughlin, E.A., Nixon, T.: Using data-driven discovery of better student models to improve student learning. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education, pp. 421–430 (2013)
Lallé, S., Conati, C.: A data-driven student model to provide adaptive support during video watching across MOOCs. In: Bittencourt, I.I., Cukurova, M., Muldner, K., Luckin, R., Millán, E. (eds.) AIED 2020. LNCS (LNAI), vol. 12163, pp. 282–295. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-52237-7_23
LaMar, M.M.: Markov decision process measurement model. Psychometrika 83(1), 67–88 (2018)
Lee, M., Wagenmakers, E.: Bayesian Cognitive Modeling: A Practical Course. Cambridge University Press, Cambridge (2014). https://doi.org/10.1017/CBO9781139087759
Lehman, B., Zapata-Rivera, D.: Student emotions in conversation-based assessments. IEEE Trans. Learn. Technol. 11(1), 1–13 (2018)
Levy, R., Mislevy, R.J.: Bayesian Psychometric Modeling. CRC Press, Boca Raton (2016)
Lin, C.F., Yeh, Y.C., Hung, Y.H., Chang, R.I.: Data mining for providing a personalized learning path in creativity: an application of decision trees. Comput. Educ. 68, 199–210 (2013)
Lord, F.M., Novick, M.R.: Statistical Theories of Mental Test Scores. Addison-Welsley Publishing Company, Reading (1968)
Loukina, A., Madnani, N., Zechner, K.: The many dimensions of algorithmic fairness in educational applications. In: Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications, Florence, Italy, pp. 1–10 (2019)
MacLellan, C.J., Koedinger, K.R.: Domain-general tutor authoring with apprentice learner models. Int. J. Artif. Intell. Educ. (2020). https://doi.org/10.1007/s40593-020-00214-2
Min, W., et al.: DeepStealth: game-based learning stealth assessment with deep neural networks. IEEE Trans. Learn. Technol. 13(2), 312–325 (2020)
Mitrovic, A., Martin, B., Suraweera, P.: Intelligent tutors for all: constraint-based modeling methodology, systems and authoring. IEEE Intell. Syst. 22, 38–45 (2007)
Mislevy, R.J.: Four metaphors we need to understand assessment. Commissioned paper for The Gordon Commission on the Future of Assessment in Education (2012). Educational Testing Service, Princeton, NJ. www.ets.org/Media/Research/pdf/mislevy_four_metaphors_understand_assessment.pdf. Retrieved 28 Apr 2020
Mislevy, R.J., Almond, R.G., Lukas, J.F.: A brief introduction to evidence-centered design. ETS Res. Rep. Ser. 2003(1), i-29 (2003)
Mislevy, R.J., et al.: Psychometric considerations in game-based assessment (2014). [white paper] Retrieved from Institute of Play website. https://web.archive.org/web/20160320151604/http://www.instituteofplay.org/wp-content/uploads/2014/02/GlassLab_GBA1_WhitePaperFull.pdf (2014)
Mislevy, R.J., Riconscente, M.M.: Evidence-centered assessment design. In: Handbook of Test Development, pp. 75–104. Routledge (2011)
Nabizadeh, A.H., Gonçalves, D., Gama, S., Jorge, J., Rafsanjani, H.N.: Adaptive learning path recommender approach using auxiliary learning objects. Comput. Educ. 147, 103777 (2020)
Novick, M.R.: The axioms and principal results of classical test theory. J. Math. Psychol. 3(1), 1–18 (1966)
Ohlsson, S.: Constraint-based student modeling. J. Artif. Intell. Educ. 3(4), 429–447 (1992)
Pardos, Z.A., Heffernan, N.T.: KT-IDEM: introducing item difficulty to the knowledge tracing model. In: Konstan, J.A., Conejo, R., Marzo, J.L. Oliver, N. (Eds.) Proceedings of the 19th International Conference User Modeling, Adaption and Personalization, pp. 243–254 (2011)
Reye, J.: Student modelling based on belief networks. Int. J. Artif. Intell. Educ. 14, 63–96 (2004)
Ribeiro, M.T., Singh, S., Guestrin, C.: Why should I trust you? Explaining the predictions of any classifier. In: Proceedings of the 22nd SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)
Rich, E.: User modeling via stereotypes. Cognitive Sci. 3(4), 329–354 (1979)
Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019)
Rudin, C., Radin, J.: Why are we using black box models in AI when we don’t need to? A lesson from an explainable AI competition. Harvard Data Sci. Rev. 1(2) (2019)
Rosé, C.P., McLaughlin, E.A., Liu, R., Koedinger, K.R.: Explanatory learner models: why machine learning (alone) is not the answer. Br. J. Educ. Technol. 50(6), 2943–2958 (2019)
Seridi-Bouchelaghem, H., Sari, T., Sellami, M.: A neural network for generating adaptive lessons. J. Comput. Sci. 1(2), 232–243 (2005)
Sison, R., Shimura, M.: Student modeling and machine learning. Int. J. Artif. Intell. Educ. 9, 128–158 (1998)
Shute, V.J., Ventura, M., Bauer, M.I., Zapata-Rivera, D.: Melding the power of serious games and embedded assessment to monitor and foster learning: Flow and grow. In: Ritterfeld, U., Cody, M.J., Vorderer, P. (eds.) Serious Games: Mechanisms and Effects, pp. 295–321. Routledge, Philadelphia (2009)
Shute, V., Wang, L., Greiff, S., Zhao, W., Moore, G.: Measuring problem solving skills via stealth assessment in an engaging video game. Comput. Hum. Behav. 63, 106–117 (2016)
Shute, V.J., Zapata-Rivera, D.: Adaptive educational systems. In: Durlach, P. (ed.) Adaptive Technologies for Training and Education, pp. 7–27. Cambridge University Press, New York (2012)
Stansfield, J.C., Carr, B., Goldstein, I.P.: Wumpus advisor I: a first implementation of a program that tutors logical and probabilistic reasoning skills. At Lab Memo 381, Massachusetts Institute of Technology, Cambridge, Massachusetts (1976)
Tenison, C., Arslan, B.: Characterizing pause behaviors in a science inquiry task. In: Stewart, T.C. (Ed.). Proceedings of the 18th International Conference on Cognitive Modeling, Applied Cognitive Science Lab, Penn State, University Park, PA, pp. 283–298 (2020)
Tatsuoka, K.K.: Rule space: an approach for dealing with misconceptions based on item response theory. J. Educ. Meas. 20, 345–354 (1983)
Traub, R.: Classical test theory in historical perspective. Educ. Meas. Issues Pract. 16(4), 8–14 (1997). https://doi.org/10.1111/j.1745-3992.1997.tb00603.x
Vassileva, J., Wasson, B.: Instructional planning approaches: From tutoring towards free learning. In: Proceedings of EuroAIED 1996, Lisbon, Portugal, 30 September–2 October 1996, pp. 1–8 (1996)
Vincent-Lancrin, S., van der Vlies, R.: Trustworthy artificial intelligence (AI) in education: Promises and challenges. OECD Education Working Papers, No. 218. OECD Publishing, Paris (2020). https://doi.org/10.1787/a6c90fa9-en.
Webb, G.I., Pazzani, M.J., Billsus, D.: Machine learning for user modeling. User Model. User-Adap. Inter. 11(1), 19–29 (2001)
Yudelson, M.V., Koedinger, K.R., Gordon, G.J.: Individualized Bayesian knowledge tracing models. In: International Conference on Artificial Intelligence in Education, 171–180 (2013)
Zadeh, L.A.: Fuzzysets. Information and Control, pp. 338–353 (1965)
Zakrzewska, D.: Cluster analysis in personalized e-learning systems. In: Nguyen, N.T., Szczerbicki, E. (Eds.) Intelligent Systems for Knowledge Management, 229–250 (2009)
Zhang, M.: Contrasting automated and human scoring of essays. R & D Connections 21(2), 1–11 (2013)
Zhou, Y., Huang, C., Hu, Q., Zhu, J., Tang, Y.: Personalized learning full-path recommendation model based on LSTM neural networks. Inf. Sci. 444, 135–152 (2018)
Zapata-Rivera, D.: Supporting human inspection of adaptive instructional systems. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 482–490. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_38
Zapata-Rivera, D.: Open student modeling research and its connections to educational assessment. Int. J. Artif. Intell. Educ. (2020). https://doi.org/10.1007/s40593-020-00206-2
Zapata-Rivera, D., Brawner, K., Jackson, G.T., Katz, I.R.: Reusing evidence in assessment and intelligent tutors. In: Sottilare, R., Graesser, A., Hu, X., Goodwin, G. (Eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 5 - Assessment Methods. U.S. Army Research Laboratory, Orlando, FL (2017). ISBN 978–0–9893923–9–6. 125–136
Zapata-Rivera, D., Graesser, A., Kay, J., Hu, X., Ososky, S.: Visualization Implications for the Validity of ITS. In: Sinatra, R.A.M., Graesser, A.C., Hu, X., Goldberg, B., Hampton, A.J. (Eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 8 – Data Visualization. U.S. Army CCDC - Soldier Center, Orlando, FL, pp. 61–68 (2020)
Zapata-Rivera, D., Greer, J.: Interacting with Bayesian student models. Int. J. Artif. Intell. Educ. 14(2), 127–163 (2004)
Zapata-Rivera, D., Lehman, B., Sparks, J.R.: Learner modeling in the context of caring assessments. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2020. LNCS, vol. 12214, pp. 422–431. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50788-6_31
Zapata-Rivera, D., Liu, L., Chen, L., Hao, J., von Davier, A.A.: Assessing science inquiry skills in an immersive, conversation-based scenario. In: Kei Daniel, B. (ed.) Big Data and Learning Analytics in Higher Education, pp. 237–252. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-06520-5_14
Zapata-Rivera, D., Vanwinkle, W., Shute, V., Underwood, J., Bauer, M.: English ABLE. In: Luckin, R., Koedinger, K., Greer, J. (Eds.) Artificial Intelligence in Education - Building Technology Rich Learning Contexts that Work, vol. 158, pp. 323–330 (2007)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zapata-Rivera, D., Arslan, B. (2021). Enhancing Personalization by Integrating Top-Down and Bottom-Up Approaches to Learner Modeling. In: Sottilare, R.A., Schwarz, J. (eds) Adaptive Instructional Systems. Adaptation Strategies and Methods. HCII 2021. Lecture Notes in Computer Science(), vol 12793. Springer, Cham. https://doi.org/10.1007/978-3-030-77873-6_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-77873-6_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77872-9
Online ISBN: 978-3-030-77873-6
eBook Packages: Computer ScienceComputer Science (R0)