# Managing items and knowledge components: domain modeling in practice

- 29 Downloads

## Abstract

Adaptive learning systems need large pools of examples for practice—thousands of items that need to be organized into hundreds of knowledge components within a domain model. Domain modeling and closely related student modeling are intensively studied in research studies. However, there is a gap between research studies and practical issues faced by developers of scalable educational technologies. The aim of this paper is to bridge this gap by connecting techniques and notions used in research papers to practical problems in development. We put specific emphasis on scalability—on techniques that enable relatively cheap and fast development of adaptive learning systems. We summarize conceptual questions in domain modeling, provide an overview of approaches in the research literature, and discuss insights based on the development and analysis of a widely used system. We conclude with recommendations for both developers and researchers in the area of adaptive learning systems.

## Keywords

Domain modeling Student modeling Adaptivity Scalability Knowledge component System development## Notes

### Acknowledgements

The author thanks Petr Jarušek, the chief developer of Umíme systems, for long-term development and thorough discussions, on which this paper is based. The author also thanks members of the Adaptive Learning group at Masaryk University for their insights and feedback.

### Compliance with ethical standards

### Conflict of interest

The author declares that he has no conflict of interest.

## References

- Abyaa, A., Idrissi, M. K., & Bennani, S. (2019). Learner modelling: Systematic review of the literature from the last 5 years.
*Educational Technology Research and Development*. https://doi.org/10.1007/s11423-018-09644-1. - Agarwal, M. & Mannem, P. (2011). Automatic gap-fill question generation from text books. In
*Proceedings of innovative use of NLP for building educational applications*(pp. 56–64). Association for Computational Linguistics.Google Scholar - Al-Yahya, M., George, R., & Alfaries, A. (2015). Ontologies in e-learning: Review of the literature.
*International Journal of Software Engineering and Its Applications*,*9*(2), 67–84.Google Scholar - Aleven, V., McLaughlin, E. A., Glenn, R. A., & Koedinger, K. R. (2016).
*Handbook of research on learning and instruction, chapter Instruction based on adaptive learning technologies*. New York: Routledge.Google Scholar - Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., et al. (2000).
*A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives*(abridged ed.). London: Pearson.Google Scholar - Attali, Y. (2018). Automatic item generation unleashed: An evaluation of a large-scale deployment of item models. In
*Proceedings of artificial intelligence in education*(pp. 17–29). New York: Springer.Google Scholar - Ayers, E. & Junker, B. (2006). Do skills combine additively to predict task difficulty in eighth grade mathematics. In
*Proceedings of educational data mining*.Google Scholar - Baker, R. S. (2016). Stupid tutoring systems, intelligent humans.
*International Journal of Artificial Intelligence in Education*,*26*(2), 600–614.CrossRefGoogle Scholar - Barnes, T. (2005). The q-matrix method: Mining student response data for knowledge. In
*Proceedings of American association for artificial intelligence 2005 educational data mining workshop*(pp. 1–8).Google Scholar - Beck, J. E., Pardos, Z. A., Heffernan, N. T., & Ruiz, C. (2008). The composition effect: Conjunctive or compensatory? An analysis of multi-skill math questions in its.
*Proceedings of Educational Data Mining*,*2008*, 147–156.Google Scholar - Bieliková, M., Šimko, M., Barla, M., Tvarožek, J., Labaj, M., Móro, R., et al. (2014). ALEF: From application to platform for adaptive collaborative learning. In N. Manouselis, H. Drachsler, K. Verbert, & O. Santos (Eds.),
*Recommender systems for technology enhanced learning*(pp. 195–225). New York: Springer.CrossRefGoogle Scholar - Bloom, B. S., Engelhart, M. B., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956).
*Taxonomy of educational objectives. The classification of educational goals. Handbook 1: Cognitive domain*. New York: Longmans Green.Google Scholar - Brusilovsky, P. (1998). Methods and techniques of adaptive hypermedia. In
*Proceedings of adaptive hypertext and hypermedia*(pp 1–43). New York: Springer.Google Scholar - Bull, S., & Kay, J. (2007). Student models that invite the learner in: The smili: Open learner modelling framework.
*International Journal of Artificial Intelligence in Education*,*17*(2), 89–120.Google Scholar - Carmona, C., Millán, E., Pérez-de-la Cruz, J.-L., Trella, M., & Conejo, R. (2005). Introducing prerequisite relations in a multi-layered bayesian student model. In
*Proceedings of user modeling*(pp. 347–356). New York: Springer.Google Scholar - Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning factors analysis–a general method for cognitive model evaluation and improvement. In
*Proceedings of intelligent tutoring systems*(pp. 164–175). New York: Springer.Google Scholar - Cen, H., Koedinger, K. R., & Junker, B. (2007). Is over practice necessary? Improving learning efficiency with the cognitive tutor through educational data mining.
*Proceedings of Artificial Intelligence in Education*,*158*, 511–518.Google Scholar - Cen, H., Koedinger, K. R., & Junker, B. (2008). Comparing two irt models for conjunctive skills. In
*Proceedings of intelligent tutoring systems*(pp. 796–798). New York: Springer.Google Scholar - Churchill, D. (2007). Towards a useful classification of learning objects.
*Educational Technology Research and Development*,*55*(5), 479–497.CrossRefGoogle Scholar - Clark, R. E., Feldon, D., van Merrienboer, J., Yates, K., & Early, S. (2007).
*Handbook of research on educational communications and technology*., Cognitive task analysis Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar - Conati, C., Gertner, A., & Vanlehn, K. (2002). Using bayesian networks to manage uncertainty in student modeling.
*User Modeling and User–Adapted Interaction*,*12*(4), 371–417.CrossRefGoogle Scholar - Corbett, A. T., & Anderson, J. R. (1994). Knowledge tracing: Modeling the acquisition of procedural knowledge.
*User Modeling and User–Adapted Interaction*,*4*(4), 253–278.CrossRefGoogle Scholar - De Ayala, R. J. (2013).
*The theory and practice of item response theory*. New York: Guilford Publications.Google Scholar - Desmarais, M. (2011). Conditions for effectively deriving a q-matrix from data with non-negative matrix factorization. In
*Proceedings of Educational Data Mining*(pp. 41–50).Google Scholar - Dicheva, D., Dichev, C., Agre, G., & Angelova, G. (2015). Gamification in education: A systematic mapping study.
*Journal of Educational Technology & Society*,*18*(3), 75–88.Google Scholar - Doignon, J.-P., & Falmagne, J.-C. (2012).
*Knowledge spaces*. New York: Springer.Google Scholar - Feng, M., Heffernan, N., Mani, M., & Heffernan, C. (2006). Using mixed-effects modeling to compare different grain-sized skill models.
*Proceedings of Educational Data Mining*,*2*, 79–92.Google Scholar - Gagné, R. (1985).
*The conditions of learning and theory of instruction*. New York: Holt, Rinehart and Winston.Google Scholar - Gong, Y., Beck, J. E., & Heffernan, N. T. (2010). Comparing knowledge tracing and performance factor analysis by using multiple model fitting procedures. In
*Proceedings of intelligent tutoring systems*(pp. 35–44). New York: Springer.Google Scholar - González-Brenes, J., Huang, Y., & Brusilovsky, P. (2014). General features in knowledge tracing: Applications to multiple subskills, temporal item response theory, and expert knowledge. In
*Proceedings of educational data mining*(pp. 84–91).Google Scholar - Heffernan, N. T. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension. In
*Proceedings of the nineteenth annual conference of the cognitive science society*(pp. 307–312).Google Scholar - Honebein, P. C., & Honebein, C. H. (2015). Effectiveness, efficiency, and appeal: Pick any two? the influence of learning domains and learning outcomes on designer judgments of useful instructional methods.
*Educational Technology Research and Development*,*63*(6), 937–955.CrossRefGoogle Scholar - Hosseini, R., & Brusilovsky, P. (2017). A study of concept-based similarity approaches for recommending program examples.
*New Review of Hypermedia and Multimedia*,*23*(3), 161–188.CrossRefGoogle Scholar - Huang, Y., Guerra-Hollstein, J. D., & Brusilovsky, P. (2016). Modeling skill combination patterns for deeper knowledge tracing. In
*Proceedings of user modeling adaptation and personalization (extended proceedings)*.Google Scholar - Käser, T., Busetto, A. G., Solenthaler, B., Baschera, G.-M., Kohn, J., Kucian, K., et al. (2013). Modelling and optimizing mathematics learning in children.
*International Journal of Artificial Intelligence in Education*,*23*(1–4), 115–135.CrossRefGoogle Scholar - Käser, T., Klingler, S., Schwing, A. G., & Gross, M. (2014). Beyond knowledge tracing: Modeling skill topologies with Bayesian networks. In
*Proceedings of intelligent tutoring systems*(pp. 188–198).Google Scholar - Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012). The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning.
*Cognitive Science*,*36*(5), 757–798.CrossRefGoogle Scholar - Koedinger, K. R., & McLaughlin, E. A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In
*Proceedings of the annual meeting of the cognitive science society*. Austin, TX: Cognitive Science Society.Google Scholar - Koedinger, K. R., & McLaughlin, E. A. (2016). Closing the loop with quantitative cognitive task analysis. In
*Proceedings of educational data mining*(pp. 412–417).Google Scholar - Koedinger, K. R., Pavlik, P. I, Jr., Stamper, J. C., Nixon, T., & Ritter, S. (2011). Avoiding problem selection thrashing with conjunctive knowledge tracing. In
*Proceedings of educational data mining*(pp. 91–100).Google Scholar - Koedinger, K. R., Stamper, J. C., McLaughlin, E. A., & Nixon, T. (2013). Using data-driven discovery of better student models to improve student learning. In
*Proceedings of artificial intelligence in education*(pp. 421–430). New York: Springer.Google Scholar - Koedinger, K. R., Yudelson, M. V., & Pavlik, P. I. (2016). Testing theories of transfer using error rate learning curves.
*Topics in Cognitive Science*,*8*(3), 589–609.CrossRefGoogle Scholar - Little, J. L., Bjork, E. L., Bjork, R. A., & Angello, G. (2012). Multiple-choice tests exonerated, at least of some charges: Fostering test-induced learning and avoiding test-induced forgetting.
*Psychological Science*,*23*(11), 1337–1344.CrossRefGoogle Scholar - Liu, R., Koedinger, K. R., & McLaughlin, E. A. (2014). Interpreting model discovery and testing generalization to a new dataset. In
*Proceedings of educational data mining*(pp. 107–113).Google Scholar - McNee, S. M., Kapoor, N., & Konstan, J. A. (2006). Don’t look stupid: avoiding pitfalls when recommending research papers. In
*Proceedings of computer supported cooperative work*(pp. 171–180). New York: ACM.Google Scholar - Millán, E., Loboda, T., & Pérez-de-la Cruz, J. L. (2010). Bayesian networks for student model engineering.
*Computers & Education*,*55*(4), 1663–1683.CrossRefGoogle Scholar - Molenaar, I. & Knoop-van Campen, C. (2017). Teacher dashboards in practice: Usage and impact. In
*Proceedings of European conference on technology enhanced learning*(pp. 125–138). New York: Springer.Google Scholar - Nkambou, R., Mizoguchi, R., & Bourdeau, J. (2010).
*Advances in intelligent tutoring systems*(Vol. 308). New York: Springer.CrossRefGoogle Scholar - Pardos, Z. A. & Heffernan, N. T. (2011). KT-IDEM: Introducing item difficulty to the knowledge tracing model. In
*Proceedings of user modeling, adaption and personalization*(pp. 243–254). New York: Springer.Google Scholar - Pardos, Z. A., Heffernan, N. T., Anderson, B., Heffernan, C. L., & Schools, W. P. (2010).
*Handbook of educational data mining. Using fine-grained skill models to fit student performance with Bayesian networks*(pp. 417–426). Boca Raton, FL: Chapman & Hall/CRC PressGoogle Scholar - Pelánek, R. (2016). Applications of the Elo rating system in adaptive educational systems.
*Computers & Education*,*98*, 169–179.CrossRefGoogle Scholar - Pelánek, R. (2017). Bayesian knowledge tracing, logistic models, and beyond: An overview of learner modeling techniques.
*User Modeling and User–Adapted Interaction*,*27*(3), 313–350.CrossRefGoogle Scholar - Pelánek, R. (2018). The details matter: Methodological nuances in the evaluation of student models.
*User Modeling and User–Adapted Interaction*,*28*, 207–235.CrossRefGoogle Scholar - Pelánek, R. (2019). Measuring similarity of educational items: An overview.
*IEEE Transactions on Learning Technologies*.Google Scholar - Pelánek, R., Papoušek, J., Řihák, J., Stanislav, V., & Nižnan, J. (2017). Elo-based learner modeling for the adaptive practice of facts.
*User Modeling and User-Adapted Interaction*,*27*(1), 89–118.CrossRefGoogle Scholar - Pelánek, R. & Řihák, J. (2016). Properties and applications of wrong answers in online educational systems. In
*Proceedings of educational data mining*.Google Scholar - Pelánek, R., & Řihák, J. (2018). Analysis and design of mastery learning criteria.
*New Review of Hypermedia and Multimedia*,*24*, 133–159.CrossRefGoogle Scholar - Porter, A. C. (2002). Measuring the content of instruction: Uses in research and practice.
*Educational Researcher*,*31*(7), 3–14.CrossRefGoogle Scholar - Rau, M. A., Aleven, V., & Rummel, N. (2010). Blocked versus interleaved practice with multiple representations in an intelligent tutoring system for fractions. In
*Proceedings of intelligent tutoring systems*(pp. 413–422). New York: Springer.Google Scholar - Reigeluth, C. M., & Carr-Chellman, A. A. (2009a). Situational principles of instruction. In C. M. Reigeluth & A. A. Carr-Chellman (Eds.),
*Instructional-design theories and models. Building a common knowledge base*(Vol. III, pp. 57–68). New York: Routledge.CrossRefGoogle Scholar - Reigeluth, C. M., & Carr-Chellman, A. A. (2009b). Understanding instructional theory. In C. M. Reigeluth & A. A. Carr-Chellman (Eds.),
*Instructional-design theories and models. Building a common knowledge base*(Vol. III, pp. 3–26). New York: Routledge.CrossRefGoogle Scholar - Reigeluth, C. M., & Keller, J. B. (2009). Understanding instruction. In C. M. Reigeluth & A. A. Carr-Chellman (Eds.),
*Instructional-design theories and models. Building a common knowledge base*(Vol. III, pp. 27–39). New York: Routledge.CrossRefGoogle Scholar - Reigeluth, C. M., Merrill, M. D., & Bunderson, C. V. (1978). The structure of subject matter content and its instructional design implications.
*Instructional Science*,*7*(2), 107–126.CrossRefGoogle Scholar - Reigeluth, C. M., Merrill, M. D., Wilson, B. G., & Spiller, R. T. (1980). The elaboration theory of instruction: A model for sequencing and synthesizing instruction.
*Instructional Science*,*9*(3), 195–219.CrossRefGoogle Scholar - Řihák, J., & Pelánek, R. (2017). Measuring similarity of educational items using data on learners’ performance.
*Proceedings of educational data mining*(pp. 16–23).Google Scholar - Roediger, H. L., & Pyc, M. A. (2012). Inexpensive techniques to improve education: Applying cognitive psychology to enhance educational practice.
*Journal of Applied Research in Memory and Cognition*,*1*(4), 242–248.CrossRefGoogle Scholar - Sottilare, R. A., Graesser, A. C., Hu, X., Olney, A., Nye, B., & Sinatra, A. M. (2016).
*Design recommendations for intelligent tutoring systems. Domain modeling*(Vol. 4). Adelphi: US Army Research Laboratory.Google Scholar - Stillson, H., & Alsup, J. (2003). Smart ALEKS... or not? teaching basic algebra using an online interactive learning system.
*Mathematics and Computer Education*,*37*(3), 329–340.Google Scholar - Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design.
*Learning and Instruction*,*4*(4), 295–312.CrossRefGoogle Scholar - Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory.
*Journal of Educational Measurement*,*20*(4), 345–354.CrossRefGoogle Scholar - Taylor, K., & Rohrer, D. (2010). The effects of interleaved practice.
*Applied Cognitive Psychology*,*24*(6), 837–848.CrossRefGoogle Scholar - Wang, Y., Heffernan, N. T., & Heffernan, C. (2015). Towards better affect detectors: effect of missing skills, class features and common wrong answers. In
*Proceedings of learning analytics and knowledge*(pp. 31–35). New York: ACM.Google Scholar - Wang, Z., Lan, A. S., Nie, W., Waters, A. E., Grimaldi, P. J., & Baraniuk, R. G. (2018). QG-net: a data-driven question generation model for educational content. In
*Proceedings of learning at scale*(pp. 7:1–7:10). New York: ACM.Google Scholar - Xu, Y., & Mostow, J. (2012). Comparison of methods to trace multiple subskills: Is LR-DBN best?
*Proceedings of educational data mining*(pp. 41–48).Google Scholar