Abstract
This paper presents an approach to student modeling in which knowledge is represented by means of probability distributions associated to a tree of concepts. A diagnosis procedure which uses adaptive testing is part of this approach. Adaptive tests provide well-founded and accurate diagnosis thanks to the underlying probabilistic theory, i.e., the Item Response Theory. Most adaptive testing proposals are based on dichotomous models, where he student answer can only be considered either correct or incorrect. In the work described here, a polytomous model has been used, i.e., answers can be given partial credits. Thus, models are more informative and diagnosis is more efficient. This paper also presents an algorithm for estimating question characteristic curves, which are necessary in order to apply the Item Response Theory to a given domain and hence must be inferred before testing begins. Most prior estimation procedures need huge sets of data. We have modified preexisting procedures in such a way that data requirements are significantly reduced. Finally, this paper presents the results of some controlled evaluations that have been carried out in order to analyze the feasibility and advantages of this approach.
Similar content being viewed by others
References
Barbero, M.I.: Gestión informatizada de bancos de ítems. In: Olea, J., Ponsoda, V., Prieto, G. (eds.) Tests Informatizados: Fundamentos y aplicaciones, pp. 63–83. Pirámide (1999)
Birnbaum, A.: Some latent trait models and their use in inferring an examinee’s mental ability’. In: Statistical Theories of Mental Test Scores. Addison-Wesley: Reading, MA (1968)
Bock R.D. (1997) The nominal categories model. In: van der Linden W.J., Hambleton R.K., (eds) Handbook of Modern Item Response Theory. Springer, Verlag New York, pp. 33–49
Burton, R.: Diagnosing bugs in a simple procedural skill. In: Sleeman, D., Brown, J. (eds.) Intelligent Tutoring Systems, pp. 57–183. Academic Press (1982)
Carver C.A., Howard R.A., Lane W.D. (1999) Enhancing student learning through hypermedia courseware and incorporation of student learning styles. IEEE Trans. Educ. 42(1):33–38
Castillo, E., Gutiérrez, J.M., Hadi, A.S.: Expert Systems and Probabilistic Network Models. Monographs in Computer Science. Springer-Verlag (1997)
Chickering, D.M., Paek, T.: Personalizing influence diagrams: applying online learning strategies to dialogue management. User Modeling and User-Adapted Interaction (in this issue) (2007)
Collins, J.A.: Adaptive testing with granularity. Master’s thesis, University of Saskatchewan (1996)
Collins J.A., Greer J.E., Huang S.X. (1996) Adaptive assessment using granularity hierarchies and bayesian nets. In: Frasson C., Gauthier G., Lesgold A. (eds) Proceedings of the 3rd International Conference on Intelligent Tutoring Systems ITS 1996. Springer Verlag, New York, pp. 569–577, Lecture Notes in Computer Science, No. 1086
Conati C. (2002) Probabilistic assessment of user’s emotions in educational games. App. Artif. Intell. 16, 555–575
Conejo R., Guzmán E., Millán E., Trella M., Pérez de la Cruz J.L., Ríos A. (2004) SIETTE: a Web-based tool for adaptive testing. J Artif. Intell. Educ. 14, 29–61
Desmarais M.C., Pu X. (2005) a bayesian inference adaptive testing framework and its comparison with item response theory. Int. J Artif. Intell. Educ. 15, 291–323
Dodd B.G., De Ayala R.J., Koch W.R. (1995) Computerized adaptive testing with polytomous items. Appl. Psychol. Measure. 19(1):5–22
Domshlak, C., Joachmis, T.: Efficient and non-parametric reasoning over user preferences. User Model. User-adap. Interac. (in this issue) (2007)
Douglas J., Cohen A. (2001) Nonparametric item response function estimation for assessing parametric model fit. Appl. Psychol. Measure. 25(3):234–243
Embretson S.E. (1991) A multidimensional latent trait model for measuring learning and change. Psychometrika 56(3):495–515
Embretson, S.E., Reise, S.P.: Item Response Theory for Psychologists. Lawrence Erlbaum, Mahwah, NJ (2000)
Gentner D., Stevens A.L. (eds) (1983) Mental Models. Lawrence Erlbaum, Hillsdale, NJ
Greer J.E., McCalla G. (1994) Granularity-based reasoning and belief revision in student models. In: Greer J.E., McCalla G. (eds) Student Modelling: The Key to Individualized Knowledge-Based Instruction, vol. 125. Springer Verlag, New York, pp. 39–62
Guzmán, E.: Un modelo de evaluación cognitiva basado en Tests Adaptativos Informatizados para el diagnóstico en Sistemas Tutores Inteligentes. Unpublished doctoral dissertation, Dpto. Lenguajes y Ciencias de la Computación. E.T.S.I. Informática., Universidad de Málaga (2005)
Guzmán, E., Conejo, R. Simultaneous evaluation of multiple topics in SIETTE. In: Cerri, S., , G., Paraguacu, F. (eds.) Proceedings of the 6th International Conference on Intelligent Tutoring Systems (ITS 2002). Lecture Notes in Computer Science, No. 2363, pp. 739–748. Springer Verlag, New York (2002)
Guzmán E., Conejo R. (2004a) A library of templates for exercise construction in an adaptive assessment system. Technol. Instruct. Cogn. Learning (TICL) 2(1–2), 21–43
Guzmán E., Conejo R. (2005b) A model for student knowledge diagnosis through adaptive testing. In: Lester J.C., Vicari R.M., Paraguau F. (eds) Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS 2004). Springer Verlag, New York, pp. 12–21, Lecture Notes in Computer Science, No. 3220
Guzmán E., Conejo R. (2005a) Self-assessment in a feasible, adaptive web-based testing system. IEEE Trans. Educ. 48(4):688–695
Guzmán E., Conejo R. (2005b) Towards efficient item calibration in adaptive testing. In: Ardissono L., Brna P., Mitrovic A. (eds) Proceedings of the 10th International Conference on User Modeling (UM 2005). Springer Verlag, New York, pp. 414–418, Lecture Notes in Artificial Intelligence, No. 3538
Habing B. (2001) Nonparametric regression and the parametric bootstrap for local dependence assessment. Appl. Psychol. Measure. 25(3):221–233
Härdle W. (1992) Applied Nonparametric Regression. University Press, Cambridge
Holt P., Dubs S., Jones M., Greer J. (1994) The state of student modelling. In: Greer J.E., McCalla G. (eds) Student Modelling: The Key to Individualized Knowledge-Based Instruction, vol. 125. Springer Verlag, New York, pp. 3–35
Hontangas P., Ponsoda V., Olea J., Abad F. (2000) Los test adaptativos informatizados en la frontera del siglo XXI: una revisión. Metodología de las Ciencias del Comportamiento 2(2):183–216
Horvitz, E., Paek, T.: Complementary computing: policies for transferring callers from dialog systems to human receptionists. User Model. User-Adapted Interact. (in this issue) (2007)
Huang, S.X.: A content-balanced adaptive testing algorithm for computer-based training systems. In: Frasson, C., Gauthier, G., Lesgold, A. (eds.). Lecture Notes in Computer Science 1086. Proceedings of the 3rd International Conference on Intelligent Tutoring Systems. ITS 1996, pp. 306–314. Springer Verlag New York (1996)
Junker B.W., Sijtsma K. (2001) Nonparametric item response theory in action: an overview of the Special Issue. Appl. Psychol. Measure. 25(3):211–220
Kingsbury, G.G., Weiss, D.J.: An adaptive testing strategy for mastery decision. Psychometric Method Program Research Report 79-5, Department of Psychology. University of Minnesota (1979)
Lilley M., Barker T., Britton C. (2004) The development and evaluation of a software prototype for computer-adaptive testing. Comp. Educ. 43, 109–123
Lord F.M. (1980) Applications of item response theory to practical testing problems. Lawrence Erlbaum Associates, Hillsdale, NJ
Mayo M., Mitrovic A. (2001) Optimising ITS behaviour with bayesian networks and decision theory. Int. J. Artif. Intell. Educ. 12, 124–153
Melis E., Andres E., Bndenbender J., Frischauf A., Goguadze G., Libbrecht P., Pollet M., Ullrich C. (2001) ACTIVEMATH: a generic and adaptive web-based learning environment. Int. J. Artif. Intell. Educ. 12, 385–407
Millán E., Pérez de la Cruz J.L. (2002) Diagnosis algorithm for student modeling diagnosis and its evaluation. User Model. User-adapted Interact. 12(2–3):281–330
Mislevy, R.J., Steinberg, L.S., Almond, R.G.: On the roles of task model variables in assessment design. Technical Report CSE Technical Report 500, Education Testing Service (1999)
Mislevy, R.J., Steinberg, L.S., Almond, R.G.: Evidence-centered assessment design. Technical Report A Submission for the NCME Award for Technical or Scientific Contributions to the Field of Educational Measurement, Education Testing Service (2000)
Muraki E. (1992) A generalized partial credit model: Application to an EM algorithm. Appl. Psychol. Measure. 16, 159–176
Murray, T.: Formative qualitative evaluation for ‘Exploratory’ ITS research. Int. Artif. Intell. Educ.: Special Issue on Evaluation 4(2/3), 179–207 (1993)
Nanas, N., Uren, V.: Exploting term dependencies for multi-topic information filtering with single user profile. User Model. User-Adapted Interact. (in this issue) (2007)
Owen, R.J.: A Bayesian approach to tailored testing. Research Report 69-92, Educational Testing Service (1969)
Owen R.J. (1975) A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. J. Am. Stat. Associ. 70(350):351–371
Paek, T., Chickering, D.M.: Improving command and control speech recognition on mobile devices: using predictive user models for language modeling. User Model. User-Adapted Interact. (in this issue) (2007)
Papanikolaou K.A., Grigoriadou M., Kornikolakis H., Magoulas G.D. (2003) Personalizing the interaction in a web-based educational hypermedia system: the case of INSPIRE. User Model. User-Adapted Interact. 13, 213–267
Ramsay J.O. (1991) Kernel smoothing approaches to nonparametric item characteristic curve estimation. Psychometrika 56, 611–630
Reye J. (2002) A belief net backbone for student modelling. In: Cerri S.A., Gouardres G., Paraguacu F. (eds) Proceedings of the 6th International Conference on Intelligent Tutoring Systems (ITS 2002). Springer Verlag, New York, pp. 596–604, Lecture Notes in Computer Science, No. 2363.
Samejima F. (1997) The graded response model. In: van der Linden W.J., Hambleton R.K. (eds) Handbook of Modern Item Response Theory. Springer Verlag, New York, pp. 85–100
Schank R.C., Cleary C. (1994) Engines for Education. Lawrence Erlbaum Associates, Hillscale, NJ
Scriven M. (1967) The methodology of evaluation. In: Stake R.E. (eds) Perspectives of Curriculum Evaluation. Rand McNally, Chicago, IL
Segall D.O. (1996) Multidimensional adaptive testing. Psychometrika 61(1):331–354
Self J.A. (1994) Formal approaches to student modeling. In: Greer J.E., McCalla G. (eds) Student Modeling: The Key to Individualized Knowledge-Based Instruction, vol. 125. Springer Verlag, New York, pp. 295–352
Shute, V.J., Graf, E.A., Hansen, E.G.: Technology-Based Education: Bringing Researchers and Practicioners Together, Chapt. Design Adaptive, Diagnostic Math Assessments for Sighted and Visually Disabled Students, pp. 169–202. Information Age Publishing (2005)
Simonoff J.S. (1996) Smoothing Methods in Statistics. Springer-Verlag, New York
Stout W. (2002) Psychometrics: from practice to theory and back. Psychometrika 67, 485–518
Tam, S.S.: A comparison of methods for adaptive estimation of a multidimensional trait. Ph.D. thesis, Graduate School of Arts and Science, Columbia University. Order number: 9221219 (1992)
Tatsuoka K. (1985) A probabilistic model for diagnosing misconceptions in the pattern classification approach. J. Educ. Stat. 12(1):55–73
Thissen D., Steinberg L. (1997) A response model for multiple choice items. In: van der Linden W.J., Hambleton R.K. (eds) Handbook of modern item response theory. Springer Verlag, New York, pp. 51–65
Twidale, M.: Redressing the balance: the advantages of informal evaluation techniques for intelligent learning environments. Int. J. Artif. Intell. Educ. Special Issue on Evaluation 4(2/3), 155–178 (1993)
van der Linden, W.J., Glas, C.A.W.: Computerized Adaptive Testing: Theory and Practice. Kluwer Academic Publishers (2000)
van der Linden, W.J., Pashley, P.J.: Item selection and ability estimation in Adaptive Testing. In: van der Linden, W.J., Glas, C.A.W. (eds.) Computerized Adaptive Testing: Theory and Practice, pp. 1–26. Kluwer Academic Publisher, Dordrecht
VanLehn, K., Niu, Z., Siler, S., Gertner, A.S.: student modeling from conventional test data: a Bayesian approach without priors. In: Goettl, B., Redfield, C.L., Halff, H.M., Shute, V.J. (eds.) Proceedings of 4th International Conference on Intelligent Tutoring Systems. ITS’98. Lecture Notes in Computer Science, vol. 1452. pp. 434–443 (1998)
Vassileva, J.: Dynamic course generation on the WWW. In: du Bolay, B., Mizoguchi, R. (eds.) Knowledge and Media in Learning Systems. Proceedings of the 8th World Conference on Artificial Intelligence in Education AIED’97. pp. 498–505 (1997)
Wand M.P., Jones M.C. (1995) Kernel Smoothing. Chapman and Hall, London
Wang W.C., Chen P.H. (2004) Implementation and measurement efficiency of multidimensional computerized adaptive testing. Appl. Psychol. Measure. 28(5):295–316
Weber G., Brusilovsky P. (2001) ELM-ART: an adaptive versatile system for web-based instruction. Int. J. Artif. Intell. Educ. 12, 351–383
Zukerman I., Albrecht D. (2001) Predictive statistical models for user modeling. User Model. User-Adapted Interact. 11, 5–18
Author information
Authors and Affiliations
Corresponding author
Additional information
This paper (or a similar version) is not currently under review by a journal or conference, nor will it be submitted to such within the next three months.
Rights and permissions
About this article
Cite this article
Guzmán, E., Conejo, R. & Pérez-de-la-Cruz, JL. Adaptive testing for hierarchical student models. User Model User-Adap Inter 17, 119–157 (2007). https://doi.org/10.1007/s11257-006-9018-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11257-006-9018-1