Abstract
When implementing a programming tutor it is often difficult to manually consider all possible errors encountered by students. An alternative is to automatically learn a bug library of erroneous patterns from students’ programs. We propose abstract-syntax-tree (AST) patterns as features for learning rules to distinguish between correct and incorrect programs. We use these rules to debug student programs: rules for incorrect programs (buggy rules) indicate mistakes, whereas rules for correct programs group programs with the same solution strategy. To generate hints, we first check buggy rules and point out incorrect patterns. If no buggy rule matches, we use rules for correct programs to recognize the student’s intent and suggest missing patterns. We evaluate our approach on past student programming data for a number of Prolog problems. For 31 out of 44 problems, the induced rules correctly classify over 85% of programs based only on their structural features. For approximately 73% of incorrect submissions, we are able to generate hints that were implemented by the student in some subsequent submission.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Binary relations like this one should be read as “X is a sister/parent/... of Y”.
- 2.
We report only a subset of results due to space restrictions. Full results and source code can be found at https://ailab.si/aied2017.
References
Bagrak, I., Shivers, O.: trx: regular-tree expressions, now in scheme. In: Proceeding of Fifth Workshop on Scheme and Functional Programming, pp. 21–32 (2004)
Clark, P., Boswell, R.: Rule induction with CN2: some recent improvements. In: Proceeding Fifth European Conference on Machine Learning, pp. 151–163 (1991)
Demšar, J., Curk, T., Erjavec, A., Gorup, Č., Hočevar, T., Milutinovič, M., Možina, M., Polajnar, M., Toplak, M., Starič, A., štajdohar, M., Umek, L., Žagar, L., Žbontar, J., Žitnik, M., Zupan, B.: Orange: data mining toolbox in Python. J. Mach. Learn. Res. 14, 2349–2353 (2013). http://jmlr.org/papers/v14/demsar13a.html
Folsom-Kovarik, J.T., Schatz, S., Nicholson, D.: Plan ahead: pricing ITS learner models. In: Proceeding 19th Behavior Representation in Modeling & Simulation Conference, pp. 47–54 (2010)
Gerdes, A., Heeren, B., Jeuring, J., van Binsbergen, L.T.: Ask-elle: an adaptable programming tutor for haskell giving automated feedback. Int. J. Artif. Intell. Educ. https://link.springer.com/article/10.1007/s40593-015-0080-x
Holland, J., Mitrovic, A., Martin, B.: J-LATTE: a constraint-based tutor for Java. In: Proceeding 17th International Conference Computers in Education (ICCE 2009), pp. 142–146 (2009)
Hong, J.: Guided programming and automated error analysis in an intelligent Prolog tutor. Int. J. Hum.-Comput. Stud. 61(4), 505–534 (2004)
Jin, W., Barnes, T., Stamper, J., Eagle, M.J., Johnson, M.W., Lehmann, L.: Program representation for automatic hint generation for a data-driven novice programming tutor. In: Proceeding of 11th International Conference Intelligent Tutoring Systems, pp. 304–309 (2012)
Johnson, W.L.: Understanding and debugging novice programs. Artif. Intell. 42(1), 51–97 (1990)
Keuning, H., Jeuring, J., Heeren, B.: Towards a systematic review of automated feedback generation for programming exercises. In: Proceeding of 2016 ACM Conference on Innovation and Technology in Computer Science Education, pp. 41–46. ACM (2016)
Le, N.T., Loll, F., Pinkwart, N.: Operationalizing the continuum between well-defined and ill-defined problems for educational technology. IEEE Trans. Learn. Technol. 6(3), 258–270 (2013)
Le, N.T., Menzel, W.: Using weighted constraints to diagnose errors in logic programming - the case of an ill-defined domain. Int. J. Artif. Intell. Educ. 19(4), 381–400 (2009)
Levy, R., Andrew, G.: Tregex and Tsurgeon: tools for querying and manipulating tree data structures. In: 5th International Conference Language Resources and Evaluation (2006)
Li, S., Xiao, X., Bassett, B., Xie, T., Tillmann, N.: Measuring code behavioral similarity for programming and software engineering education. In: Proceeding 38th International Conference on Software Engineering Companion, pp. 501–510. ACM (2016)
Mitrovic, A.: Fifteen years of constraint-based tutors: what we have achieved and where we are going. User Model. User-Adap. Inter. 22(1–2), 39–72 (2012)
Nguyen, A., Piech, C., Huang, J., Guibas, L.: Codewebs: scalable homework search for massive open online programming courses. In: Proceeding 23rd International Conference World Wide Web, pp. 491–502 (2014)
Rivers, K., Koedinger, K.R.: Data-driven hint generation in vast solution spaces: a self-improving Python programming tutor. Int. J. Artif. Intell. Educ. https://link.springer.com/article/10.1007/s40593-015-0070-z
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Lazar, T., Možina, M., Bratko, I. (2017). Automatic Extraction of AST Patterns for Debugging Student Programs. In: André, E., Baker, R., Hu, X., Rodrigo, M., du Boulay, B. (eds) Artificial Intelligence in Education. AIED 2017. Lecture Notes in Computer Science(), vol 10331. Springer, Cham. https://doi.org/10.1007/978-3-319-61425-0_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-61425-0_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-61424-3
Online ISBN: 978-3-319-61425-0
eBook Packages: Computer ScienceComputer Science (R0)