Skip to main content
SpringerLink
Log in

What’s Most Broken? Design and Evaluation of a Tool to Guide Improvement of an Intelligent Tutor

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11625))

Included in the following conference series:

Abstract

Intelligent Tutoring Systems (ITS) have great potential to change the educational landscape by bringing scientifically tested one-to-one tutoring to remote and under-served areas. However, effective ITSs are too complex to perfect. Instead, a practical guiding principle for ITS development and improvement is to fix what’s most broken. This paper presents SPOT (Statistical Probe of Tutoring), a tool that mines data logged by an ITS to identify ‘hot spots’ most detrimental to its efficiency and effectiveness in terms of its software reliability, usability, task difficulty, student engagement, and other criteria. SPOT uses heuristics and machine learning to discover, characterize, and prioritize such hot spots in order to focus ITS refinement on what matters most. We applied SPOT to data logged by RoboTutor, an ITS that teaches children basic reading, writing and arithmetic. A panel-of-experts experimental evaluation found SPOT’s selected video clips of RoboTutor’s hot spots as significantly more informative than video clips selected randomly.

S. Mian and M. Goswami—authors contributed equally. Work was partially done while the primary authors were Summer Scholars at the Robotics Institute, Carnegie Mellon University, PA.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aleven, V., McLaren, B.M., Sewall, J., Koedinger, K.R.: The cognitive tutor authoring tools (CTAT): preliminary evaluation of efficiency gains. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053, pp. 61–70. Springer, Heidelberg (2006). https://doi.org/10.1007/11774303_7

    Chapter  Google Scholar 

  2. Apté, C., Weiss, S.: Data mining with decision trees and decision rules. Future Gener. Comput. Syst. 13(2–3), 197–210 (1997)

    Article  Google Scholar 

  3. Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z.: Off-task behavior in the cognitive tutor classroom: when students game the system. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 383–390. ACM (2004)

    Google Scholar 

  4. Baker, R.S.J., Corbett, A.T., Koedinger, K.R., Roll, I.: Generalizing detection of gaming the system across a tutoring curriculum. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053, pp. 402–411. Springer, Heidelberg (2006). https://doi.org/10.1007/11774303_40

    Chapter  Google Scholar 

  5. d Baker, R.S., et al.: Towards sensor-free affect detection in cognitive tutor algebra. Int. Educ. Data Min. Soc. (2012)

    Google Scholar 

  6. Baker, R.S.: Modeling and understanding students’ off-task behavior in intelligent tutoring systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1059–1068. ACM (2007)

    Google Scholar 

  7. Baker, R.S., de Carvalho, A., Raspat, J., Aleven, V., Corbett, A.T., Koedinger, K.R.: Educational software features that encourage and discourage “gaming the system”. In: Proceedings of the 14th International Conference on Artificial Intelligence in Education, pp. 475–482 (2009)

    Google Scholar 

  8. Breiman, L., Friedman, J., Olshen, R., Stone, C.: Classification and regression trees (1984)

    Google Scholar 

  9. Chrysafiadi, K., Virvou, M.: Usability factors for an intelligent tutoring system on computer programming. In: Damiani, E., Jeong, J., Howlett, R.J., Jain, L.C. (eds.) New Directions in Intelligent Interactive Multimedia Systems and Services-2, vol. 226, pp. 339–347. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02937-0_31

    Chapter  Google Scholar 

  10. Chughtai, R., Zhang, S., Craig, S.D.: Usability evaluation of intelligent tutoring system: its from a usability perspective. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 59, pp. 367–371, SAGE Publications, Los Angeles (2015)

    Google Scholar 

  11. Cocea, M., Hershkovitz, A., Baker, R.S.: The impact of off-task and gaming behaviors on learning: immediate or aggregate? In: Proceeding of the 2009 Conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling. IOS Press (2009)

    Google Scholar 

  12. Cocea, M., Weibelzahl, S.: Can log files analysis estimate learners’ level of motivation? In: LWA. University of Hildesheim, Institute of Computer Science (2006)

    Google Scholar 

  13. Cocea, M., Weibelzahl, S.: Cross-system validation of engagement prediction from log files. In: Duval, E., Klamma, R., Wolpers, M. (eds.) EC-TEL 2007. LNCS, vol. 4753, pp. 14–25. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-75195-3_2

    Chapter  Google Scholar 

  14. Goel, A.L.: Software reliability models: assumptions, limitations, and applicability. IEEE Trans. Softw. Eng. 12, 1411–1423 (1985)

    Article  Google Scholar 

  15. Granić, A., Glavinić, V.: An approach to usability evaluation of an intelligent tutoring system. In: Mastorakis, N., Kluev, V. (eds.) Advances in Multimedia, Video and Signal Processing Systems (2002)

    Google Scholar 

  16. Henry, G., Lin, J., Park, C.Y.: SPOT: refining robotutor. HCI senior capstone project presentation, Carnegie Mellon University (2017)

    Google Scholar 

  17. Joseph, E.: Engagement tracing: using response times to model student disengagement. Artif. Intell. Educ.: Support. Learn. Through Intell. Soc. Inf. Technol. 125, 88 (2005)

    Google Scholar 

  18. Karunanithi, N., Whitley, D., Malaiya, Y.K.: Prediction of software reliability using connectionist models. IEEE Trans. Softw. Eng. 18(7), 563–574 (1992)

    Article  Google Scholar 

  19. Koedinger, K.R., Booth, J.L., Klahr, D.: Instructional complexity and the science needed to constrain it. Science 342, 935–937 (2014)

    Article  Google Scholar 

  20. LearnSphere: LearnSphere (2017). http://learnsphere.org/

  21. Mostow, J., Beck, J., Cen, H., Cuneo, A., Gouvea, E., Heiner, C.: An educational data mining tool to browse tutor-student interactions: time will tell. In: Proceedings of the Workshop on Educational Data Mining, National Conference on Artificial Intelligence, pp. 15–22. AAAI Press (2005)

    Google Scholar 

  22. Murthy, S.K.: Automatic construction of decision trees from data: a multi-disciplinary survey. Data Min. Knowl. Discov. 2(4), 345–389 (1998)

    Article  Google Scholar 

  23. Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12(Oct), 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  24. Play, G.: AZ Screen Recorder (2018). https://play.google.com/store/apps/details?id=com.hecorat.screenrecorder.free

  25. RoboTutor: RoboTutor (2015). http://robotutor.org

  26. Romero, C., Ventura, S.: Educational data mining: a review of the state of the art. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 40(6), 601–618 (2010)

    Article  Google Scholar 

  27. XPRIZE: Global Learning XPRIZE (2015). http://learning.xprize.org

Download references

Author information

Authors and Affiliations

  1. IIIT-Delhi, Delhi, India

    Shiven Mian

  2. Delhi Technological University, Delhi, India

    Mononito Goswami

  3. Carnegie Mellon University, Pittsburgh, PA, USA

    Jack Mostow

Authors
  1. Shiven Mian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mononito Goswami
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jack Mostow
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shiven Mian .

Editor information

Editors and Affiliations

  1. University of Sao Paulo, Sao Paulo, Brazil

    Seiji Isotani

  2. University of Malaga, Málaga, Spain

    Eva Millán

  3. Carnegie Mellon University, Pittsburgh, PA, USA

    Amy Ogan

  4. DePaul University, Chicago, IL, USA

    Peter Hastings

  5. Carnegie Mellon University, Pittsburgh, PA, USA

    Bruce McLaren

  6. University College London, London, UK

    Rose Luckin

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mian, S., Goswami, M., Mostow, J. (2019). What’s Most Broken? Design and Evaluation of a Tool to Guide Improvement of an Intelligent Tutor. In: Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., Luckin, R. (eds) Artificial Intelligence in Education. AIED 2019. Lecture Notes in Computer Science(), vol 11625. Springer, Cham. https://doi.org/10.1007/978-3-030-23204-7_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-23204-7_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-23203-0

  • Online ISBN: 978-3-030-23204-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation