Advertisement

Quizbot: Exploring Formative Feedback with Conversational Interfaces

  • Bharathi Vijayakumar
  • Sviatlana HöhnEmail author
  • Christoph Schommer
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1014)

Abstract

Conversational interfaces (also called chatbots) have recently disrupted the Internet and opened up endless opportunities for assessment and learning. Formative feedback that provides learners with practical instructions for improvement is one of the challenging tasks in self-assessment settings and self-directed learning. This becomes even more challenging if a user’s personal information such as learning history and previous achievements cannot be exploited for data protection reasons or are simply not available. This study seeks to explore the opportunities of providing formative feedback in chatbot-based self-assessment. Two main challenges were faced: the limitations of the messenger as an interface that restricts visual representation of the quiz questions, and zero information about the user to generate adaptive feedback. Two types of feedback were investigated regarding their formative effect: immediate feedback, which was given after answering a question, and cumulative feedback detailing strengths and weaknesses of the user in each of the topics covered along with the directives for improvement. A chatbot called SQL Quizbot was deployed on Facebook Messenger for the purposes of this study (Try out the prototype at https://www.messenger.com/t/2076690849324267). A survey conducted to disclose users’ perception of the feedback reveals that more than 80% of the users find immediate feedback helpful. Overall this study shows that chatbots have a great potential as an aiding tool for e-learning systems to include an interactive component into feedback in order to increase user motivation and retention.

Keywords

Formative feedback Educational chatbot Quizbot 

References

  1. 1.
    Muirhead, B., Juwah, C.: Interactivity in computer-mediated college and university education: a recent review of the literature. J. Educ. Technol. Soc. 7(1), 12–20 (2004)Google Scholar
  2. 2.
    Zheng, S., Rosson, M.B., Shih, P.C., Carroll, J.M.: Understanding student motivation, behaviors and perceptions in MOOCs. In: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & #38; Social Computing, CSCW 2015, pp. 1882–1895. ACM, New York (2015)Google Scholar
  3. 3.
    Weizenbaum, J.: ELIZA - a computer program for the study of natural language communication between man and machine. Commun. ACM 9, 36–45 (1966)CrossRefGoogle Scholar
  4. 4.
    Shute, V.J.: Focus on formative feedback. Rev. Educ. Res. 78(1), 153–189 (2008)CrossRefGoogle Scholar
  5. 5.
    Hattie, J., Timperley, H.: The power of feedback. Rev. Educ. Res. 77(1), 81–112 (2007)CrossRefGoogle Scholar
  6. 6.
    Lipowsky, F.: Unterricht. In: Wild, E., Möller, J. (eds.) Pädagogische Psychologie. SLB, pp. 69–105. Springer, Heidelberg (2015).  https://doi.org/10.1007/978-3-642-41291-2_4CrossRefGoogle Scholar
  7. 7.
    Wiliam, D.: Embedded Formative Assessment. Solution Tree Press, Bloomington (2011)Google Scholar
  8. 8.
    Höhn, S., Ras, E.: Designing formative and adaptive feedback using incremental user models. In: Chiu, D.K.W., Marenzi, I., Nanni, U., Spaniol, M., Temperini, M. (eds.) ICWL 2016. LNCS, vol. 10013, pp. 172–177. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-47440-3_19CrossRefGoogle Scholar
  9. 9.
    Denton, P., Madden, J., Roberts, M., Rowe, P.: Students’ response to traditional and computer-assisted formative feedback: a comparative case study. Br. J. Educ. Technol. 39(3), 486–500 (2008)CrossRefGoogle Scholar
  10. 10.
    Black, P., Wiliam, D.: Developing the theory of formative assessment. Educ. Assess. Eval. Account. (Former.: J. Pers. Eval. Educ.) 21(1), 5 (2009)CrossRefGoogle Scholar
  11. 11.
    Wiggins, G.: Seven keys to effective feedback. Educ. Leadersh. 70(1), 10–16 (2012)Google Scholar
  12. 12.
    Espasa, A., Guasch, T., Mayordomo, R., Martínez-Melo, M., Carless, D.: A dialogic feedback index measuring key aspects of feedback processes in online learning environments. High. Educ. Res. Dev. 37(3), 499–513 (2018)CrossRefGoogle Scholar
  13. 13.
    Narciss, S.: Designing and evaluating tutoring feedback strategies for digital learning. Digit. Educ. Rev. 23, 7–26 (2013)Google Scholar
  14. 14.
    Butler, A.C., Roediger, H.L.: Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Mem. Cogn. 36(3), 604–616 (2008)CrossRefGoogle Scholar
  15. 15.
    Bälter, O., Enström, E., Klingenberg, B.: The effect of short formative diagnostic web quizzes with minimal feedback. Comput. Educ. 60(1), 234–242 (2013)CrossRefGoogle Scholar
  16. 16.
    De Klerk, S., Veldkamp, B.P., Eggen, T.: The psychometric evaluation of a summative multimedia-based performance assessment. In: Ras, E., Joosten-ten Brinke, D. (eds.) CAA 2015. CCIS, vol. 571, pp. 1–11. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-27704-2_1CrossRefGoogle Scholar
  17. 17.
    Novacek, P.: Confidence-based assessments within an adult learning environment. In: International Association for Development of the Information Society, pp. 403–406 (2013)Google Scholar
  18. 18.
    Hench, T.L.: Using confidence as feedback in multi-sized learning environments. In: Kalz, M., Ras, E. (eds.) CAA 2014. CCIS, vol. 439, pp. 88–99. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-08657-6_9CrossRefGoogle Scholar
  19. 19.
    Lundeberg, M.A., Fox, P.W., Punćcohaí, J.: Highly confident but wrong: gender differences and similarities in confidence judgments. J. Educ. Psychol. 86(1), 114 (1994)CrossRefGoogle Scholar
  20. 20.
    Jonsson, A.C., Allwood, C.M.: Stability and variability in the realism of confidence judgments over time, content domain, and gender. Pers. Individ. Differ. 34(4), 559–574 (2003)CrossRefGoogle Scholar
  21. 21.
    Burns, K.M., Burns, N.R., Ward, L.: Confidence–more a personality or ability trait? It depends on how it is measured: a comparison of young and older adults. Front. Psychol. 7, 518 (2016)CrossRefGoogle Scholar
  22. 22.
    West, R.F., Stanovich, K.E.: The domain specificity and generality of overconfidence: individual differences in performance estimation bias. Psychon. Bull. Rev. 4(3), 387–392 (1997)CrossRefGoogle Scholar
  23. 23.
    Gardner-Medwin, A.: Confidence assessment in the teaching of basic science. ALT-J 3(1), 80–85 (1995)CrossRefGoogle Scholar
  24. 24.
    Gardner-Medwin, A.: 12 confidence-based marking. In: Innovative Assessment in Higher Education, p. 141 (2006)Google Scholar
  25. 25.
    Ericsson, K.A., Krampe, R.T., Tesch-Römer, C.: The role of deliberate practice in the acquisition of expert performance. Psychol. Rev. 100(3), 363–406 (1993)CrossRefGoogle Scholar
  26. 26.
    Christodoulou, D., Wiliam, D.: Making Good Progress?: The Future of Assessment for Learning. Oxford University Press, Oxford (2017)Google Scholar
  27. 27.
    Roediger, H.L., Karpicke, J.D.: The power of testing memory: basic research and implications for educational practice. Perspect. Psychol. Sci. 1(3), 181–210 (2006). PMID: 26151629CrossRefGoogle Scholar
  28. 28.
    Petersen, K.A.: Implicit corrective feedback in computer-guided interaction: does mode matter? Ph.D. thesis, Georgetown University (2010)Google Scholar
  29. 29.
    Wilske, S.: Form and meaning in dialog-based computer-assisted language learning. Ph.D. thesis, University of Saarland (2014)Google Scholar
  30. 30.
    Kerly, A., Hall, P., Bull, S.: Bringing chatbots into education: towards natural language negotiation of open learner models. Knowl.-Based Syst. 20(2), 177–185 (2007)CrossRefGoogle Scholar
  31. 31.
    Kane, D.A.: The role of chatbots in teaching and learning. In: E-Learning and the Academic Library: Essays on Innovative Initiatives, UC Irvine, pp. 1–26 (2016)Google Scholar
  32. 32.
    Soliman, M., Guetl, C.: Intelligent pedagogical agents in immersive virtual learning environments: a review. In: MIPRO 2010 Proceedings of the 33rd International Convention, pp. 827–832. IEEE (2010)Google Scholar
  33. 33.
    MacTear, M., Callejas, Z., Griol, D.: The Conversational Interface: Talking to Smart Devices. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-32967-3CrossRefGoogle Scholar
  34. 34.
    DeSmedt, W.H.: Herr Kommissar: an ICALL conversation simulator for intermediate German. In: Holland, V.M., Sams, M.R., Kaplan, J.D. (eds.) Intelligent Language Tutors: Theory Shaping Technology. Routledge, New York (1995)Google Scholar
  35. 35.
    Lu, C.-H., Chiou, G.-F., Day, M.-Y., Ong, C.-S., Hsu, W.-L.: Using instant messaging to provide an intelligent learning environment. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053, pp. 575–583. Springer, Heidelberg (2006).  https://doi.org/10.1007/11774303_57CrossRefGoogle Scholar
  36. 36.
    Jia, J.: CSIEC: a computer assisted english learning chatbot based on textual knowledge and reasoning. Knowl.-Based Syst. 22(4), 249–255 (2009)CrossRefGoogle Scholar
  37. 37.
    Höhn, S.: A data-driven model of explanations for a chatbot that helps to practice conversation in a foreign language. In: Proceedings of SIGDIAL 2017 Conference. ACM (2017)Google Scholar
  38. 38.
    Lyster, R., Ranta, L.: Corrective feedback and learner uptake. Stud. Second Lang. Acquis. 19(01), 37–66 (1997)CrossRefGoogle Scholar
  39. 39.
    Lyster, R., Saito, K., Sato, M.: Oral corrective feedback in second language classrooms. Lang. Teach. 46, 1–40 (2013)CrossRefGoogle Scholar
  40. 40.
    Amaral, L.A., Meurers, D.: On using intelligent computer-assisted language learning in real-life foreign language teaching and learning. ReCALL 23(01), 4–24 (2011)CrossRefGoogle Scholar
  41. 41.
    Gross, S., Pinkwart, N.: Towards an integrative learning environment for Java programming. In: 2015 IEEE 15th International Conference on Advanced Learning Technologies (ICALT), pp. 24–28, July 2015Google Scholar
  42. 42.
    Perikos, I., Grivokostopoulou, F., Hatzilygeroudis, I.: Assistance and feedback mechanism in an intelligent tutoring system for teaching conversion of natural language into logic. Int. J. Artif. Intell. Educ. 27(3), 475–514 (2017)CrossRefGoogle Scholar
  43. 43.
    Ryan, T., Henderson, M.: Feeling feedback: students’ emotional responses to educator feedback. Assess. Eval. High. Educ. 43(6), 880–892 (2018)CrossRefGoogle Scholar
  44. 44.
    Palminteri, S., Khamassi, M., Joffily, M., Coricelli, G.: Contextual modulation of value signals in reward and punishment learning. Nat. Commun. 6, 8096 (2015).  https://doi.org/10.1038/ncomms9096 CrossRefGoogle Scholar
  45. 45.
    Ras, E., Baudet, A., Foulonneau, M.: A hybrid engineering process for semi-automatic item generation. In: Joosten-ten Brinke, D., Laanpere, M. (eds.) TEA 2016. CCIS, vol. 653, pp. 105–116. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-57744-9_10CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Bharathi Vijayakumar
    • 1
  • Sviatlana Höhn
    • 2
    Email author
  • Christoph Schommer
    • 3
  1. 1.University of LuxembourgEsch-sur-AlzetteLuxembourg
  2. 2.Artificial Companions and Chatbots LabUniversity of LuxembourgEsch-sur-AlzetteLuxembourg
  3. 3.Interdisciplinary Lab for Intelligent and Adaptive SystemsUniversity of LuxembourgEsch-sur-AlzetteLuxembourg

Personalised recommendations