Skip to main content
SpringerLink
Log in

Predicting Question Quality Using Recurrent Neural Networks

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2018)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10947))

Included in the following conference series:

Abstract

This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this dataset based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22%, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6%). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    www.cdlponline.org.

References

  1. Snow, C.: Reading for Understanding Toward an R&D Program in Reading Comprehension. Rand Corporation, Santa Monica (2002)

    Google Scholar 

  2. Palincsar, A.S., Brown, A.L.: Interactive promote learning teaching independent from text to. Read. Teach. 39, 771–777 (1986)

    Google Scholar 

  3. Rosenshine, B., Meister, C.: Reciprocal teaching: a review of the research. Rev. Educ. Res. 64, 479–530 (1994)

    Article  Google Scholar 

  4. Rosenshine, B., Meister, C., Chapman, S.: Teaching students to generate questions: a review of the intervention studies. Rev. Educ. Res. 66, 181–221 (1996)

    Article  Google Scholar 

  5. McNamara, D.S., O’Reilly, T., Rowe, M., Boonthum, C., Levinstein, I.: iSTART: a web-based tutor that teaches self-explanation and metacognitive reading strategies. In: Reading Comprehension Strategies: Theories, Interventions, and Technologies, pp. 397–420 (2007)

    Google Scholar 

  6. VanLehn, K., Graesser, A.C., Jackson, G.T., Jordan, P., Olney, A., Rosé, C.P.: When are tutorial dialogues more effective than reading? Cogn. Sci. 31, 3–62 (2007)

    Article  Google Scholar 

  7. Graesser, A.C., McMahen, C.L.: Anomalous information triggers questions when adults solve quantitative problems and comprehend stories. J. Educ. Psychol. 85, 136–151 (1993)

    Article  Google Scholar 

  8. Wisher, R.A., Graesser, A.C.: Question-asking in advanced distributed learning environments. In: Toward a Science of Distributed Learning and Training, pp. 209–234. American Psychological Association, Washington, D.C. (2007)

    Google Scholar 

  9. Beck, I., McKeown, M.G., Hamilton, R.L., Kucan, L.: Questioning the Author: An Approach for Enhancing Student Engagement (1997). https://eric.ed.gov/?id=ED408562

  10. Kintsch, W.: Comprehension: A Paradigm for Cognition. Cambridge University Press, New York (1998)

    Google Scholar 

  11. Graesser, A.C., Person, N.K.: Question asking during tutoring. Am. Educ. Res. J. 31, 104–137 (1994)

    Article  Google Scholar 

  12. Davey, B., McBride, S.: Effects of question-generation training on reading comprehension. J. Educ. Psychol. 78, 256–262 (1986)

    Article  Google Scholar 

  13. Craig, S.D., Gholson, B., Ventura, M., Graesser, A.C.: Overhearing dialogues and monologues in virtual tutoring sessions: effects on questioning and vicarious learning. Int. J. Artif. Intell. Educ. 11, 242–253 (2000)

    Google Scholar 

  14. Silva, J., Coheur, L., Mendes, A.C., Wichert, A.: From symbolic to sub-symbolic information in question classification. Artif. Intell. Rev. 35, 137–154 (2011)

    Article  Google Scholar 

  15. Ittycheriah, A., Franz, M., Zhu, W., Ratnaparkhi, A., Mammone, R.J.: IBM’s statistical question answering system. In: Proceedings of TREC-9 Conference, pp. 229–234 (2000)

    Google Scholar 

  16. Hovy, E., Gerber, L., Hermjakob, U., Lin, C.-Y., Ravichandran, D.: Toward semantics-based answer pinpointing. In: Proceedings of the First International Conference on Human Language Technology Research, pp. 1–7 (2001)

    Google Scholar 

  17. Harabagiu, S., Moldovan, D., Pasca, M., Mihalcea, R., Surdeanu, M., Bunescu, R., Girju, R., Rus, V., Morarescu, P.: FALCON: boosting knowledge for answer engines. In: Proceedings of Ninth Text Retrieval Conference (TREC 2000), pp. 479–488 (2000)

    Google Scholar 

  18. Gerber, L.: A QA Typology for Webclopedia (2001)

    Google Scholar 

  19. Li, X., Roth, D.: Learning question classifiers. In: Proceedings of the 19th International Conference on Computational Linguistics, vol. 1, pp. 1–7 (2002)

    Google Scholar 

  20. Bloom, B.S.: Taxonomy of Educational Objectives, Cognitive Domain, pp. 20–24. McKay, New York (1956)

    Google Scholar 

  21. Mosenthal, P.B.: Understanding the strategies of document literacy and their conditions of use. J. Educ. Psychol. 88, 314–332 (1996)

    Article  Google Scholar 

  22. Olney, A., Louwerse, M., Matthews, E., Marineau, J., Hite-Mitchell, H., Graesser, A.: Utterance classification in AutoTutor. In: Proceedings of HLT-NAACL 2003 Workshop on Building Educational Applications Using Natural Language Processing, vol. 2, pp. 1–8. ACL, Morristown (2003)

    Google Scholar 

  23. Pietra, S.D., Pietra, V.D., Lafferty, J.: Inducing features of random fields. IEEE Trans. Pattern Anal. Mach. Intell. 19, 380–393 (1997)

    Article  Google Scholar 

  24. Chinchor, N., Robinson, P.: MUC-7 named entity task definition. In: Proceedings of the 7th Conference on Message Understanding, MUC6, p. 21 (1997)

    Google Scholar 

  25. Hacioglu, K., Ward, W.: Question classification with support vector machines and error correcting codes. In: Proceedings of HLT-NAACL 2003, pp. 28–30. Association for Computational Linguistics, Morristown (2003)

    Google Scholar 

  26. Zhang, D., Lee, W.S.: Question classification using support vector machines. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2003, p. 26. ACM Press, New York (2003)

    Google Scholar 

  27. Blunsom, P., Kocik, K., Curran, J.R.: Question classification with log-linear models. In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2006, p. 615 (2006)

    Google Scholar 

  28. Kopp, K.J., Johnson, A.M., Crossley, S.A., McNamara, D.S.: Assessing question quality using NLP. In: André, E., Baker, R., Hu, X., Rodrigo, Ma.M.T., du Boulay, B. (eds.) AIED 2017. LNCS, vol. 10331, pp. 523–527. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61425-0_55

    Chapter  Google Scholar 

  29. Krishnan, V., Das, S., Chakrabarti, S.: Enhanced answer type inference from questions using sequential models. In: Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, HLT 2005, pp. 315–322 (2005)

    Google Scholar 

  30. Suzuki, J., Taira, H., Sasaki, Y., Maeda, E.: Question classification using HDAG kernel. In: Proceedings of the ACL 2003 Workshop on Multilingual Summarization and Question Answering, vol. 12, pp. 61–68 (2003)

    Google Scholar 

  31. Hermjakob, U.: Parsing and question classification for question answering. In: Proceedings of Working Open-domain Question Answering, vol. 12, pp. 1–6 (2001)

    Google Scholar 

  32. Mishra, M., Mishra, V.K., Sharma, H.R.: Question classification using semantic, syntactic and lexical features. Int. J. Web Semant. Technol. 4, 39–47 (2013)

    Article  Google Scholar 

  33. Elman, J.L.: Finding structure in time. Cogn. Sci. 14, 179–211 (1990)

    Article  Google Scholar 

  34. Yih, W., He, X., Meek, C.: Semantic parsing for single-relation question answering. In: Association for Computational Linguistics, pp. 643–648 (2014)

    Google Scholar 

  35. Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics, pp. 655–665 (2014)

    Google Scholar 

  36. Iyyer, M., Boyd-graber, J., Claudino, L., Socher, R., Daum, H.: A neural network for factoid question answering over paragraphs. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014)

    Google Scholar 

  37. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (Almost) from scratch. J. Mach. Learn. Res. 12, 2493–2537 (2011)

    MATH  Google Scholar 

  38. Fei, T., Heng, W.J., Toh, K.C., Qi, T.: Question classification for e-learning by artificial neural network. In: ICICS-PCM 2003, pp. 1757–1761. IEEE (2003)

    Google Scholar 

  39. Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of 2014 Conference on Empirical Methods Natural Language Processing (EMNLP 2014), pp. 1746–1751 (2014)

    Google Scholar 

  40. Lai, S., Xu, L., Liu, K., Zhao, J.: Recurrent convolutional neural networks for text classification. In: Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 2267–2273 (2015)

    Google Scholar 

  41. Crump, M.J.C., McDonnell, J.V., Gureckis, T.M.: Evaluating Amazon’s mechanical turk as a tool for experimental behavioral research. PLoS One 8, e57410 (2013)

    Article  Google Scholar 

  42. Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  43. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching Word Vectors with Subword Information (2016)

    Google Scholar 

  44. Mikolov, T., Corrado, G., Chen, K., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of ICLR 2013, pp. 1–12 (2013)

    Google Scholar 

  45. Darío Gutiérrez, E., Levy, R., Bergen, B.K.: Finding non-arbitrary form-meaning systematicity using string-metric learning for Kernel regression. In: ACL, pp. 2379–2388 (2016)

    Google Scholar 

  46. Hochreiter, S., Urgen Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)

    Article  Google Scholar 

  47. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: EMNLP 2014, pp. 1724–1734 (2014)

    Google Scholar 

  48. Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18, 602–610 (2005)

    Article  Google Scholar 

  49. dos Santos, C., Tan, M., Xiang, B., Zhou, B.: Attentive Pooling Networks. CoRR, abs/1602.03609. 2, 4 (2016)

    Google Scholar 

  50. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations (2015)

    Google Scholar 

  51. Xiong, C., Zhong, V., Socher, R.: Dynamic coattention networks for question answering. In: ICLR Submission, pp. 1–13 (2017)

    Google Scholar 

  52. Seo, M., Kembhavi, A., Farhadi, A., Hajishirzi, H.: Bidirectional attention flow for machine comprehension. In: ICLR 2017 (2017)

    Google Scholar 

  53. Masala, M., Ruseti, S., Rebedea, T.: Sentence selection with neural networks using string kernels. Proc. Comput. Sci. 112, 1774–1782 (2017)

    Article  Google Scholar 

Download references

Acknowledgments

This research was partially supported by the 644187 EC H2020 Realising an Applied Gaming Eco-system (RAGE) project, the FP7 2008-212578 LTfLL project, the Department of Education, Institute of Education Sciences - Grant R305A130124, as well as the Department of Defense, Office of Naval Research - Grants N00014140343 and N000141712300.

Author information

Authors and Affiliations

  1. Faculty of Automatic Control and Computers, University “Politehnica” of Bucharest, 313 Splaiul Independenței, 60042, Bucharest, Romania

    Stefan Ruseti, Mihai Dascalu & Stefan Trausan-Matu

  2. Academy of Romanian Scientists, Splaiul Independenţei 54, 050094, Bucharest, Romania

    Mihai Dascalu & Stefan Trausan-Matu

  3. Institute for the Science of Teaching and Learning, Arizona State University, PO Box 872111, Tempe, AZ, 85287, USA

    Amy M. Johnson, Renu Balyan, Kristopher J. Kopp & Danielle S. McNamara

  4. Department of Applied Linguistics/ESL, Georgia State University, Atlanta, GA, 30303, USA

    Scott A. Crossley

Authors
  1. Stefan Ruseti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mihai Dascalu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amy M. Johnson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Renu Balyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kristopher J. Kopp
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Danielle S. McNamara
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Scott A. Crossley
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Stefan Trausan-Matu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mihai Dascalu .

Editor information

Editors and Affiliations

  1. Carnegie Mellon University, Pittsburgh, PA, USA

    Carolyn Penstein Rosé

  2. University of Technology, Sydney, NSW, Australia

    Roberto Martínez-Maldonado

  3. University of Duisburg-Essen, Duisburg, Germany

    H. Ulrich Hoppe

  4. UCL Institute of Education, London, UK

    Rose Luckin

  5. UCL Institute of Education, London, UK

    Manolis Mavrikis

  6. UCL Institute of Education, London, UK

    Kaska Porayska-Pomsta

  7. Carnegie Mellon University, Pittsburgh, PA, USA

    Bruce McLaren

  8. University of Sussex, Brighton, UK

    Benedict du Boulay

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ruseti, S. et al. (2018). Predicting Question Quality Using Recurrent Neural Networks. In: Penstein Rosé, C., et al. Artificial Intelligence in Education. AIED 2018. Lecture Notes in Computer Science(), vol 10947. Springer, Cham. https://doi.org/10.1007/978-3-319-93843-1_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-93843-1_36

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-93842-4

  • Online ISBN: 978-3-319-93843-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation