Skip to main content
SpringerLink
Log in
Book cover

International Computer Assisted Assessment Conference

TEA 2015: Computer Assisted Assessment. Research into E-Assessment pp 80–95Cite as

Automatically Generated Metrics for Multilingual Inline Choice Questions on Reading Comprehension

  • Conference paper
  • First Online:

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 571))

Abstract

Assessment Item Generation (AIG) aims at creating semi-automatically many items from a template. This type of approaches has been used in various domains, including language learning and mathematics to support adaptation of tests to learners or allow the item authoring process to scale through decreasing the cost of items. We illustrate in this paper the automatic creation of inline choice items for reading comprehension skills using state-of-the-art approaches. However we show how the AIG process can be implemented to support the creation of items in multiple languages (English, French, and German) and how it can be complemented by the creation of item quality metrics to improve the selection of the generated items.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    http://www.oecd.org/pisa/.

  2. 2.

    http://www.eagle-learning.eu/.

  3. 3.

    http://www.interlingua-project.eu/.

  4. 4.

    http://www3.nccu.edu.tw/~smithsgj/teddclog.html.

  5. 5.

    http://libots.sourceforge.net/.

  6. 6.

    http://autosummarizer.com/.

  7. 7.

    http://classifier4j.sourceforge.net/index.html.

  8. 8.

    http://www.researchgate.net/publication/262375542_Automatic_Text_Summarizer.

  9. 9.

    http://sujitpal.blogspot.com/2009/02/summarization-with-lucene.html.

  10. 10.

    http://nlp.stanford.edu/software/tagger.shtml.

  11. 11.

    http://www.linguatools.de/disco/disco.html.

  12. 12.

    https://books.google.com/ngrams.

  13. 13.

    Computer. http://en.wikipedia.org/wiki/Computer.

    Computer. http://de.wikipedia.org/wiki/Computer.

    Ordinateur. http://fr.wikipedia.org/wiki/Ordinateur. Accessed: 27 Feb. 2015.

  14. 14.

    Roman Empire. http://en.wikipedia.org/wiki/Roman_Empire.

    Römische Kaiserzeit. http://de.wikipedia.org/wiki/R%C3%B6mische_Kaiserzeit.

    Empire romain. http://fr.wikipedia.org/wiki/Empire_romain. Accessed: 27 Feb. 2015.

  15. 15.

    http://www.guichet.public.lu/entreprises/en/ressources-humaines/fin-relation-travail/licenciement-resiliation/licencier-preavis/.

    http://www.guichet.public.lu/entreprises/de/ressources-humaines/fin-relation-travail/licenciement-resiliation/licencier-preavis/.

    http://www.guichet.public.lu/entreprises/fr/ressources-humaines/fin-relation-travail/licenciement-resiliation/licencier-preavis/.

References

  1. ACRL: Framework for Information Literacy for Higher Education (2015)

    Google Scholar 

  2. Brown, J.C., Frishkoff, G.A., Eskenazi, M.: Automatic question generation for vocabulary assessment. In: Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP). pp. 819–826. Association for Computational Linguistics, Vancouver, Canada (2005)

    Google Scholar 

  3. Chen, C.Y., Liou, H.C., Chang, J.S.: Fast: an automatic generation system for grammar tests. In: COLING/ACL on Interactive Presentation Sessions, pp. 1–4. Association for Computational Linguistics, Morristown, NJ, USA (2006)

    Google Scholar 

  4. Mostow, J., et al.: Using automated questions to assess reading comprehension, vocabulary, and effects of tutorial interventions. Technol. Instr. Cogn. Learn. 2, 97–134 (2004)

    Google Scholar 

  5. Papasalouros, A., Kanaris, K., Kotis, K.: Automatic generation of multiple choice questions from domain ontologies. In: e-Learning, Citeseer (2008)

    Google Scholar 

  6. Gierl, M.J., Lai, H.: Using weak and strong theory to create item models for automatic item generation. Autom. Item Gener.: Theor. Pract., 26 (2012)

    Google Scholar 

  7. Haladyna, T.M., Downing, S.M., Rodriguez, M.C.: A review of multiple-choice item-writing guidelines for classroom assessment. Appl. Measur. Educ. 15(3), 309–333 (2002)

    Article  Google Scholar 

  8. Mitkov, R., Ha, L.A., Karamanis, N.: A computer-aided environment for generating multiple-choice test items. Nat. Lang. Eng. 12(2), 177–194 (2006)

    Article  Google Scholar 

  9. Shah, R.: Automatic question generation using discourse cues and distractor selection for cloze questions. In: Language Technology and Research Center (LTRC), International Institute of Information Technology, Hyderabad (2012)

    Google Scholar 

  10. Becker, L., Basu, S., Vanderwende, L.: Mind the gap: learning to choose gaps for question generation. In: Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics (2012)

    Google Scholar 

  11. Heilman, M.: Automatic Factual Question Generation from Text. Carnegie Mellon University, Pittsburgh (2011)

    Google Scholar 

  12. Iwata, T., et al.: Automatic Generation of English Cloze Questions Based on Machine Learning. NTT Communication Science Laboratories, Kyoto (2011)

    Google Scholar 

  13. Agarwal, M., Shah, R., Mannem, P.: Automatic question generation using discourse cues. In: Proceedings of the 6th Workshop on Innovative use of NLP for Building Educational Applications, Association for Computational Linguistics (2011)

    Google Scholar 

  14. Agarwal, M.: Cloze and Open Cloze Question Generation Systems and their Evaluation Guidelines. International Institute of Information Technology, Hyderabad (2012)

    Google Scholar 

  15. Wu, J.-C., Chang, J., Chang, J.S.: Correcting serial grammatical errors based on N-grams and syntax. 中文計算語言學期刊 18(4): 31–44 (2013)

    Google Scholar 

  16. Haladyna, T.M.: Automatic item generation - a hitorical perspective. In: Gierl, M.J., Haladyna, T.M. (eds.) Automatic Item Generation. Routledge, New York (2013)

    Google Scholar 

  17. Linnebank, F., Liem, J., Bredeweg, B.: Question generation and answering. DynaLearn, Deliverable D3.3, EC FP7 STREP Project 231526 (2010)

    Google Scholar 

  18. Foulonneau, M.: Generating educational assessment items from linked open data: the case of DBpedia. In: García-Castro, R., Fensel, D., Antoniou, G. (eds.) ESWC 2011. LNCS, vol. 7117, pp. 16–27. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  19. Foulonneau, M., Grouès, G.: Common vs. expert knowledge: making the semantic web an educational model. In: 2nd International Workshop on Learning and Education with the Web of Data (LiLe–2012 at WWW-2012 conference), Lyon, France (2012)

    Google Scholar 

  20. Mitkov, R., et al.: Semantic similarity of distractors in multiple-choice tests: extrinsic evaluation. In: Workshop on Geometrical Models of Natural Language Semantics (GEMS 2009), pp. 49–56. Association for Computational Linguistics, Morristown, NJ, USA (2009)

    Google Scholar 

  21. Gütl, C., Lankmayr, K., Weinhofer, J.: Enhanced approach of automatic creation of test items to foster modern learning setting. In: 9th European Conference on e-Learning (ECEL 2010), Porto, Portugal, pp. 225–234. (2010)

    Google Scholar 

  22. Aldabe, I., Maritxalar, M.: Automatic distractor generation for domain specific texts. In: Loftsson, H., Rögnvaldsson, E., Helgadóttir, S. (eds.) IceTAL 2010. LNCS, vol. 6233, pp. 27–38. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  23. Moser, J.R., Gütl, C., Liu, W.: Refined distractor generation with LSA and stylometry for automated multiple choice question generation. In: Thielscher, M., Zhang, D. (eds.) AI 2012. LNCS, vol. 7691, pp. 95–106. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  24. Holmes, D.: The evolution of stylometry in humanities scholarship. Literary Linguist. Comput. 13(3), 111–117 (1998)

    Article  Google Scholar 

  25. Huang, Y.-T., et al.: TEDQuiz: automatic quiz generation for TED talks video clips to assess listening comprehension. In: 2014 IEEE 14th International Conference on Advanced Learning Technologies (ICALT), IEEE (2014)

    Google Scholar 

  26. Deane, P., et al.: Creating vocabulary item types that measure students’ depth of semantic knowledge. In: ETS Research Report Series, pp. 1–19 (2014)

    Google Scholar 

  27. Smith, S., Avinesh, P., Kilgarriff, A.: Gap-fill tests for language learners: corpus-driven item generation. In: International Conference on Natural Language Processing, Macmillan Publishers, India (2010)

    Google Scholar 

  28. Zesch, T., Melamud, O.: Automatic generation of challenging distractors using context-sensitive inference rules. In: Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications (2014)

    Google Scholar 

  29. Hoshino, A., Nakagawa, H.: Assisting cloze test making with a web application. In: Society for Information Technology and Teacher Education International Conference (2007)

    Google Scholar 

  30. Gierl, M.J., Lai, H.: Using weak and strong theory to create item models for automatic item generation. In: Gierl M.J., Haladyna T.M. (eds.) Automatic Item Generation. Routledge, New York (2013)

    Google Scholar 

  31. Sonnleitner, P.: Using the LLTM to evaluate an item-generating system for reading comprehension. Psychol. Sci. 50(3), 345 (2008)

    Google Scholar 

  32. Pho, V.-M., et al.: Multiple choice question corpus analysis for distractor characterization. In: 9th International Conference on Language Resources and Evaluation (LREC 2014) (2014)

    Google Scholar 

  33. Kolb, P.: Disco: a multilingual database of distributionally similar words. In: Proceedings of KONVENS-2008, Berlin (2008)

    Google Scholar 

  34. François, T., Miltsakaki, E.: Do NLP and machine learning improve traditional readability formulas? In: Proceedings of the First Workshop on Predicting and Improving Text Readability for Target Reader Populations, Association for Computational Linguistics (2012)

    Google Scholar 

  35. Washburne, C., Vogel, M.: Are any number combinations inherently difficult? J. Educ. Res. 17(4), 235–254 (1928)

    Article  Google Scholar 

  36. Dale, E., Chall, J.S.: A formula for predicting readability: Instructions. Educ. Res. Bull. 27, 37–54 (1948)

    Google Scholar 

  37. Flesch, R.: A new readability yardstick. J. Appl. Psychol. 32(3), 221 (1948)

    Article  Google Scholar 

  38. Kandel, L., Moles, A.: Application de l’indice de flesch à la langue française. J. Educ. Res. 21, 283–287 (1958)

    Google Scholar 

  39. Amstad, T.: Wie verständlich sind unsere Zeitungen? Studenten-Schreib-Service (1978)

    Google Scholar 

  40. Dale, E., Tyler, R.W.: A study of the factors influencing the difficulty of reading materials for adults of limited reading ability. Libr. Q. 4(3), 384–412 (1934)

    Article  Google Scholar 

  41. Halliday, M.A.K., Hasan, R.: Cohesion in English. Routledge, New York (2014)

    Google Scholar 

  42. Zeid, E.A., Foulonneau, M., Atéchian, T.: Réutiliser des textes dans un contexte éducatif. Document Numérique 15(3), 119–142 (2012)

    Article  Google Scholar 

  43. François, T., Fairon, C.: An AI readability formula for French as a foreign language. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Association for Computational Linguistics (2012)

    Google Scholar 

  44. Mesnager, J.: Mots fréquents ou mots usuels. Communication et langages 84(1), 33–46 (1990)

    Article  Google Scholar 

  45. François, T., Fairon, C.: Les apports du TAL à la lisibilité du français langue étrangère (2013)

    Google Scholar 

  46. Goosse, A., Grevisse, M.: Le bon usage: grammaire française, Duculot (1993)

    Google Scholar 

  47. Haladyna, T.M., Downing, S.M.: A taxonomy of multiple-choice item-writing rules. Appl. Measur. Educ. 2(1), 37–50 (1989)

    Article  Google Scholar 

Download references

Acknowledgements

The research leading to these results was funded by the INTERREG IV A “Grande Région” 2007–2013 under grant agreement No. 138 GR DELUX 2 3 274.

Author information

Authors and Affiliations

  1. Luxembourg Institute of Science and Technology, IT for Innovative Services, 5, Avenue des Hauts-Fourneaux, 4362, Esch-sur-Alzette, Luxembourg

    Alain Pfeiffer, Anna Kuraeva, Muriel Foulonneau, Younès Djaghloul, Eric Tobias & Eric Ras

Authors
  1. Alain Pfeiffer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anna Kuraeva
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Muriel Foulonneau
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Younès Djaghloul
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eric Tobias
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Eric Ras
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eric Ras .

Editor information

Editors and Affiliations

  1. Luxembourg Inst. of Science and Technol., Esch/Alzette, Luxembourg

    Eric Ras

  2. Open University of the Netherlands, Heerlen, The Netherlands

    Desirée Joosten-ten Brinke

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Pfeiffer, A., Kuraeva, A., Foulonneau, M., Djaghloul, Y., Tobias, E., Ras, E. (2015). Automatically Generated Metrics for Multilingual Inline Choice Questions on Reading Comprehension. In: Ras, E., Joosten-ten Brinke, D. (eds) Computer Assisted Assessment. Research into E-Assessment. TEA 2015. Communications in Computer and Information Science, vol 571. Springer, Cham. https://doi.org/10.1007/978-3-319-27704-2_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-27704-2_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-27703-5

  • Online ISBN: 978-3-319-27704-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation