Skip to main content
SpringerLink
Log in

The Recognizing Textual Entailment Challenges: Datasets and Methodologies

  • Chapter
  • First Online:
Handbook of Linguistic Annotation

Abstract

While semantic inference has always been a major focus in Computational Linguistics, the topic has benefited of new attention in the field thanks to the Recognizing Textual Entailment (RTE) framework, first launched in 2004, which has provided an operational definition of entailment based on human judgements over portions of text. On top of such definition, a task has been designed, which includes both guidelines for dataset annotation and evaluation metrics for assessing systems’ performance. This chapter presents the successful experience of creating Textual Entailment datasets. We show how, during the years, RTE datasets have been developed in several variants, not only to address complex phenomena underlying entailment, but also to demonstrate the potential application of entailment inference into concrete scenarios, including summarization, knowledge base population, answer validation for question answering, and student answer assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 349.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 449.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 449.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.pascal-network.org/.

  2. 2.

    http://www.nist.gov/tac/tracks/index.html.

  3. 3.

    http://www.cs.york.ac.uk/semeval-2013/task7/.

  4. 4.

    www.aclweb.org/aclwiki/index.php?title=Textual_Entailment_Resource_Pool.

  5. 5.

    For details and examples regarding the T-H pair collection methodology followed for each of the applications considered in the datasets, see the organizers’ overview papers [3, 14, 21, 22].

  6. 6.

    The complete annotation guidelines can be found at: http://www.nist.gov/tac/2009/RTE/RTE5_Main_Guidelines.pdf.

  7. 7.

    The retrieval threshold was set such that, on average, about 80% of the actually-entailing sentences in the document cluster would be included among the retrieved candidate sentences.

  8. 8.

    http://aclweb.org/aclwiki/index.php?title=Textual_Entailment_Resource_Pool#Other_data_sets.

  9. 9.

    http://nlp.uned.es/clef-qa/ave/.

  10. 10.

    http://nlp.uned.es/clef-qa/.

  11. 11.

    http://pete.yuret.com.

  12. 12.

    http://www.evalita.it/2009.

  13. 13.

    http://www.cs.york.ac.uk/semeval-2013/task8/.

  14. 14.

    http://research.nii.ac.jp/ntcir/ntcir-11/data.html#rite.

  15. 15.

    http://artigas.lti.cs.cmu.edu/rite/Resources.

  16. 16.

    http://alt.qcri.org/semeval2014/task1/.

  17. 17.

    http://www-nlp.stanford.edu/projects/contradiction/.

  18. 18.

    https://hlt.fbk.eu/technologies/rte-3-ita.

  19. 19.

    http://www.excitement-project.eu/index.php/results.

  20. 20.

    https://github.com/hltfbk/EOP-1.2.1/wiki/Data-Sets.

  21. 21.

    https://hlt-nlp.fbk.eu/technologies/textual-entailment-graph-dataset.

References

  1. Abad, A., Bentivogli, L., Dagan, I., Giampiccolo, D., Mirkin, S., Pianta, E., Stern, A.: A resource for investigating the impact of anaphora and coreference on inference. In: Language Resources and Evaluation Conference (LREC-2010) (2010)

    Google Scholar 

  2. Adler, M., Berant, J., Dagan, I.: Entailment-based text exploration with application to the health-care domain. In: Proceedings of the ACL Demo Session (2012)

    Google Scholar 

  3. Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., Szpektor, I.: The second pascal recognising textual entailment challenge. In: Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. Venice, Italy (2006)

    Google Scholar 

  4. Bentivogli, L., Dagan, I., Dang, H., Giampiccolo, D., Leggio, M.L., Magnini, B.: Considering discourse references in textual entailment annotation. In: 5th International Conference on Generative Approaches to the Lexicon (GL 2009), Pisa, Italy (2009)

    Google Scholar 

  5. Bentivogli, L., Dagan, I., Dang, H.T., Giampiccolo, D., Magnini, B.: The fifth PASCAL recognizing textual entailment challenge. In: Proceedings of the Text Analysis Conference (TAC 2009) (2009)

    Google Scholar 

  6. Bentivogli, L., Cabrio, E., Dagan, I., Giampiccolo, D., Leggio, M.L., Magnini, B.: Building textual entailment specialized data sets: a methodology for isolating linguistic phenomena relevant to inference. In: LREC (2010)

    Google Scholar 

  7. Bentivogli, L., Clark, P., Dagan, I., Dang, H.T., Giampiccolo, D.: The sixth PASCAL recognizing textual entailment challenge. In: Proceedings of the Text Analysis Conference (TAC 2010) (2010)

    Google Scholar 

  8. Bentivogli, L., Clark, P., Dagan, I., Dang, H.T., Giampiccolo, D.: The seventh PASCAL recognizing textual entailment challenge. In: Proceedings of the Text Analysis Conference (TAC 2011) (2011)

    Google Scholar 

  9. Bentivogli, L., Magnini, B.: An Italian dataset of textual entailment graphs for text exploration of customer interactions. In: Proceedings of First Italian Conference on Computational Linguistics (CLiC-it 2014), pp. 63–66, Pisa, Italy (2014)

    Google Scholar 

  10. Berant, J., Dagan, I., Goldberger, J.: Global learning of focused entailment graphs. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 1220–1229, Uppsala, Sweden (2010)

    Google Scholar 

  11. Cabrio, E., Magnini, B.: Decomposing Semantic Inferences. LILT - Linguistic Issues in Language Technology, Special issue on Semantic Inferences (2013)

    Google Scholar 

  12. Clark, P., Harrison, P., Balasubramanian, N.: Answering biology questions using textual reasoning. In: Proceedings of the Pacific Northwest Regional NLP Workshop (NW-NLP 2012) (2012)

    Google Scholar 

  13. Clark, P., Harrison, P., Yao, X.: An entailment-based approach to the qa4mre challenge. In: Proceedings of CLEF 2012 (Conference and Labs of the Evaluation Forum) - QA4MRE Lab (2012)

    Google Scholar 

  14. Dagan, I., Glickman, O., Magnini, B.: The pascal recognising textual entailment challenge. In: Quinonero, J., et al. (eds.) Machine Learning Challenges. Lecture Notes in Computer Science, vol. 3944, pp. 177–190. Springer, Milan (2006)

    Google Scholar 

  15. Dzikovska, M.O., Moore, J.D., Steinhauser, N., Campbell, G., Farrow, E., Callaway, C.B.: Beetle II: a system for tutoring and computational linguistics experimentation. In: Proceedings of the ACL 2010 System Demonstrations, pp. 13–18 (2010)

    Google Scholar 

  16. Dzikovska, M.O., Nielsen, R.D., Brew, C.: Towards effective tutorial feedback for explanation questions: a dataset and baselines. In: Proceedings of the 2012 Conference of NAACL: Human Language Technologies, pp. 200–210 (2012)

    Google Scholar 

  17. Dzikovska, M., Nielsen, R., Brew, C., Leacock, C., Giampiccolo, D., Bentivogli, L., Clark, P., Dagan, I., Dang, H.T.: Semeval-2013 task 7: the joint student response analysis and 8th recognizing textual entailment challenge. In: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), Second Joint Conference on Lexical and Computational Semantics (*SEM), vol. 2, Atlanta, Georgia, USA (2013)

    Google Scholar 

  18. Ferrández, Ó., Spurk, C., Kouylekov, M., Dornescu, I., Ferrández, S., Negri, M., Izquierdo, R., Toms, D., Orasan, C., Neumann, G., Magnini, B., Vicedo, J.L.: The qall-me framework: a specifiable-domain multilingual question answering architecture. Web Semanti. Sci. Serv. Agents World Wide Web 9(2), 137–145 (2011). doi:10.1016/j.websem.2011.01.002. http://www.sciencedirect.com/science/article/pii/S1570826811000126

    Article  Google Scholar 

  19. Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychol. Bull. 76(5), 378–382 (1971)

    Article  Google Scholar 

  20. Garoufi, K.: Towards a Better Understanding of Applied Textual Entailment. Master thesis. Saarland University, Saarbrucken, Germany (2007)

    Google Scholar 

  21. Giampiccolo, D., Magnini, B., Dagan, I., Dolan, B.: The third pascal recognizing textual entailment challenge. In: Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 1–9. Association for Computational Linguistics, Prague (2007). http://www.aclweb.org/anthology/W/W07/W07-1401

  22. Giampiccolo, D., Dang, H.T., Magnini, B., Dagan, I., Dolan, B.: The fourth PASCAL recognizing textual entailment challenge. In: Proceedings of the Text Analysis Conference (TAC 2008) (2008)

    Google Scholar 

  23. Harabagiu, S., Hickl, A.: Methods for using textual entailment in open-domain question answering. In: Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pp. 905–912. Association for Computational Linguistics, Sydney (2006). doi:10.3115/1220175.1220289. http://www.aclweb.org/anthology/P06-1114

  24. Harabagiu, S., Hickl, A., Lacatusu, F.: Satisfying information needs with multi-document summaries. Inf. Process. Manag. 43(6), 1619–1642 (2007). doi:10.1016/j.ipm.2007.01.004. http://www.sciencedirect.com/science/article/B6VC8-4N7YH7R-2/2/37401872a230e527648845fd8aa81908

  25. Ji, H., Grishman, R., Dang, H.T., Griffitt, K., Ellis, J.: Overview of the tac 2010 knowledge base population track. In: The Text Analysis Conference (TAC 2010) (2010)

    Google Scholar 

  26. Kotlerman, L., Dagan, I., Magnini, B., Bentivogli, L.: Textual entailment graphs. J. Nat. Lang. Eng. Spec. Issue Graphs NLP 21, 699–724 (2015)

    Article  Google Scholar 

  27. Landis, J., Koch, G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977)

    Article  Google Scholar 

  28. Marelli, M., Bentivogli, L., Baroni, M., Bernardi, R., Menini, S., Zamparelli, R.: Semeval-2014 task 1: evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014) (2014)

    Google Scholar 

  29. Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., Zamparelli, R.: A sick cure for the evaluation of compositional distributional semantic models. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014) (2014)

    Google Scholar 

  30. McNamee, P., Dang, H.T.: Overview of the tac 2009 knowledge base population track. In: The Text Analysis Conference (TAC 2009) (2009)

    Google Scholar 

  31. Mirkin, S., Specia, L., Cancedda, N., Dagan, I., Dymetman, M., Szpektor, I.: Source-language entailment modeling for translating unknown terms. In: Proceedings of ACL-IJCNLP (2009)

    Google Scholar 

  32. Miyao, Y., Shima, H., Kanayama, H., Mitamura, T.: Evaluating textual entailment recognition for university entrance examinations. ACM Trans. Asian Lang. Inf. Process. (TALIP) 11(4), 13 (2012)

    Google Scholar 

  33. Monz, C., Nastase, V., Negri, M., Fahrni, A., Mehdad, Y., Strube, M.: Cosyne: a framework for multilingual content synchronization of wikis. In: Proceedings of the 7th International Symposium on Wikis and Open Collaboration, pp. 217–218. ACM (2011)

    Google Scholar 

  34. Negri, M., Kouylekov, M., Magnini, B., Mehdad, Y., Cabrio, E.: Towards extensible textual entailment engines: the edits package. In: Proceedings of the 11th Conference of the Italian Association for Artificial Intelligence (AI*IA) (2009)

    Google Scholar 

  35. Negri, M., Mehdad, Y.: Creating a Bi-lingual entailment corpus through translations with mechanical turk: $100 for a 10-day rush. In: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazonś Mechanical Turk (2010)

    Google Scholar 

  36. Negri, M., Bentivogli, L., Mehdad, Y., Giampiccolo, D., Marchetti, A.: Divide and conquer: crowdsourcing the creation of cross-lingual textual entailment corpora. In: Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP 2011) (2011)

    Google Scholar 

  37. Negri, M., Marchetti, A., Mehdad, Y., Bentivogli, L., Giampiccolo, D.: Semeval-2012 task 8: cross-lingual textual entailment for content synchronization. In: Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012) (2012)

    Google Scholar 

  38. Negri, M., Marchetti, A., Mehdad, Y., Bentivogli, L., Giampiccolo, D.: Semeval-2013 task 8: cross-lingual textual entailment for content synchronization. In: Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013) (2013)

    Google Scholar 

  39. Nielsen, R.D., Ward, W., Martin, J.H., Palmer, M.: Annotating students’ understanding of science concepts. In: Sixth International Language Resources and Evaluation Conference, (LREC 2008) (2008)

    Google Scholar 

  40. Padó, S., Galley, M., Jurafsky, D., Manning, C.: Robust machine translation evaluation with entailment features. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: vol. 1 - vol. 1, ACL 2009, pp. 297–305. Association for Computational Linguistics, Stroudsburg (2009). http://dl.acm.org/citation.cfm?id=1687878.1687922

  41. Pado, S., Noh, T.G., Stern, A., Wang, R., Zanoli, R.: Design and realization of a modular architecture for textual entailment. Nat. Lang. Eng. FirstView, 1–34 (2013). doi:10.1017/S1351324913000351. http://journals.cambridge.org/article_S1351324913000351

  42. Peñas, A., Rodrigo, Á., Sama, V., Verdejo, F.: Overview of the answer validation exercise 2006. In: CLEF, pp. 257–264 (2006)

    Google Scholar 

  43. Peñas, A., Magnini, B., Forner, P., Sutcliffe, R., Giampiccolo, D., Rodrigo, Á.: Question answering at the cross-language evaluation forum 20032010. J. Lang. Resour. Eval. 46(2), 177–217 (2012). doi:10.1007/s10579-012-9177-0

    Article  Google Scholar 

  44. Romano, L., Kouylekov, M., Szpektor, I., Dagan, I., Lavelli, A.: Investigating a generic paraphrase-based approach for relation extraction. In: 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2006), pp. 409–416. European Chapter of the Association for Computational Linguistics, Trento (2006). http://acl.ldc.upenn.edu/E/E06/E06-1052.pdf

  45. Roth, D., Sammons, M., Vydiswaran, V.: A framework for entailed relation recognition. In: Proceedings of Annual Meeting of the Association of Computational Linguistics (2009)

    Google Scholar 

  46. Sammons, M., Vydiswaran, V., Roth, D.: Ask not what textual entailment can do for you.... In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 1199–1208, Uppsala, Sweden (2010). http://www.aclweb.org/anthology/P10-1122

  47. Siegel, S., Castellan, N.J.: Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill, New York (1988)

    Google Scholar 

  48. Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, pp. 254–263. Association for Computational Linguistics, Stroudsburg (2008). http://dl.acm.org/citation.cfm?id=1613715.1613751

  49. Stern, A., Dagan, I.: A confidence model for syntactically-motivated entailment proofs. In: Proceedings of Recent Advances in Natural Language Processing, pp. 455–462. Hissar, Bulgaria (2011)

    Google Scholar 

  50. Stern, A., Stern, R., Dagan, I., Felner, A.: Efficient search for transformation-based inference. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers, vol. 1, pp. 283–291. Association for Computational Linguistics (2012)

    Google Scholar 

  51. Toledo, A., Alexandropoulou, S., Katrenko, S., Klockmann, H., Kokke, P., Winter, Y.: Semantic annotation of textual entailment. In: Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers, pp. 240–251. Association for Computational Linguistics, Potsdam (2013). http://www.aclweb.org/anthology/W13-0121

  52. Wang, R., Callison-Burch, C.: Cheap facts and counter-facts. In: Proceedings of the NAACL 2010 Workshop on Creating Speech and Language Data With Amazons Mechanical Turk (2010)

    Google Scholar 

  53. Zanzotto, F.M., Pennacchiotti, M., Tsioutsiouliklis, K.: Linguistic redundancy in twitter. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 659–669. Association for Computational Linguistics (2011)

    Google Scholar 

Download references

Acknowledgements

All the RTE challenges were partially supported by the European PASCAL and PASCAL-2 Networks of Excellence (IST-2002-506778, ICT-216886-NOE). This work was partially supported by the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 287923 (EXCITEMENT). We would like to acknowledge the precious contribution of the organizers of the RTE Challenges: Oren Glickman, Roy Bar-Haim, Lisa Ferro, Idan Szpektor, and especially Danilo Giampiccolo, Hoa Trang Dang, Peter Clark, and Bill Dolan, who were actively involved in several rounds of the Challenge. We also thank CELCT and NIST annotators, without whose dedication this successful initiative would not have been possible.

Author information

Authors and Affiliations

  1. Fondazione Bruno Kessler, Trento, Italy

    Luisa Bentivogli & Bernardo Magnini

  2. Bar-Ilan University, Ramat Gan, Israel

    Ido Dagan

Authors
  1. Luisa Bentivogli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ido Dagan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bernardo Magnini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luisa Bentivogli .

Editor information

Editors and Affiliations

  1. Department of Computer Science, Vassar College, Poughkeepsie, New York, USA

    Nancy Ide

  2. Department of Computer Science, Volen Center for Complex Systems, Brandeis University, Waltham, Massachusetts, USA

    James Pustejovsky

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer Science+Business Media Dordrecht

About this chapter

Cite this chapter

Bentivogli, L., Dagan, I., Magnini, B. (2017). The Recognizing Textual Entailment Challenges: Datasets and Methodologies. In: Ide, N., Pustejovsky, J. (eds) Handbook of Linguistic Annotation. Springer, Dordrecht. https://doi.org/10.1007/978-94-024-0881-2_42

Download citation

  • DOI: https://doi.org/10.1007/978-94-024-0881-2_42

  • Published:

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-94-024-0879-9

  • Online ISBN: 978-94-024-0881-2

  • eBook Packages: Social SciencesSocial Sciences (R0)

Publish with us

Policies and ethics

Search

Navigation