Advertisement

The Multiple Language Question Answering Track at CLEF 2003

  • Bernardo Magnini
  • Simone Romagnoli
  • Alessandro Vallin
  • Jesús Herrera
  • Anselmo Peñas
  • Víctor Peinado
  • Felisa Verdejo
  • Maarten de Rijke
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3237)

Abstract

This paper reports on the pilot question answering track that was carried out within the CLEF initiative this year. The track was divided into monolingual and bilingual tasks: monolingual systems were evaluated within the frame of three non-English European languages, Dutch, Italian and Spanish, while in the cross-language tasks an English document collection constituted the target corpus for Italian, Spanish, Dutch, French and German queries. Participants were given 200 questions for each task, and were allowed to submit up to two runs per task with up to three responses (either exact answers or 50 byte long strings) per question.

We give here an overview of the track: we report on each task and discuss the creation of the multilingual test sets and the participants’ results.

Keywords

Bovine Spongiform Encephalopathy Document Collection Question Answering Exact Answer Question Answering System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Braschler, M., Peters, C.: The CLEF Campaigns: Evaluation of Cross-Language Information Retrieval Systems. UPGRADE (The European Online Magazine for the IT Professional) III(3) (June 2002), URL: http://www.upgrade-cepis.org/issues/2002/3/up3-3Braschler.pdf
  2. 2.
    Burger, J., Cardie, C., Chaudhri, V., Gaizauskas, R., Harabagiu, S., Israel, D., Jacquemin, C., Lin, C.-Y., Maiorano, S., Miller, G., Moldovan, D., Ogden, B., Prager, J., Riloff, E., Singhal, A., Shrihari, R., Strzalkowski, T., Voorhees, E., Weishedel, R.: Issues Tasks and Program Structures to Roadmap Research in Question & Answering (Q&A) (2001), URL: http://www-nlpir.nist.gov/projects/duc/papers/qa.Roadmap-paper_v2.doc
  3. 3.
    CLEF 2003 Question Answering Track: Guidelines for the Monolingual and Bilingual Tasks, URL: http://clef-qa.itc.it/guidelines.htm
  4. 4.
    Knott, A., Bayard, I., de Jager, S., Smith, L., Moorfield, J., O’Keefe, R.: A Question-Answering System for English and Maōri. In: Proceedings of the Fifth Biannual Conference on Artificial Neural Networks and Expert Systems (ANNES), University of Otago (November 2001)Google Scholar
  5. 5.
    Liddy, E.D.: Why are People Asking these Questions? A Call for Bringing Situation into Question-Answering System Evaluation. In: LREC Workshop Proceedings on Question Answering – Strategy and Resources, Grand Canary Island, Spain (2002)Google Scholar
  6. 6.
    Magnini, B., Negri, M., Prevete, R., Tanev, H.: Multilingual Question/Answering: the DIOGENE System. In: Proceedings of the Tenth Text REtrieval Conference (TREC 2001), Gaithersburg, MD (2001)Google Scholar
  7. 7.
    Magnini, B.: Evaluation of Cross-Language Question Answering Systems. In: Proposal presentation held at the CLEF Workshop (2002)Google Scholar
  8. 8.
    Maybury, M.: Toward a Question Answering Roadmap (2002), URL: www.mitre.org/work/tech_papers/tech_papers_02/maybury_toward
  9. 9.
    Voorhees, E.M.: The TREC-8 Question Answering Track Report. In: Proceedings of the Eighth Text REtrieval Conference (TREC-8), Gaithersburg, MD (1999)Google Scholar
  10. 10.
    Voorhees, E.M.: Overview of the TREC 2001 Question Answering Track. In: Proceedings of the Tenth Text REtrieval Conference (TREC 2001), Gaithersburg, MD (2001)Google Scholar
  11. 11.
    Voorhees, E.M.: Overview of the TREC 2002 Question Answering Track. In: Proceedings of the Eleventh Text REtrieval Conference (TREC 2002), Gaithersburg, MD (2002)Google Scholar
  12. 12.
    Voorhees, E.M., Tice, D.M.: Building a Question Answering Test Collection. In: Proceedings of SIGIR 2000, Athens, Greece (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Bernardo Magnini
    • 1
  • Simone Romagnoli
    • 1
  • Alessandro Vallin
    • 1
  • Jesús Herrera
    • 2
  • Anselmo Peñas
    • 2
  • Víctor Peinado
    • 2
  • Felisa Verdejo
    • 2
  • Maarten de Rijke
    • 3
  1. 1.ITC-irst, Centro per la Ricerca Scientifica e TecnologicaPovo (TN)Italy
  2. 2.Departamento de Lenguajes y Sistemas Informaticos, Ciudad UniversitariaUNED, Spanish Distance Learning UniversityMadridSpain
  3. 3.Language and Inference Technology Group, ILLCUniversity of AmsterdamAmsterdamThe Netherlands

Personalised recommendations