Advertisement

Evaluation of MWE Acquisition

  • Carlos Ramisch
Chapter
Part of the Theory and Applications of Natural Language Processing book series (NLP)

Abstract

The result of automatic the MWE acquisition methods described in Sects.  3.2.1 and  3.2.2 can be viewed as a list of MWE candidates. We can evaluate the quality of a given approach for MWE acquisition by assessing the utility of the resulting MWE candidate list for a given application. This list has often an internal structure, and each candidate contains attached information, coming from corpora or from external resources. However, if we ignore this extra information (which is often the case), it is possible to define objective criteria for determining the quality of the list, and, indirectly, of the acquisition method.

Keywords

Native Speaker Manual Annotation Mean Average Precision Automatic Annotation Acquisition Method 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. Acosta O, Villavicencio A, Moreira V (2011) Identification and treatment of multiword expressions applied to information retrieval. In: Kordoni V, Ramisch C, Villavicencio A (eds) Proceedings of the ALC workshop on multiword expressions: from parsing and generation to the real world (MWE 2011), Portland. Association for Computational Linguistics, pp 101–109. http://www.aclweb.org/anthology/W/W11/W11-0815
  2. Artstein R, Poesio M (2008) Inter-coder agreement for computational linguistics. Comput Linguist 34(4):555–596CrossRefGoogle Scholar
  3. Baldwin T (2005) Deep lexical acquisition of verb-particle constructions. Comput Speech Lang Spec Issue MWEs 19(4):398–414CrossRefGoogle Scholar
  4. Baldwin T (2008) A resource for evaluating the deep lexical acquisition of English verb-particle constructions. In: Grégoire N, Evert S, Krenn B (eds) Proceedings of the LREC workshop towards a shared task for multiword expressions (MWE 2008), Marrakech, pp 1–2Google Scholar
  5. Bonin F, Dell’Orletta F, Venturi G, Montemagni S (2010) Contrastive filtering of domain-specific multi-word terms from different types of corpora. In: Laporte É, Nakov P, Ramisch C, Villavicencio A (eds) Proceedings of the COLING workshop on multiword expressions: from theory to applications (MWE 2010), Beijing. Association for Computational Linguistics, pp 76–79Google Scholar
  6. Carpuat M, Diab M (2010) Task-based evaluation of multiword expressions: a pilot study in statistical machine translation. In: Proceedings of human language technology: the 2010 annual conference of the North American chapter of the Association for Computational Linguistics (NAACL 2003), Los Angeles. Association for Computational Linguistics, pp 242–245. http://www.aclweb.org/anthology/N10-1029
  7. Church K (2011) How many multiword expressions do people know? In: Kordoni V, Ramisch C, Villavicencio A (eds) Proceedings of the ALC workshop on multiword expressions: from parsing and generation to the real world (MWE 2011), Portland. Association for Computational Linguistics, pp 137–144. http://www.aclweb.org/anthology/W/W11/W11-0823
  8. Cook P, Fazly A, Stevenson S (2007) Pulling their weight: exploiting syntactic forms for the automatic identification of idiomatic expressions in context. In: Grégoire N, Evert S, Kim SN (eds) Proceedings of the ACL workshop on a broader perspective on multiword expressions (MWE 2007), Prague. Association for Computational Linguistics, pp 41–48. http://www.aclweb.org/anthology/W/W07/W07-1106
  9. Cook P, Fazly A, Stevenson S (2008) The VNC-tokens dataset. In: Grégoire N, Evert S, Krenn B (eds) Proceedings of the LREC workshop towards a shared task for multiword expressions (MWE 2008), Marrakech, pp 19–22Google Scholar
  10. da Silva JF, Dias G, Guilloré S, Lopes JGP (1999) Using localmaxs algorithm for the extraction of contiguous and non-contiguous multiword lexical units. In: Proceedings of the 9th Portuguese conference on artificial intelligence: progress in artificial intelligence, (EPIA 1999), Évora. Springer, London, pp 113–132. http://dl.acm.org/citation.cfm?id=645377.651205
  11. Doucet A, Ahonen-Myka H (2004) Non-contiguous word sequences for information retrieval. In: Tanaka T, Villavicencio A, Bond F, Korhonen A (eds) Proceedings of the ACL workshop on multiword expressions: integrating processing (MWE 2004), Barcelona. Association for Computational Linguistics, pp 88–95Google Scholar
  12. Dunning T (1993) Accurate methods for the statistics of surprise and coincidence. Comput Linguist 19(1):61–74Google Scholar
  13. Duran MS, Ramisch C (2011) How do you feel? investigating lexical-syntactic patterns in sentiment expression. In: Proceedings of corpus linguistics 2011: discourse and corpus linguistics conference, BirminghamGoogle Scholar
  14. Eugenio BD, Glass M (2004) The kappa statistic: a second look. Comput Linguist 30(1):95–101CrossRefzbMATHGoogle Scholar
  15. Evert S (2004) The statistics of word cooccurrences: word pairs and collocations. PhD thesis, Institut für maschinelle Sprachverarbeitung, University of Stuttgart, Stuttgart, 353pGoogle Scholar
  16. Evert S, Krenn B (2005) Using small random samples for the manual evaluation of statistical association measures. Comput Speech Lang Spec Issue MWEs 19(4):450–466CrossRefGoogle Scholar
  17. Fazly A, Stevenson S (2007) Distinguishing subtypes of multiword expressions using linguistically-motivated statistical measures. In: Grégoire N, Evert S, Kim SN (eds) Proceedings of the ACL workshop on a broader perspective on multiword expressions (MWE 2007), Prague. Association for Computational Linguistics, pp 9–16. http://www.aclweb.org/anthology/W/W07/W07-1102
  18. Finlayson M, Kulkarni N (2011) Detecting multi-word expressions improves word sense disambiguation. In: Kordoni V, Ramisch C, Villavicencio A (eds) Proceedings of the ALC workshop on multiword expressions: from parsing and generation to the real world (MWE 2011), Portland. Association for Computational Linguistics, pp 20–24. http://www.aclweb.org/anthology/W/W11/W11-0805
  19. Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5):378–382CrossRefGoogle Scholar
  20. Fritzinger F, Weller M, Heid U (2010) A survey of idiomatic preposition-noun-verb triples on token level. In: Proceedings of the seventh international conference on language resources and evaluation (LREC 2010), Valetta. European Language Resources Association, pp 2908–2914Google Scholar
  21. Green S, de Marneffe MC, Bauer J, Manning CD (2011) Multiword expression identification with tree substitution grammars: a parsing tour de force with French. In: Barzilay R, Johnson M (eds) Proceedings of the 2011 conference on empirical methods in natural language processing (EMNLP 2011), Edinburgh. Association for Computational Linguistics, pp 725–735. http://www.aclweb.org/anthology/D11-1067
  22. Grégoire N, Evert S, Krenn B (eds) (2008) Proceedings of the LREC workshop towards a shared task for multiword expressions (MWE 2008), Marrakech. http://www.lrec-conf.org/proceedings/lrec2008/workshops/W20_Proceedings.pdf, 57p
  23. Justeson JS, Katz SM (1995) Technical terminology: some linguistic properties and an algorithm for identification in text. Nat Lang Eng 1(1):9–27CrossRefGoogle Scholar
  24. Krenn B (2008) Description of evaluation resource – German PP-verb data. In: Grégoire N, Evert S, Krenn B (eds) Proceedings of the LREC workshop towards a shared task for multiword expressions (MWE 2008), Marrakech, pp 7–10Google Scholar
  25. Langer S (2004) A linguistic test battery for support verb constructions. Spec Issue Linguist Investig 27(2):171–184CrossRefGoogle Scholar
  26. Laporte É, Voyatzi S (2008) An electronic dictionary of French multiword adverbs. In: Grégoire N, Evert S, Krenn B (eds) Proceedings of the LREC workshop towards a shared task for multiword expressions (MWE 2008), Marrakech, pp 31–34Google Scholar
  27. Lee L, Aw A, Zhang M, Li H (2010) EM-based hybrid model for bilingual terminology extraction from comparable corpora. In: Huang CR, Jurafsky D (eds) Proceedings of the 23rd international conference on computational linguistics (COLING 2010)—Posters, The Coling 2010 Organizing Committee, Beijing, pp 639–646. http://www.aclweb.org/anthology/C10-2073
  28. Linardaki E, Ramisch C, Villavicencio A, Fotopoulou A (2010) Towards the construction of language resources for Greek multiword expressions: extraction and evaluation. In: Piperidis S, Slavcheva M, Vertan C (eds) Proceedings of the LREC workshop on exploitation of multilingual resources and tools for central and (South) Eastern European languages, Valetta, pp 31–40Google Scholar
  29. McCarthy D, Keller B, Carroll J (2003) Detecting a continuum of compositionality in phrasal verbs. In: Bond F, Korhonen A, McCarthy D, Villavicencio A (eds) Proceedings of the ACL workshop on multiword expressions: analysis, acquisition and treatment (MWE 2003), Sapporo. Association for Computational Linguistics, pp 73–80. doi:10.3115/1119282.1119292, http://www.aclweb.org/anthology/W03-1810
  30. Nakov P (2008) Paraphrasing verbs for noun compound interpretation. In: Grégoire N, Evert S, Krenn B (eds) Proceedings of the LREC workshop towards a shared task for multiword expressions (MWE 2008), Marrakech, pp 46–49Google Scholar
  31. Nicholson J, Baldwin T (2008) Interpreting compound nominalisations. In: Grégoire N, Evert S, Krenn B (eds) Proceedings of the LREC workshop towards a shared task for multiword expressions (MWE 2008), Marrakech, pp 43–45Google Scholar
  32. Pal S, Naskar SK, Pecina P, Bandyopadhyay S, Way A (2010) Handling named entities and compound verbs in phrase-based statistical machine translation. In: Laporte É, Nakov P, Ramisch C, Villavicencio A (eds) Proceedings of the COLING workshop on multiword expressions: from theory to applications (MWE 2010), Beijing. Association for Computational Linguistics, pp 45–53Google Scholar
  33. Pearce D (2002) A comparative evaluation of collocation extraction techniques. In: Proceedings of the third international conference on language resources and evaluation (LREC 2002), Las Palmas. European Language Resources Association, pp 1530–1536Google Scholar
  34. Pecina P (2005) An extensive empirical study of collocation extraction methods. In: Proceedings of the ACL 2005 student research workshop, Ann Arbor. Association for Computational Linguistics, pp 13–18. http://www.aclweb.org/anthology/P/P05/P05-2003
  35. Pedersen T (1996) Fishing for exactness. In: Proceedings of the south-central SAS users group conference (SCSUG-96), Austin, pp 188–200Google Scholar
  36. Ramisch C (2009) Multiword terminology extraction for domain-specific documents. Master’s thesis, École Nationale Supérieure d’Informatique et de Mathématiques Appliquées, Grenoble, 79pGoogle Scholar
  37. Ramisch C (2012) A generic framework for multiword expressions treatment: from acquisition to applications. In: Proceedings of the ACL 2012 student research workshop, Jeju. Association for Computational Linguistics, pp 61–66. http://www.aclweb.org/anthology/W12-3311
  38. Ramisch C, Schreiner P, Idiart M, Villavicencio A (2008a) An evaluation of methods for the extraction of multiword expressions. In: Grégoire N, Evert S, Krenn B (eds) Proceedings of the LREC workshop towards a shared task for multiword expressions (MWE 2008), Marrakech, pp 50–53Google Scholar
  39. Ramisch C, Villavicencio A, Moura L, Idiart M (2008b) Picking them up and figuring them out: verb-particle constructions, noise and idiomaticity. In: Clark A, Toutanova K (eds) Proceedings of the twelfth conference on natural language learning (CoNLL 2008), The Coling 2008 Organizing Committee, Manchester, pp 49–56. http://www.aclweb.org/anthology/W08-2107
  40. Ramisch C, Villavicencio A, Boitet C (2010) Web-based and combined language models: a case study on noun compound identification. In: Huang CR, Jurafsky D (eds) Proceedings of the 23rd international conference on computational linguistics (COLING 2010)—Posters, The Coling 2010 Organizing Committee, Beijing, pp 1041–1049. http://www.aclweb.org/anthology/C10-2120
  41. Schone P, Jurafsky D (2001) Is knowledge-free induction of multiword unit dictionary headwords a solved problem? In: Lee L, Harman D (eds) Proceedings of the 2001 conference on empirical methods in natural language processing (EMNLP 2001), Pittsburgh. Association for Computational Linguistics, pp 100–108Google Scholar
  42. Seretan V (2008) Collocation extraction based on syntactic parsing. PhD thesis, University of Geneva, Geneva, 249pGoogle Scholar
  43. Venkatsubramanyan S, Perez-Carballo J (2004) Multiword expression filtering for building knowledge. In: Tanaka T, Villavicencio A, Bond F, Korhonen A (eds) Proceedings of the ACL workshop on multiword expressions: integrating processing (MWE 2004), Barcelona. Association for Computational Linguistics, pp 40–47Google Scholar
  44. Villavicencio A, Bond F, Korhonen A, McCarthy D (2005) Introduction to the special issue on multiword expressions: having a crack at a hard nut. Comput Speech Lang Spec Issue MWEs 19(4):365–377CrossRefGoogle Scholar
  45. Villavicencio A, Idiart M, Ramisch C, Araujo VD, Yankama B, Berwick R (2012) Get out but don’t fall down: verb-particle constructions in child language. In: Berwick R, Korhonen A, Poibeau T, Villavicencio A (eds) Proceedings of the EACL 2012 workshop on computational models of language acquisition and loss, Avignon. Association for Computational Linguistics, pp 43–50Google Scholar
  46. Xu Y, Goebel R, Ringlstetter C, Kondrak G (2010) Application of the tightness continuum measure to Chinese information retrieval. In: Laporte É, Nakov P, Ramisch C, Villavicencio A (eds) Proceedings of the COLING workshop on multiword expressions: from theory to applications (MWE 2010), Beijing. Association for Computational Linguistics, pp 54–62Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Carlos Ramisch
    • 1
  1. 1.Aix Marseille UniversityMarseilleFrance

Personalised recommendations