Abstract
This article describes an automatic evaluation procedure for NLP system robustness under the strain of noisy and ill-formed input. The procedure requires no manual work or annotated resources. It is language and annotation scheme independent and produces reliable estimates on the robustness of NLP systems. The only requirement is an estimate on the NLP system accuracy. The procedure was applied to five parsers and one part-of-speech tagger on Swedish text. To establish the reliability of the procedure, a comparative evaluation involving annotated resources was carried out on the tagger and three of the parsers.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Carroll, J., Briscoe, T., Sanfilippo, A.: Parser evaluation: a survey and a new proposal. In: Proceedings of LREC 1998, Granada, Spain, pp. 447–454 (1998)
Hogenhout, W.I., Matsumoto, Y.: Towards a more careful evaluation of broad coverage parsing systems. In: Proceedings of Coling 1996, San Francisco, USA, pp. 562–567 (1996)
Menzel, W.: Robust processing of natural language. In: Proceedings of 19th Annual German Conference on Artificial Intelligence, Berlin, Germany, pp. 19–34 (1995)
Bigert, J., Knutsson, O., Sjöbergh, J.: Automatic evaluation of robustness and degradation in tagging and parsing. In: Proceedings of RANLP 2003, Bovorets, Bulgaria (2003)
Vilares, M., Darriba, V.M., Vilares, J., Rodriguez, R.: Robust parsing using dynamic programming. In: Ibarra, O.H., Dang, Z. (eds.) CIAA 2003. LNCS, vol. 2759, pp. 258–268. Springer, Heidelberg (2003)
Foster, J.: Parsing ungrammatical input: An evaluation procedure. In: Proceedings of LREC 2004, Lisbon, Portugal, pp. 2039–2042 (2004)
Li, X., Roth, D.: Exploring evidence for shallow parsing. In: Daelemans, W., Zajac, R. (eds.) Proceedings of CoNLL 2001, Toulouse, France, pp. 38–44 (2001)
Bigert, J., Ericson, L., Solis, A.: Missplel and AutoEval: Two generic tools for automatic evaluation. In: Proceedings of Nodalida 2003, Reykjavik, Iceland (2003)
Knutsson, O., Bigert, J., Kann, V.: A robust shallow parser for Swedish. In: Proceedings of Nodalida 2003, Reykjavik, Iceland (2003)
Voutilainen, A.: Parsing Swedish. In: Proceedings of Nodalida 2001, Uppsala, Sweden (2001)
Nivre, J.: An efficient algorithm for projective dependency parsing. In: Proceedings of IWPT 2003, Nancy, France, pp. 149–160 (2003)
Nivre, J., Hall, J., Nilsson, J.: Memory-based dependency parsing. In: Proceedings of CoNLL, Boston, MA (2004)
Megyesi, B.: Data-Driven Syntactic Analysis – Methods and Applications for Swedish. PhD thesis, KTH, Stockholm, Sweden (2002)
Ejerhed, E., Källgren, G., Wennstedt, O., Åström, M.: The Linguistic Annotation System of the Stockholm-Umeå Project. Department of Linguistics, University of Umeå, Sweden (1992)
Brants, T.: TnT – a statistical part-of-speech tagger. In: Proceedings of ANLP 2000, Seattle, USA (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bigert, J., Sjöbergh, J., Knutsson, O., Sahlgren, M. (2005). Unsupervised Evaluation of Parser Robustness. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2005. Lecture Notes in Computer Science, vol 3406. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30586-6_14
Download citation
DOI: https://doi.org/10.1007/978-3-540-30586-6_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-24523-0
Online ISBN: 978-3-540-30586-6
eBook Packages: Computer ScienceComputer Science (R0)