Abstract
According to some estimations (e.g. [9]), web corpora contain over 6% of foreign material (borrowings, language mixing, named entities). Since annotation pipelines are usually built upon standard and correct data, the resulting annotation of web corpora often contains serious errors.
We studied in depth annotation errors of the web corpus czTenTen 12 and proposed an extension to the tagger desamb that had been used for czTenTen annotation. First, the subcorpus was made using the most problematic documents from czTenTen. Second, measures were established for the most frequent annotation errors. Third, we established several experiments in which we extended the annotation pipeline so it could annotate foreign material and multi-word expressions. Finally, we compared the new annotations of the subcorpus with the original ones.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Benko, V.: Language code switching in web corpora. In: The 11th Workshop on Recent Advances in Slavonic Natural Languages Processing, RASLAN 2017, Karlova Studanka, Czech Republic, 1–3 December 2017, pp. 97–105 (2017). http://nlp.fi.muni.cz/raslan/2017/paper11-Benko.pdf
Constant, M., Eryiit, G., Monti, J., Plas, L.V.D., Ramisch, C., Rosner, M., Todirascu, A.: Multiword expression processing: a survey. Comput. Linguist. 0(ja), 1–92 (2017). https://doi.org/10.1162/COLI_a_00302
Crystal, D.: Language and the Internet. Cambridge University Press (2006). https://books.google.cz/books?id=cnhnO0AO45AC
Jakubek, M., Kilgarriff, A., Kov, V., Rychl, P., Suchomel, V.: The TenTen Corpus Family. In: 7th International Corpus Linguistics Conference CL 2013, Lancaster, pp. 125–127 (2013). http://ucrel.lancs.ac.uk/cl2013/
Kilgarriff, A., Grefenstette, G.: Introduction to the special issue on the web as corpus. Comput. Linguist. 29(3), 333–347 (2003). https://doi.org/10.1162/089120103322711569
Kilgarriff, A., Rychl, P., Smr, P., Tugwell, D.: The sketch engine. In: Proceedings of the Eleventh EURALEX International Congress, pp. 105–116 (2004). http://www.fit.vutbr.cz/research/view_pub.php?id=7703
Nevilov, Z.: Annotation of multi-word expressions in czech texts. In: Hork, A., Rychl, P., Rambousek, A. (eds.) Ninth Workshop on Recent Advances in Slavonic Natural Language Processing, pp. 103–112. Tribun EU, Brno (2015)
Schfer, R.: Processing and querying large web corpora with the COW14 architecture. In: Baski, P., Biber, H., Breiteneder, E., Kupietz, M., Lngen, H., Witt, A. (eds.) Proceedings of Challenges in the Management of Large Corpora 3 (CMLC-3). IDS, Lancaster (2015). http://rolandschaefer.net/?p=749
Schfer, R., Bildhauer, F.: Web corpus construction. Synth. Lect. Hum. Lang. Technol. (2013). https://doi.org/10.2200/S00508ED1V01Y201305HLT022
Taul, M., et al.: Spanish treebank annotation of informal non-standard web text. In: Daniel, F., Diaz, O. (eds.) Current Trends in Web Engineering, pp. 15–27. Springer International Publishing, Cham (2015). https://doi.org/10.1007/978-3-319-24800-4_2
Šmerk, P.: Unsupervised learning of rules for morphological disambiguation. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2004. LNCS (LNAI), vol. 3206, pp. 211–216. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30120-2_27
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Peliknov, Z., Nevilov, Z. (2018). Corpus Annotation Pipeline for Non-standard Texts. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds) Text, Speech, and Dialogue. TSD 2018. Lecture Notes in Computer Science(), vol 11107. Springer, Cham. https://doi.org/10.1007/978-3-030-00794-2_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-00794-2_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00793-5
Online ISBN: 978-3-030-00794-2
eBook Packages: Computer ScienceComputer Science (R0)