Abstract
Natural Language Processing (NLP) is an important technology that motivates the form of AI applications today. Many NLP libraries are available for researchers and developers to perform standard NLP tasks (such as segmentation, tokenization, lemmatization, POS tagging, and NER) without the need to develop from scratch. However, there are some challenges in selecting the most suitable library such as data type, performance, and the compatibility. In this paper, we assessed five popular NLP libraries for performing the standard processing tasks on datasets crawled from different online news sources in Malaysia. The obtained results are analysed and differences of those libraries are listed. The goal of this study is to provide a clear view for users to select the suitable NLP library for their text analysis task.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Al Omran, F.N.A., Treude, C.: Choosing an NLP library for analyzing software documentation: a systematic literature review and a series of experiments, pp. 187–197. IEEE Press, Piscataway (2017)
Palmar, D.D.: Text preprocessing. In: Indurkhya, N., Damerau, F.J. (eds.) Handbook of natural language processing, vol. 2, pp. 9–30. CRC Press, Boca Raton (2010)
Aker, A., Petrak, J., Sabbah, F.: An extensible multilingual open source lemmatizer. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP, pp. 40– 45. ACL (2017)
Marcus, M., Beatrice, S., Mary, A.: Building a large annotated corpus of. English: The Penn Treebank (1993)
Epaminondas, K., Tatar, D., Sacarea, C.: Named entity recognition. In: Natural Language Processing: Semantic Aspects, pp. 297–309. CRC Press, Boca Raton (2013)
Pinto, A., Gonçalo Oliveira, H., Oliveira Alves, A.: Comparing the performance of different NLP toolkits in formal and social media Text. In: 5th Symposium on Languages, Applications and Technologies (SLATE2016). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2016)
Loper, E., Bird, S.: NLTK: the natural language. arXiv preprint cs/0205028 (2002)
Foundation, T.A.: Apache OpenNLP Developer Documentation. (The Apache Software Foundation) (2011). https://opennlp.apache.org/docs/1.9.1/manual/opennlp.html, Accessed 01 Mar 2019
John Snow Labs: SparkNLP - Documentation and Reference (2019). https://nlp.johnsnowlabs.com/components.html, Accessed 11 Mar 2019
Honnibal, M.: Introducing spaCy (2016). https://explosion.ai/blog/introducing-spacy, Accessed 01 Mar 2019
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., McClosky, D.: The Stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55–60 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Weiying, K., Pham, D.N., Eftekharypour, Y., Pheng, A.J. (2019). Benchmarking NLP Toolkits for Enterprise Application. In: Nayak, A., Sharma, A. (eds) PRICAI 2019: Trends in Artificial Intelligence. PRICAI 2019. Lecture Notes in Computer Science(), vol 11672. Springer, Cham. https://doi.org/10.1007/978-3-030-29894-4_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-29894-4_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-29893-7
Online ISBN: 978-3-030-29894-4
eBook Packages: Computer ScienceComputer Science (R0)