Abstract
This paper aims at developing and designing a social media based text corpus of San’ani Dialect (SMTCSD). The corpus is considered the first in the research area that codifies one of the most popular and spoken dialects in Yemen representing nearly 30% of Yemeni speakers. Our primary objective is a compilation of authentic and unmodified texts gathered from different open-source social media platforms mainly Facebook and Telegram Apps. As a result, we obtained a corpus of 447,401 tokens and 51,073 types with an 11.42% Token:Type Ratio (TTR) that is composed in entirely manual and non-experimental conditions. The corpus represents daily natural conversations which are found in the form of fictional dialogues, representing different situations and topics during the years 2017 and 2018. The data is preprocessed and normalized which then is classified into ten different categories. The analysis of the corpus is made using LancsBox, and different statistical analyses are performed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Change history
03 August 2019
In the original version of the book, the affiliations of the author group of the chapter “Developing Social-Media Based Text Corpus for San’ani Dialect (SMTCSD)” have been changed. The correction chapter has been updated with the changes.
References
Habash NY (27 Aug 2010) Introduction to Arabic natural language processing. Synth Lect Hum Lang Technol 3(1):1–87
Alshutayri A, Atwell E (May 2018) Creating an Arabic dialect text corpus by exploring Twitter, Facebook, and online newspapers. In: OSACT 3: The 3rd Workshop on Open-Source Arabic Corpora and Processing Tools, p 54
Biadsy F, Hirschberg J, Habash N, (March 2009) Spoken Arabic dialect identification using phonotactic modeling. In: Proceedings of the eacl 2009 workshop on computational approaches to semitic languages, pp 53–61. Association for Computational Linguistics
Zaghouani W (25 Feb 2017) Critical survey of the freely available Arabic corpora. arXiv preprint arXiv:1702.07835
Al-Sulaiti L, Atwell ES (Jan 2006) The design of a corpus of contemporary Arabic. Int J Corpus Linguist 11(2):135–171
Saad, MK, Ashour WM (2010) Osac: open source arabic corpora. In: 6th ArchEng Int. Symposiums, EEECS, vol. 10. http://hdl.handle.net/20.500.12358/25195
Cotterell R, Callison-Burch C (May 2014) A multi-dialect, multi-genre corpus of informal written Arabic. InLREC, pp 241–245
Sinclair J (2004) Corpus and text — basic principles. In Wynne M (ed) Developing linguistic corpora: a guide to good practice. Oxbow Books, Oxford, pp. 1–16. http://ahds.ac.uk/linguistic-corpora
Zimmermann T, Weißgerber P (25 May 2004) Preprocessing CVS data for fine-grained analysis. In: 26th International Conference on Software Engineering Conference Proceedings. Edinburgh, UK. https://doi.org/10.1049/ic:20040466
Atkins S, Clear J, Ostler N (1992) Corpus design criteria. Lit Linguist Comput 7(1):1–16
Sadjirin R, Aziz RA, Nordin NM, Ismail MR, Baharum ND (2018) The development of Malaysian Corpus of Financial English (MaCFE). GEMA Online® J Lang Stud 18(3)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Sharaf Addin, M., Al-Shehabi, S. (2020). Developing Social-Media Based Text Corpus for San’ani Dialect (SMTCSD). In: Satapathy, S.C., Raju, K.S., Shyamala, K., Krishna, D.R., Favorskaya, M.N. (eds) Advances in Decision Sciences, Image Processing, Security and Computer Vision. ICETE 2019. Learning and Analytics in Intelligent Systems, vol 3. Springer, Cham. https://doi.org/10.1007/978-3-030-24322-7_60
Download citation
DOI: https://doi.org/10.1007/978-3-030-24322-7_60
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-24321-0
Online ISBN: 978-3-030-24322-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)