Developing Social-Media Based Text Corpus for San’ani Dialect (SMTCSD)

  • Mohammed Sharaf AddinEmail author
  • Sabah Al-Shehabi
Conference paper
Part of the Learning and Analytics in Intelligent Systems book series (LAIS, volume 3)


This paper aims at developing and designing a social media based text corpus of San’ani Dialect (SMTCSD). The corpus is considered the first in the research area that codifies one of the most popular and spoken dialects in Yemen representing nearly 30% of Yemeni speakers. Our primary objective is a compilation of authentic and unmodified texts gathered from different open-source social media platforms mainly Facebook and Telegram Apps. As a result, we obtained a corpus of 447,401 tokens and 51,073 types with an 11.42% Token:Type Ratio (TTR) that is composed in entirely manual and non-experimental conditions. The corpus represents daily natural conversations which are found in the form of fictional dialogues, representing different situations and topics during the years 2017 and 2018. The data is preprocessed and normalized which then is classified into ten different categories. The analysis of the corpus is made using LancsBox, and different statistical analyses are performed.


Corpus design San’ani dialect Social media Token Type Category LancsBox Statistical analysis 


  1. 1.
    Habash NY (27 Aug 2010) Introduction to Arabic natural language processing. Synth Lect Hum Lang Technol 3(1):1–87Google Scholar
  2. 2.
    Alshutayri A, Atwell E (May 2018) Creating an Arabic dialect text corpus by exploring Twitter, Facebook, and online newspapers. In: OSACT 3: The 3rd Workshop on Open-Source Arabic Corpora and Processing Tools, p 54Google Scholar
  3. 3.
    Biadsy F, Hirschberg J, Habash N, (March 2009) Spoken Arabic dialect identification using phonotactic modeling. In: Proceedings of the eacl 2009 workshop on computational approaches to semitic languages, pp 53–61. Association for Computational LinguisticsGoogle Scholar
  4. 4.
    Zaghouani W (25 Feb 2017) Critical survey of the freely available Arabic corpora. arXiv preprint arXiv:1702.07835Google Scholar
  5. 5.
    Al-Sulaiti L, Atwell ES (Jan 2006) The design of a corpus of contemporary Arabic. Int J Corpus Linguist 11(2):135–171Google Scholar
  6. 6.
    Saad, MK, Ashour WM (2010) Osac: open source arabic corpora. In: 6th ArchEng Int. Symposiums, EEECS, vol. 10.
  7. 7.
    Cotterell R, Callison-Burch C (May 2014) A multi-dialect, multi-genre corpus of informal written Arabic. InLREC, pp 241–245Google Scholar
  8. 8.
    Sinclair J (2004) Corpus and text — basic principles. In Wynne M (ed) Developing linguistic corpora: a guide to good practice. Oxbow Books, Oxford, pp. 1–16.
  9. 9.
    Zimmermann T, Weißgerber P (25 May 2004) Preprocessing CVS data for fine-grained analysis. In: 26th International Conference on Software Engineering Conference Proceedings. Edinburgh, UK.
  10. 10.
    Atkins S, Clear J, Ostler N (1992) Corpus design criteria. Lit Linguist Comput 7(1):1–16CrossRefGoogle Scholar
  11. 11.
    Sadjirin R, Aziz RA, Nordin NM, Ismail MR, Baharum ND (2018) The development of Malaysian Corpus of Financial English (MaCFE). GEMA Online® J Lang Stud 18(3)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.CALTSUniversity of HyderabadHyderabadIndia
  2. 2.Department of English, Faculty of Education, MahweetSana’a UniversitySana’aYemen

Personalised recommendations