Skip to main content
SpringerLink
Log in

A Trie Based Model for SMS Text Normalization

  • Conference paper
  • First Online:
Intelligent Computing (CompCom 2019)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 997))

Included in the following conference series:

Abstract

Language takes varied forms as used on different platforms. The instinctive nature of humans to use shorter message length, facilitating faster typing while maintaining semantic clarity, shapes the structure of a non-standard form of written text known as the texting language. The present work focuses on developing a Trie-based technique to model words in texting language for normalizing SMS text to Standard English text. The model is conceived and developed through systematic analysis of training data on user behaviour of texting language. Trie as a data structure is not only compact, but also is easy to manipulate for performing operations related to text normalization. Although Trie is a well-known data structure, its application to model SMS text has so far been unexplored. The results obtained using this model, despite the scheme being computationally cheap, are comparable with existing HMM based schemes that are available in the literature.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    91% of mobile Internet access is for social activities vs. 79% on desktops, according to Microsoft (http://www.social4retail.com/the-growth-of-mobile-marketing-and-tagging-info-graphic.html).

References

  1. Jurafsky, D., James, H.M.: Speech & Language Processing, 2nd edn. Pearson Education, India (2000)

    Google Scholar 

  2. Kobus, C., Yvon, F., Damnati, G.: Normalizing SMS: are two metaphors better than one? In: Proceedings of the 22nd International Conference on Computational Linguistics, vol. 1, pp. 441–448. Association for Computational Linguistics (2008)

    Google Scholar 

  3. Choudhury, M., Saraf, R., Jain, V., Mukherjee, A., Sarkar, S., Basu, A.: Investigation and modeling of the structure of texting language. Int. J. Doc. Anal. Recogn. (IJDAR) 10(3–4), 157–174 (2007)

    Google Scholar 

  4. Bali, R.: A theoretical review on SMS normalization using hidden Markov models (HMMs). Int. J. Comput. Trends Technol. (IJCTT) 4(7), 2387–2388 (2013)

    Google Scholar 

  5. Grinter, R.E., Eldridge, M.A.: y do tngrs luv 2 txt msg? In: Proceedings of ECSCW, pp. 219–238. Springer, Netherlands (2001)

    Google Scholar 

  6. Pennell, D., Liu, Y.: Toward text message normalization: modeling abbreviation generation. In: Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 5364–5367 (2011)

    Google Scholar 

  7. Jose, G., Raj, N.S.: Lexical normalization model for noisy SMS text. In: First International Conference on Computational Systems and Communications (ICCSC), pp. 57–62 (2014)

    Google Scholar 

  8. Jose, G., Raj, N.S.: Noisy SMS text normalization model. In: International Conference for Convergence of Technology, pp. 1–6, (2014)

    Google Scholar 

  9. Beaufort, R., et al.: A hybrid rule/model-based finite-state framework for normalizing SMS messages. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, pp. 770–779 (2010)

    Google Scholar 

  10. Nocon, N., et al.: Norm API: an API for normalizing Filipino shortcut texts. In: Asian Language Processing (IALP), pp. 207–210. IEEE (2014)

    Google Scholar 

  11. https://gist.github.com/h3xx/1976236

  12. http://www.math.sjsu.edu/~foster/dictionary.txt

  13. Al-Saudi, J.: Error analysis and spelling mistakes of EFL learners at Tafila Technical University: a case study. Eur. Sci. J. 8(21), 56–75 (2013)

    Google Scholar 

Download references

Acknowledgement

The author thankfully acknowledge the contribution of students Tushar Singla in conducting the experiments and preparing the draft.

Author information

Authors and Affiliations

  1. Indian Institute of Technology Delhi, New Delhi, 110016, India

    Niladri Chatterjee

Authors
  1. Niladri Chatterjee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Niladri Chatterjee .

Editor information

Editors and Affiliations

  1. Faculty of Science and Engineering, Saga University, Saga, Japan

    Kohei Arai

  2. The Science and Information SAI Organization, Bradford, West Yorkshire, UK

    Rahul Bhatia

  3. The Science and Information SAI Organization, Bradford, West Yorkshire, UK

    Supriya Kapoor

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chatterjee, N. (2019). A Trie Based Model for SMS Text Normalization. In: Arai, K., Bhatia, R., Kapoor, S. (eds) Intelligent Computing. CompCom 2019. Advances in Intelligent Systems and Computing, vol 997. Springer, Cham. https://doi.org/10.1007/978-3-030-22871-2_60

Download citation

Publish with us

Policies and ethics

Search

Navigation