Automatic Text Preprocessing for Intelligent Dialog Agents

  • Alessandro MaistoEmail author
  • Serena Pelosi
  • Massimiliano Polito
  • Michele Stingo
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 927)


The paper describes a new Text Preprocessing Pipeline based on a Hybrid approach which involve rule-based and stochastic approaches. The presented pipeline is part of a larger project titled Big Data for Multi-Agent Specialized System developed by Network Contacts in collaboration with University of Salerno and other institutional partners. The aim of the project is to build an Hybrid Question Answering System composed by sets of Dialog Bots able to process great volumes of data. Due to the importance of unstructured textual data, a particular focus of the project is on automatic processing of Text. The paper will describe the three main modules of the preprocessing pipeline, which involve a Style Correction Module, a Clitic Decomposition Module and a POS Tagging and Lemmatization Module.


  1. 1.
    Amato, F., Colace, F., Greco, L., Moscato, V., Picariello, A.: Semantic processing of multimedia data for e-government applications. J. Vis. Lang. Comput. 32, 35–41 (2016).
  2. 2.
    Amato, F., Mazzeo, A., Penta, A., Picariello, A.: Knowledge representation and management for e-government documents. In: IFIP International Federation for Information Processing, vol. 280, pp. 31–40 (2008).
  3. 3.
    Amato, F., Moscato, F.: A model driven approach to data privacy verification in e-health systems. Trans. Data Privacy 8(3), 273–296 (2015). Scholar
  4. 4.
    Attardi, G., Fuschetto, A., Tamberi, F., Simi, M., Vecchi, E.M.: Experiments in tagger combination: arbitrating, guessing, correcting, suggesting. In: Proceedings of Workshop Evalita, p. 10 (2009)Google Scholar
  5. 5.
    Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
  6. 6.
    Choi, J.D.: Dynamic feature induction: the last gist to the state-of-the-art. In: Proceedings of NAACL-HLT, pp. 271–281 (2016)Google Scholar
  7. 7.
    Chrupała, G., Dinu, G., Van Genabith, J.: Learning morphology with Morfette (2008)Google Scholar
  8. 8.
    Collins, M.: Discriminative training methods for hidden Markov models: theory and experiments with perceptron algorithms. In: Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing, vol. 10, pp. 1–8. Association for Computational Linguistics (2002)Google Scholar
  9. 9.
    Constant, M., Tellier, I., Duchier, D., Dupont, Y., Sigogne, A., Billot, S.: Intégrer des connaissances linguistiques dans un crf: application à l’apprentissage d’un segmenteur-étiqueteur du français. In: TALN, vol. 1, p. 321 (2011)Google Scholar
  10. 10.
    Dahlmeier, D., Ng, H.T.: Better evaluation for grammatical error correction. In: Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 568–572. Association for Computational Linguistics (2012)Google Scholar
  11. 11.
    Dell’Orletta, F.: Ensemble system for part-of-speech tagging. In: Proceedings of EVALITA, vol. 9, pp. 1–8 (2009)Google Scholar
  12. 12.
    Delmonte, R.: Generating and parsing clitics with getarun. In: CLIN (1999)Google Scholar
  13. 13.
    Favretti, R.R., Tamburini, F., De Santis, C.: CORIS/CODIS: a corpus of written Italian based on a defined and a dynamic model. A Rainbow of Corpora: Corpus Linguistics and the Languages of the World. Munich: Lincom-Europa (2002)Google Scholar
  14. 14.
    Felice, M., Yuan, Z., Andersen, Ø.E., Yannakoudakis, H., Kochmar, E.: Grammatical error correction using hybrid systems and type filtering. In: Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pp. 15–24 (2014)Google Scholar
  15. 15.
    Giménez, J., Marquez, L.: Svmtool: a general POS tagger generator based on support vector machines. In: Proceedings of the 4th International Conference on Language Resources and Evaluation. Citeseer (2004)Google Scholar
  16. 16.
    Han, N.R., Chodorow, M., Leacock, C.: Detecting errors in English article usage with a maximum entropy classifier trained on a large, diverse corpus. In: LREC (2004)Google Scholar
  17. 17.
    Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015)
  18. 18.
    Ingason, A.K., Helgadóttir, S., Loftsson, H., Rögnvaldsson, E.: A mixed method lemmatization algorithm using a hierarchy of linguistic identities (HOLI). In: Advances in Natural Language Processing, pp. 205–216. Springer, Heidelberg (2008)Google Scholar
  19. 19.
    Junczys-Dowmunt, M., Grundkiewicz, R.: The AMU system in the CoNLL-2014 shared task: grammatical error correction by data-intensive and feature-rich statistical machine translation. In: Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pp. 25–33 (2014)Google Scholar
  20. 20.
    Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Soviet Phys. doklady 10, 707–710 (1966)MathSciNetGoogle Scholar
  21. 21.
    Lyding, V., Stemle, E., Borghetti, C., Brunello, M., Castagnoli, S., Dell’Orletta, F., Dittmann, H., Lenci, A., Pirrelli, V.: The PAISA corpus of Italian web texts. In: Proceedings of the 9th Web as Corpus Workshop (WaC-9), pp. 36–43 (2014)Google Scholar
  22. 22.
    Manning, C.D.: Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In: International Conference on Intelligent Text Processing and Computational Linguistics, pp. 171–189. Springer, Heidelberg (2011)Google Scholar
  23. 23.
    Mizumoto, T., Hayashibe, Y., Komachi, M., Nagata, M., Matsumoto, Y.: The effect of learner corpus size in grammatical error correction of ESL writings. In: Proceedings of COLING 2012: Posters, pp. 863–872 (2012)Google Scholar
  24. 24.
    Monachesi, P.: A grammar of Italian clitics (1996)Google Scholar
  25. 25.
    Monachesi, P.: A lexical approach to Italian cliticization (1999)Google Scholar
  26. 26.
    Palmero Aprosio, A., Moretti, G.: Italy goes to Stanford: a collection of CoreNLP modules for Italian. ArXiv e-prints (2016)Google Scholar
  27. 27.
    Pianta, E., Zanoli, R.: TagPro: a system for Italian PoS tagging based on SVM. Intelligenza Artificiale 4(2), 8–9 (2007)Google Scholar
  28. 28.
    Porter, M.F.: An algorithm for suffix stripping. Program 14(3), 130–137 (1980)CrossRefGoogle Scholar
  29. 29.
    Rozovskaya, A., Roth, D.: Algorithm selection and model adaptation for ESL correction tasks. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 924–933. Association for Computational Linguistics (2011)Google Scholar
  30. 30.
    Russi, C.: Italian clitics: an empirical study, vol. 193. Walter de Gruyter (2008)Google Scholar
  31. 31.
    Schmid, H.: Treetagger—a language independent part-of-speech tagger. Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart 43, 28 (1995)Google Scholar
  32. 32.
    Shen, L., Satta, G., Joshi, A.: Guided learning for bidirectional sequence classification. In: ACL, vol. 7, pp. 760–767. Citeseer (2007)Google Scholar
  33. 33.
    Smedt, T.D., Daelemans, W.: Pattern for python. J. Mach. Learn. Res. 13(Jun), 2063–2067 (2012)Google Scholar
  34. 34.
    Srinivasan, A., Compton, P., Malor, R., Edwards, G., Lazarus, L.: Knowledge acquisition in context for a complex domain. Pre-print of Proceedings of the Fifth EKAW91 (1991)Google Scholar
  35. 35.
    Sun, C., Jin, X., Lin, L., Zhao, Y., Wang, X.: Convolutional neural networks for correcting English article errors. In: Natural Language Processing and Chinese Computing, pp. 102–110. Springer, Heidelberg (2015)Google Scholar
  36. 36.
    Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)Google Scholar
  37. 37.
    Toutanova, K., Klein, D., Manning, C.D., Singer, Y.: Feature-rich part-of-speech tagging with a cyclic dependency network. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, vol. 1, pp. 173–180. Association for Computational Linguistics (2003)Google Scholar
  38. 38.
    Vietri, S.: Dizionari elettronici e grammatiche a stati finiti: metodi di analisi formale della lingua italiana. Plectica (2008)Google Scholar
  39. 39.
    Vietri, S.: The Italian module for NooJ. In: Proceedings of the First Italian Conference on Computational Linguistics, CLiC-it, pp. 389–393 (2014)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Alessandro Maisto
    • 1
    Email author
  • Serena Pelosi
    • 1
  • Massimiliano Polito
    • 2
  • Michele Stingo
    • 1
  1. 1.Università di SalernoFiscianoItaly
  2. 2.Network ContactsMolfettaItaly

Personalised recommendations