Advertisement

Using Unsupervised Deep Learning for Automatic Summarization of Arabic Documents

  • Nabil Alami
  • Noureddine En-nahnahi
  • Said Alaoui Ouatik
  • Mohammed Meknassi
Research Article - Computer Engineering and Computer Science
  • 52 Downloads

Abstract

Traditional Arabic text summarization (ATS) systems are based on bag-of-words representation, which involve a sparse and high-dimensional input data. Thus, dimensionality reduction is greatly needed to increase the power of features discrimination. In this paper, we present a new method for ATS using variational auto-encoder (VAE) model to learn a feature space from a high-dimensional input data. We explore several input representations such as term frequency (tf), tf-idf and both local and global vocabularies. All sentences are ranked according to the latent representation produced by the VAE. We investigate the impact of using VAE with two summarization approaches, graph-based and query-based approaches. Experiments on two benchmark datasets specifically designed for ATS show that the VAE using tf-idf representation of global vocabularies clearly provides a more discriminative feature space and improves the recall of other models. Experiment results confirm that the proposed method leads to better performance than most of the state-of-the-art extractive summarization approaches for both graph-based and query-based summarization approaches.

Keywords

Arabic text summarization Deep learning Unsupervised feature learning Variational auto-encoder, Graph-based summarization Query-based summarization 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Luhn, H.P.: The automatic creation of literature abstracts. IBM J. Res. Dev. 2(2), 159–165 (1958)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Ferreira, R.; de Souza Cabral, L.; Freitas, F.; Lins, R.D.; de Frana Silva, G.; Simske, S.J.; Favaro, L.: A multi-document summarization system based on statistics and linguistic treatment. Expert Syst. Appl. 41(13), 5780–5787 (2014)CrossRefGoogle Scholar
  3. 3.
    Ferreira, R.; De Souza, L.; Dueire, R.; et al.: Assessing sentence scoring techniques for extractive text summarization. Expert Syst. Appl. 40(14), 5755–5764 (2013).  https://doi.org/10.1016/j.eswa.2013.04.023 CrossRefGoogle Scholar
  4. 4.
    Erkan, G.; Radev, D.: LexRank: graph-based lexical centrality as salience in text summarization. J. Artif. Intell. Res. 22, 457–479 (2004)Google Scholar
  5. 5.
    Baralis, E.; Cagliero, L.; Mahoto, N.; Fiori, A.: GRAPHSUM : discovering correlations among multiple terms for graph-based summarization. Inf. Sci. 249, 96–109 (2013).  https://doi.org/10.1016/j.ins.2013.06.046 MathSciNetCrossRefGoogle Scholar
  6. 6.
    Mihalcea, R.; Tarau, P.: TextRank: Bringing order into texts. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Spain, pp. 404–411 (2004)Google Scholar
  7. 7.
    Fattah, M.A.: A hybrid machine learning model for multi-document summarization. Appl. Intell. 40(4), 592–600 (2014).  https://doi.org/10.1007/s10489-013-0490-0 CrossRefGoogle Scholar
  8. 8.
    Alguliyev, R.M.; Aliguliyev, R.M.; Isazade, N.R.: An unsupervised approach to generating generic summaries of documents. Appl. Soft Comput. 34, 236–250 (2015).  https://doi.org/10.1016/j.asoc.2015.04.050 CrossRefGoogle Scholar
  9. 9.
    Yang, L.; Cai, X.; Zhang, Y.; Shi, P.: Enhancing sentence-level clustering with ranking-based clustering framework for theme-based summarization. Inf. Sci. 260, 37–50 (2014).  https://doi.org/10.1016/j.ins.2013.11.026 CrossRefGoogle Scholar
  10. 10.
    Yousefi-Azar, M.; Hamey, L.: Text summarization using unsupervised deep learning. Expert Syst. Appl. 68, 93–105 (2017).  https://doi.org/10.1016/j.eswa.2016.10.017 CrossRefGoogle Scholar
  11. 11.
    Akbarizadeh, G.: Segmentation of SAR satellite images using cellular learning automata and adaptive chains. J. Remote Sens. Technol. pp. 44–51 (2013).  https://doi.org/10.18005/jrst0102003
  12. 12.
    Akbarizadeh, G.; Moghaddam, A.E.: Detection of lung nodules in CT scans based on unsupervised feature learning and fuzzy inference. J. Med. Imaging Health Inform. 6(2), 477–483 (2016).  https://doi.org/10.1166/jmihi.2016.1720 CrossRefGoogle Scholar
  13. 13.
    Rahmani, M.; Akbarizadeh, G.: Unsupervised feature learning based on sparse coding and spectral clustering for segmentation of synthetic aperture radar images. IET Comput. Vision 9(5), 629–638 (2015).  https://doi.org/10.1049/iet-cvi.2014.0295 CrossRefGoogle Scholar
  14. 14.
    Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–127 (2009)CrossRefzbMATHGoogle Scholar
  15. 15.
    Krizhevsky, A.; Sutskever, I.; Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS’12), Lake Tahoe, Nevada, USA, pp. 1090–1098 (2012)Google Scholar
  16. 16.
    Sermanet, P.; Eigen, D.; Zhang, X.; Mathieu, M.; Fergus, R.; LeCun, Y.: OverFeat: integrated recognition, localization and detection using convolutional networks. In: Proceedings of the 2nd International Conference On Learning Representation (ICLR2014), Banff, Canada (2014)Google Scholar
  17. 17.
    Donahue, J.; Anne Hendricks, L.; Rohrbach, M.; Venugopalan, S.; Guadarrama, S.; Saenko, K.; Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 677–691 (2017)CrossRefGoogle Scholar
  18. 18.
    Er, M.J.; Zhang, Y.; Wang, N.; Pratama, M.: Attention pooling-based convolutional neural network for sentence modelling. Inf. Sci. 373, 388–403 (2016).  https://doi.org/10.1016/j.ins.2016.08.084 CrossRefGoogle Scholar
  19. 19.
    Li, F.; Zhang, M.; Tian, B.; Chen, B.; Fu, G.; Ji, D.: Recognizing irregular entities in biomedical text via deep neural networks. Pattern Recognit. Lett. (2017).  https://doi.org/10.1016/j.patrec.2017.06.009
  20. 20.
    Ayinde, B.O.; Zurada, J.M.: Deep learning of constrained autoencoders for enhanced understanding of data. IEEE Trans. Neural Netw. Learn. Syst. 99, 1–11 (2017).  https://doi.org/10.1109/tnnls.2017.2747861 CrossRefGoogle Scholar
  21. 21.
    Firat, O.; Cho, K.; Sankaran, B.; Yarman Vural, F.T.; Bengio, Y.: Multi-way, multilingual neural machine translation. Comput. Speech Lang. 45, 236–252 (2017).  https://doi.org/10.1016/j.csl.2016.10.006 CrossRefGoogle Scholar
  22. 22.
    Zhong, Sh; Liu, Y.; Li, B.; Long, J.: Query-oriented unsupervised multi-document summarization via deep learning model. Expert Syst. Appl. 42(21), 8146–8155 (2015)CrossRefGoogle Scholar
  23. 23.
    Kingma, D.P.; Welling, M.: Auto-encoding variational bayes. In: Proceedings of the International Conference on Learning Representations, Banff, Canada (2014)Google Scholar
  24. 24.
    Li, H.; Misra, S.: Prediction of subsurface NMR T2 distributions in a shale petroleum system using variational autoencoder-based neural networks. IEEE Geosci. Remote Sens. Lett. 14(12), 2395–2397 (2017).  https://doi.org/10.1109/lgrs.2017.2766130 CrossRefGoogle Scholar
  25. 25.
    Akbarizadeh, G.; Tirandaz, Z.; Kooshesh, M.: A new curvelet based texture classification approach for land cover recognition of SAR satellite images. Malays. J. Comput. Sci. 27(3), 218–239 (2014)Google Scholar
  26. 26.
    Ahmadi, N.; Akbarizadeh, G.: Hybrid robust iris recognition approach using iris image pre-processing, two-dimensional gabor features and multi-layer perceptron neural network/PSO. IET Biom. (2017).  https://doi.org/10.1049/iet-bmt.2017.0041
  27. 27.
    Wang, L.; Zhang, J.; Liu, P.; Choo, K.-K.R.; Huang, F.: Spectral-spatial multi-feature-based deep learning for hyperspectral remote sensing image classification. Soft. Comput. 21(1), 213–221 (2016).  https://doi.org/10.1007/s00500-016-2246-3 CrossRefzbMATHGoogle Scholar
  28. 28.
    Vincent, P.; Larochelle, H.; Bengio, Y.; Manzagol, P.-A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning—ICML ’08.  https://doi.org/10.1145/1390156.1390294 (2008)
  29. 29.
    Noda, K.; Yamaguchi, Y.; Nakadai, K.; Okuno, H.G.; Ogata, T.: Audio-visual speech recognition using deep learning. Appl. Intell. 42(4), 722–737 (2014).  https://doi.org/10.1007/s10489-014-0629-7 CrossRefGoogle Scholar
  30. 30.
    Lecun, Y.; Bottou, L.; Bengio, Y.; Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998).  https://doi.org/10.1109/5.726791 CrossRefGoogle Scholar
  31. 31.
    Kim, E.; Corte-Real, M.; Baloch, Z.: A deep semantic mobile application for thyroid cytopathology. In: Medical Imaging 2016: PACS and Imaging Informatics: Next Generation and Innovations (2016).  https://doi.org/10.1117/12.2216468
  32. 32.
    Esteva, A.; Kuprel, B.; Novoa, R.A.; Ko, J.; Swetter, S.M.; Blau, H.M.; Thrun, S.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115–118 (2017).  https://doi.org/10.1038/nature21056 CrossRefGoogle Scholar
  33. 33.
    Gulshan, V.; Peng, L.; Coram, M.; et al.: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316(22), 2402 (2016).  https://doi.org/10.1001/jama.2016.17216 CrossRefGoogle Scholar
  34. 34.
    Edmundson, H.P.: New methods in automatic extracting. J. ACM 16(2), 264–285 (1969)CrossRefzbMATHGoogle Scholar
  35. 35.
    Heu, J.U.; Qasim, I.; Lee, D.H.: FoDoSu: multi-document summarization exploiting semantic analysis based on social Folksonomy. Inf. Process. Manag. 51(1), 212–225 (2015).  https://doi.org/10.1016/j.ipm.2014.06.003 CrossRefGoogle Scholar
  36. 36.
    Fang, H.; Lu, W.; Wu, F.; Zhang, Y.; Shang, X.; Shao, J.; Zhuang, Y.: Topic aspect-oriented summarization via group selection. Neurocomputing 149, 1613–1619 (2015).  https://doi.org/10.1016/j.neucom.2014.08.031 CrossRefGoogle Scholar
  37. 37.
    Denil, M.; Demiraj, A.; de Freitas, N.: Extraction of salient sentences from labelled documents. arXiv preprint arXiv:1412.6815 (2014)
  38. 38.
    Ha, J.W.; Kang, D.; Pyo, H.; Kim, J.: News2Images: automatically summarizing news articles into image-based contents via deep learning. In: 3rd International Workshop on News Recommendation and Analytics (INRA 2015) (with RECSYS 2015), Vienna, Austria (2015)Google Scholar
  39. 39.
    Cao, Z.; Wei, F.; Dong, L.; Li, S.; Zhou, M.: Ranking with recursive neural networks and its application to multi-document summarization. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, Texas, pp. 2153–2159 (2015)Google Scholar
  40. 40.
    Rezende, D.J.; Mohamed, S.; Wierstra, D.: Stochastic backpropagation and approximate inference in deep generative models. In: Proceedings of the 31st International Conference on International Conference on Machine Learning (ICML’14), vol. 32, Beijing, China, pp. 1278–1286 (2014)Google Scholar
  41. 41.
    Hinton, G.E.; Osindero, S.; The, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Hinton, G.E.; Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Kingma, D.P.; Mohamed, S.; Rezende, D.J.; Welling, M.: Semi-supervised learning with deep generative models. In: Proceedings of Neural Information Processing Systems (NIPS’14), pp. 3581–3589 (2014)Google Scholar
  44. 44.
    El-Haj, M.; Kruschwitz, U.; Fox, C.: Using mechanical Turk to create a corpus of Arabic summaries. In: Proceedings of the International Conference on Language Resources and Evaluation (LREC), Valletta, Malta, pp. 36–39, in the Language Resources (LRs) and Human Language Technologies (HLT) for Semitic Languages workshop held in conjunction with the 7th international language resources and evaluation conference (2010)Google Scholar
  45. 45.
    Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Proceedings of workshop on text summarization branches out, post-conference workshop of ACL, pp. 74–81 (2004)Google Scholar
  46. 46.
    Mashechkin, I.V.; Petrovskiy, M.I.; Popov, D.S.; Tsarev, D.V.: Automatic text summarization using latent semantic analysis. Program. Comput. Softw. 37(6), 299–305 (2011).  https://doi.org/10.1134/s0361768811060041 MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Brin, S.; Page, L.: The anatomy of a large-scale hypertextual web search engine. Comput. Netw. ISDN Syst. 30(1–7), 107–117 (1998).  https://doi.org/10.1016/s0169-7552(98)00110-x CrossRefGoogle Scholar

Copyright information

© King Fahd University of Petroleum & Minerals 2018

Authors and Affiliations

  • Nabil Alami
    • 1
  • Noureddine En-nahnahi
    • 1
  • Said Alaoui Ouatik
    • 1
  • Mohammed Meknassi
    • 1
  1. 1.Faculty of Science Dhar EL Mahraz, Laboratory of Informatics and Modeling (LIM)Sidi Mohamed Ben Abdellah UniversityFezMorocco

Personalised recommendations