Skip to main content
SpringerLink
Log in

Assessment of Autoencoder Architectures for Data Representation

  • Chapter
  • First Online:
Deep Learning: Concepts and Architectures

Part of the book series: Studies in Computational Intelligence ((SCI,volume 866))

Abstract

Efficient representation learning of data distribution is part and parcel of successful execution of any machine learning based model. Autoencoders are good at learning the representation of data with lower dimensions. Traditionally, autoencoders have been widely used for data compression in order to represent the structural data. Data compression is one of the most important tasks in applications based on Computer Vision, Information Retrieval, Natural Language Processing , etc. The aim of data compression is to convert the input data into smaller representation retaining the quality of input data. Many lossy and lossless data compression techniques like Flate/deflate compression, Lempel–Ziv–Welch compression, Huffman compression, Run-length encoding compression, JPEG compression are available. Similarly, autoencoders are unsupervised neural networks used for representing the structural data by data compression. Due to wide availability of high-end processing chips and large datasets, deep learning has gained a lot attention from academia, industries and research centers to solve multitude of problems. Considering the state-of-the-art literature, autoencoders are widely used architectures in many deep learning applications for representation and manifold learning and serve as popular option for dimensionality reduction . Therefore, this chapter aims to shed light upon applicability of variants of autoencoders to multiple application domains. In this chapter, basic architecture and variants of autoencoder viz. Convolutional autoencoder, Variational autoencoder, Sparse autoencoder, stacked autoencoder, Deep autoencoder , to name a few, have been thoroughly studied. How the layer size and depth of deep autoencoder model affect the overall performance of the system has also been discussed. We also outlined the suitability of various autoencoder architectures to different application areas. This would help the research community to choose the suitable autoencoder architecture for the problem to be solved.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 (2013)

    Article  Google Scholar 

  2. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)

    Google Scholar 

  3. Pathak, A.R., Pandey, M., Rautaray, S.: Adaptive framework for deep learning based dynamic and temporal topic modeling from big data. Recent Pat. Eng. 13, 1 (2019). https://doi.org/10.2174/1872212113666190329234812

    Article  Google Scholar 

  4. Pathak, A.R., Pandey, M., Rautaray, S.: Adaptive model for dynamic and temporal topic modeling from big data using deep learning architecture. Int. J. Intell. Syst. Appl. 11(6), 13–27 (MECS-Press)

    Google Scholar 

  5. Pathak, A.R., Pandey, M., Rautaray, S., Pawar, K.: Assessment of object detection using deep convolutional neural networks. In: Bhalla, S., Bhateja, V., Chandavale, A.A., Hiwale, A.S., Satapathy, S.C. (eds.) Intelligent Computing and Information and Communication, pp. 457–466. Springer Singapore (2018)

    Chapter  Google Scholar 

  6. Pathak, A.R., Pandey, M., Rautaray, S.: Deep learning approaches for detecting objects from images: a review. In: Pattnaik, P.K., Rautaray, S.S., Das, H., Nayak, J. (eds.) Progress in Computing, Analytics and Networking, pp. 491–499. Springer Singapore (2018)

    Google Scholar 

  7. Pathak, A.R., Pandey, M., Rautaray, S.: Application of deep learning for object detection. Procedia Comput. Sci. 132, 1706–1717 (2018)

    Article  Google Scholar 

  8. Pawar, K., Attar, V.: Deep learning approaches for video-based anomalous activity detection. World Wide Web 22, 571–601 (2019)

    Article  Google Scholar 

  9. Pawar, K., Attar, V.: Deep Learning approach for detection of anomalous activities from surveillance videos. In: CCIS. Springer (2019, in Press)

    Google Scholar 

  10. Khare, K., Darekar, O., Gupta, P., Attar, V.Z.: Short term stock price prediction using deep learning. In: 2nd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT), pp. 482–486 (2017)

    Google Scholar 

  11. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  12. Kurtz, K.J.: The divergent autoencoder (DIVA) model of category learning. Psychon. Bull. Rev. 14, 560–576 (2007)

    Article  Google Scholar 

  13. Odena, A., Dumoulin, V., Olah, C.: Deconvolution and checkerboard artifacts. Distill (2016). https://doi.org/10.23915/distill.00003

  14. Zhang, Z., et al: Depth-based subgraph convolutional auto-encoder for network representation learning. Pattern Recognit. (2019)

    Google Scholar 

  15. Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014). http://arxiv.org/abs/1406.1078

  16. Srivastava, N., Mansimov, E., Salakhudinov, R.: Unsupervised learning of video representations using LSTMs. In: International Conference on Machine Learning, pp. 843–852 (2015)

    Google Scholar 

  17. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016)

    Google Scholar 

  18. Poultney, C., Chopra, S., Cun, Y.L., et al.: Efficient learning of sparse representations with an energy-based model. In: Advances in Neural Information Processing Systems, pp. 1137–1144 (2007)

    Google Scholar 

  19. Lee, H., Ekanadham, C., Ng, A.Y.: Sparse deep belief net model for visual area V2. In: Advances in Neural Information Processing Systems, pp. 873–880 (2008)

    Google Scholar 

  20. Zou, W.Y., Ng, A.Y., Yu, K.: Unsupervised learning of visual invariance with temporal coherence. In: NIPS 2011 Workshop on Deep Learning and Unsupervised Feature Learning, vol. 3 (2011)

    Google Scholar 

  21. Jiang, X., Zhang, Y., Zhang, W., Xiao, X.: A novel sparse auto-encoder for deep unsupervised learning. In 2013 Sixth International Conference on Advanced Computational Intelligence (ICACI), pp. 256–261 (2013)

    Google Scholar 

  22. Le, Q.V., et al.: Building high-level features using large scale unsupervised learning (2011). http://arxiv.org/abs/1112.6209

  23. Chen, J., et al.: Cross-covariance regularized autoencoders for nonredundant sparse feature representation. Neurocomputing 316, 49–58 (2018)

    Article  Google Scholar 

  24. Goroshin, R., LeCun, Y.: Saturating auto-encoders (2013). http://arxiv.org/abs/1301.3577

  25. Liu, W., Ma, T., Tao, D., You, J.H.S.A.E.: A Hessian regularized sparse auto-encoders. Neurocomputing 187, 59–65 (2016)

    Article  Google Scholar 

  26. Rifai, S., Vincent, P., Muller, X., Glorot, X., Bengio, Y.: Contractive auto-encoders: explicit invariance during feature extraction. In: Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 833–840 (2011)

    Google Scholar 

  27. Rifai, S., et al.: Higher order contractive auto-encoder. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 645–660 (2011)

    Chapter  Google Scholar 

  28. Alain, G., Bengio, Y.: What regularized auto-encoders learn from the data-generating distribution. J. Mach. Learn. Res. 15, 3563–3593 (2014)

    MathSciNet  MATH  Google Scholar 

  29. Mesnil, G., et al.: Unsupervised and transfer learning challenge: a deep learning approach. In: Proceedings of the 2011 International Conference on Unsupervised and Transfer Learning Workshop, vol. 27, pp. 97–111 (2011)

    Google Scholar 

  30. Konda, K., Memisevic, R., Krueger, D.: Zero-bias autoencoders and the benefits of co-adapting features (2014). http://arxiv.org/abs/1402.3337

  31. Makhzani, A., Frey, B.: K-sparse autoencoders (2013). http://arxiv.org/abs/1312.5663

  32. Makhzani, A., Frey, B.J.: Winner-take-all autoencoders. In: Advances in Neural Information Processing Systems, pp. 2791–2799 (2015)

    Google Scholar 

  33. Ng, A.: Sparse Autoencoder. CS294A Lecture Notes, vol. 72, pp. 1–19 (2011)

    Google Scholar 

  34. Liang, K., Chang, H., Cui, Z., Shan, S., Chen, X.: Representation learning with smooth autoencoder. In: Asian Conference on Computer Vision, pp. 72–86 (2014)

    Chapter  Google Scholar 

  35. Kampffmeyer, M., Løkse, S., Bianchi, F.M., Jenssen, R., Livi, L.: The deep kernelized autoencoder. Appl. Soft Comput. 71, 816–825 (2018)

    Article  Google Scholar 

  36. Majumdar, A.: Graph structured autoencoder. Neural Netw. 106, 271–280 (2018)

    Article  Google Scholar 

  37. Sankaran, A., Vatsa, M., Singh, R., Majumdar, A.: Group sparse autoencoder. Image Vis. Comput. 60, 64–74 (2017)

    Article  Google Scholar 

  38. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103 (2008)

    Google Scholar 

  39. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)

    MathSciNet  MATH  Google Scholar 

  40. Ferles, C., Papanikolaou, Y., Naidoo, K.J.: Denoising autoencoder self-organizing map (DASOM). Neural Netw. 105, 112–131 (2018)

    Article  Google Scholar 

  41. Chen, M., Weinberger, K., Sha, F., Bengio, Y.: Marginalized denoising auto-encoders for nonlinear representations. In: International Conference on Machine Learning, pp. 1476–1484 (2014)

    Google Scholar 

  42. Maheshwari, S., Majumdar, A.: Hierarchical autoencoder for collaborative filtering. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–7 (2018)

    Google Scholar 

  43. Kingma, D.P., Welling, M.: Auto-encoding variational bayes (2013). http://arxiv.org/abs/1312.6114

  44. Burda, Y., Grosse, R., Salakhutdinov, R.: Importance weighted autoencoders (2015). http://arxiv.org/abs/1509.00519

  45. Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., Frey, B.: Adversarial autoencoders (2015). http://arxiv.org/abs/1511.05644

  46. Wang, X., Peng, D., Hu, P., Sang, Y.: Adversarial correlated autoencoder for unsupervised multi-view representation learning. Knowl. Based Syst. (2019)

    Google Scholar 

  47. Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders (2017). http://arxiv.org/abs/1711.01558

  48. Kim, Y., Zhang, K., Rush, A.M., LeCun, Y., et al.: Adversarially regularized autoencoders (2017). http://arxiv.org/abs/1706.04223

  49. Yan, X., Chang, H., Shan, S., Chen, X.: Modeling video dynamics with deep dynencoder. In: European Conference on Computer Vision, pp. 215–230 (2014)

    Chapter  Google Scholar 

  50. Zhao, J., Mathieu, M., Goroshin, R., Lecun, Y.: Stacked what-where auto-encoders (2015). http://arxiv.org/abs/1506.02351

  51. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)

    Article  Google Scholar 

  52. Zeiler, M.D., Krishnan, D., Taylor, G.W., Fergus, R.: Deconvolutional networks. In: Conference on Computer Vision and Pattern Recognition, pp. 2528–2535. IEEE (2010)

    Google Scholar 

  53. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat., 400–407 (1951)

    Article  MathSciNet  MATH  Google Scholar 

  54. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2014). http://arxiv.org/abs/1412.6980

  55. Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12, 2121–2159 (2011)

    MathSciNet  MATH  Google Scholar 

  56. Le, Q.V., et al.: On optimization methods for deep learning. In: Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 265–272 (2011)

    Google Scholar 

  57. Rumelhart, D.E., Hinton, G.E., Williams, R.J., et al.: Learning representations by back-propagating errors. Cogn. Model. 5, 1 (1988)

    Google Scholar 

  58. Hinton, G.E., McClelland, J.L.: Learning representations by recirculation. In: Neural Information Processing Systems, pp. 358–366 (1988)

    Google Scholar 

  59. Zhou, Y., Arpit, D., Nwogu, I., Govindaraju, V.: Is joint training better for deep auto-encoders? (2014). http://arxiv.org/abs/1405.1380

  60. Qi, Y., Wang, Y., Zheng, X., Wu, Z.: Robust feature learning by stacked autoencoder with maximum correntropy criterion. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6716–6720 (2014)

    Google Scholar 

  61. KukaÄŤka, J., Golkov, V., Cremers, D.: Regularization for deep learning: a taxonomy (2017). http://arxiv.org/abs/1710.10686

  62. Lamb, A., Dumoulin, V., Courville, A.: Discriminative regularization for generative models (2016). http://arxiv.org/abs/1602.03220

  63. Kamyshanska, H., Memisevic, R.: The potential energy of an autoencoder. IEEE Trans. Pattern Anal. Mach. Intell. 37, 1261–1273 (2015)

    Article  Google Scholar 

  64. Kamyshanska, H., Memisevic, R.: On autoencoder scoring. In: International Conference on Machine Learning, pp. 720–728 (2013)

    Google Scholar 

  65. Krogh, A., Hertz, J.A.: A simple weight decay can improve generalization. In: Advances in Neural Information Processing Systems, pp. 950–957 (1992)

    Google Scholar 

  66. Fan, Y.J.: Autoencoder node saliency: selecting relevant latent representations. Pattern Recognit. 88, 643–653 (2019)

    Article  Google Scholar 

  67. LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.-R.: Efficient backprop. In: Neural Networks: Tricks of the Trade, pp 9–48. Springer (2012)

    Google Scholar 

  68. Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. In: Advances in Neural Information Processing Systems, pp. 971–980 (2017)

    Google Scholar 

  69. Leonard, M.: Deep Learning Nanodegree Foundation Course. Lecture Notes in Autoencoders. Udacity (2018)

    Google Scholar 

  70. Xiong, Y., Zuo, R.: Recognition of geochemical anomalies using a deep autoencoder network. Comput. Geosci. 86, 75–82 (2016)

    Article  Google Scholar 

  71. Leng, B., Guo, S., Zhang, X., Xiong, Z.: 3D object retrieval with stacked local convolutional autoencoder. Sig. Process. 112, 119–128 (2015)

    Article  Google Scholar 

  72. Ribeiro, M., Lazzaretti, A.E., Lopes, H.S.: A study of deep convolutional auto-encoders for anomaly detection in videos. Pattern Recognit. Lett. 105, 13–22 (2018)

    Article  Google Scholar 

  73. Li, L., Li, X., Yang, Y., Dong, J.: Indoor tracking trajectory data similarity analysis with a deep convolutional autoencoder. Sustain. Cities Soc. 45, 588–595 (2019)

    Article  Google Scholar 

  74. Wan, X., Zhao, C., Wang, Y., Liu, W.: Stacked sparse autoencoder in hyperspectral data classification using spectral-spatial, higher order statistics and multifractal spectrum features. Infrared Phys. Technol. 86, 77–89 (2017)

    Article  Google Scholar 

  75. McCoy, J.T., Kroon, S., Auret, L.: Variational autoencoders for missing data imputation with application to a simulated milling circuit. IFAC PapersOnLine 51, 141–146 (2018)

    Article  Google Scholar 

  76. Wu, C., et al.: Semi-supervised dimensional sentiment analysis with variational autoencoder. Knowl. Based Syst. 165, 30–39 (2019)

    Article  Google Scholar 

  77. Zhang, J., Shan, S., Kan, M., Chen, X.: Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. In: European Conference on Computer Vision, pp. 1–16 (2014)

    Google Scholar 

  78. Masci, J., Meier, U., Cirecsan, D., Schmidhuber, J.: Stacked convolutional auto-encoders for hierarchical feature extraction. In: International Conference on Artificial Neural Networks, pp. 52–59 (2011)

    Google Scholar 

  79. Liou, C.-Y., Cheng, W.-C., Liou, J.-W., Liou, D.-R.: Autoencoder for words. Neurocomputing 139, 84–96 (2014)

    Article  Google Scholar 

  80. Carreira-Perpinan, M.A., Raziperchikolaei, R.: Hashing with binary autoencoders. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  81. Pan, S., et al.: Adversarially regularized graph autoencoder for graph embedding (2018). http://arxiv.org/abs/1802.04407

  82. Li, M., et al.: GRAINS: generative recursive autoencoders for INdoor scenes. ACM Trans. Graph. 38, 12:1–12:16 (2019)

    Article  Google Scholar 

  83. Alaverdyan, Z., Chai, J., Lartizien, C.: Unsupervised feature learning for outlier detection with stacked convolutional autoencoders, siamese networks and wasserstein autoencoders: application to epilepsy detection. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 210–217. Springer (2018)

    Google Scholar 

  84. Hou, L., et al.: Sparse autoencoder for unsupervised nucleus detection and representation in histopathology images. Pattern Recognit. 86, 188–200 (2019)

    Article  Google Scholar 

  85. Ullah, A., Muhammad, K., Haq, I.U., Baik, S.W.: Action recognition using optimized deep autoencoder and CNN for surveillance data streams of non-stationary environments. Futur. Gener. Comput. Syst. (2019)

    Google Scholar 

  86. Zhao, C., Zhang, L.: Spectral-spatial stacked autoencoders based on low-rank and sparse matrix decomposition for hyperspectral anomaly detection. Infrared Phys. Technol. 92, 166–176 (2018)

    Article  Google Scholar 

  87. Singh, M., Nagpal, S., Vatsa, M., Singh, R.: Are you eligible? Predicting adulthood from face images via class specific mean autoencoder. Pattern Recognit. Lett. 119, 121–130 (2019)

    Article  Google Scholar 

  88. Tasnim, S., Rahman, A., Oo, A.M.T., Haque, M.E.: Autoencoder for wind power prediction. Renewables Wind. Water Sol. 4, 6 (2017)

    Google Scholar 

  89. Lv, S.-X., Peng, L., Wang, L.: Stacked autoencoder with echo-state regression for tourism demand forecasting using search query data. Appl. Soft Comput. 73, 119–133 (2018)

    Article  Google Scholar 

  90. Xie, R., Wen, J., Quitadamo, A., Cheng, J., Shi, X.: A deep auto-encoder model for gene expression prediction. BMC Genom. 18, 845 (2017)

    Article  Google Scholar 

  91. Zhang, J., Li, K., Liang, Y., Li, N.: Learning 3D faces from 2D images via stacked contractive autoencoder. Neurocomputing 257, 67–78 (2017)

    Article  Google Scholar 

  92. Gareis, I.E., Vignolo, L.D., Spies, R.D., Rufiner, H.L.: Coherent averaging estimation autoencoders applied to evoked potentials processing. Neurocomputing 240, 47–58 (2017)

    Article  Google Scholar 

  93. Mehta, J., Majumdar, A.: RODEO: robust DE-aliasing autoencoder for real-time medical image reconstruction. Pattern Recognit. 63, 499–510 (2017)

    Article  Google Scholar 

  94. Liu, Y., Feng, X., Zhou, Z.: Multimodal video classification with stacked contractive autoencoders. Sig. Process. 120, 761–766 (2016)

    Article  Google Scholar 

  95. Zhang, Z., et al.: Deep neural network-based bottleneck feature and denoising autoencoder-based dereverberation for distant-talking speaker identification. EURASIP J. Audio Speech Music Process. 2015, 12 (2015)

    Google Scholar 

  96. Makkie, M., Huang, H., Zhao, Y., Vasilakos, A.V., Liu, T.: Fast and scalable distributed deep convolutional autoencoder for fMRI big data analytics. Neurocomputing 325, 20–30 (2019)

    Article  Google Scholar 

  97. Guo, Q., et al.: Learning robust uniform features for cross-media social data by using cross autoencoders. Knowl. Based Syst. 102, 64–75 (2016)

    Article  Google Scholar 

  98. Su, J., et al.: A neural generative autoencoder for bilingual word embeddings. Inf. Sci. (Ny) 424, 287–300 (2018)

    Article  MathSciNet  Google Scholar 

  99. Gianniotis, N., Kügler, S.D., Tino, P., Polsterer, K.L.: Model-coupled autoencoder for time series visualization. Neurocomputing 192, 139–146 (2016)

    Article  Google Scholar 

  100. Hwang, U., Park, J., Jang, H., Yoon, S., Cho, N.I.: PuVAE: a variational autoencoder to purify adversarial examples (2019). http://arxiv.org/abs/1903.00585

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering & IT, College of Engineering Pune (COEP), Pune, India

    Karishma Pawar & Vahida Z. Attar

Authors
  1. Karishma Pawar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vahida Z. Attar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Karishma Pawar or Vahida Z. Attar .

Editor information

Editors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB, Canada

    Witold Pedrycz

  2. Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan

    Shyi-Ming Chen

Appendix

Appendix

List of abbreviations used in this chapter are mentioned in Table 2.

Table 2 List of abbreviations
Full size table

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Pawar, K., Attar, V.Z. (2020). Assessment of Autoencoder Architectures for Data Representation. In: Pedrycz, W., Chen, SM. (eds) Deep Learning: Concepts and Architectures. Studies in Computational Intelligence, vol 866. Springer, Cham. https://doi.org/10.1007/978-3-030-31756-0_4

Download citation

Publish with us

Policies and ethics

Search

Navigation