Abstract
This chapter revisits the principal component analysis and the notion of distributed representations. The main focus here is on filling the part that was left out in Chap. 3, completing the exposition of the principal component analysis, and demonstrating what a distributed representation is in mathematical terms. The chapter then introduces the main unsupervised learning technique for deep learning, the autoencoder. The structural aspects are presented in detail with both explanations and illustrations, and several different types of autoencoders are presented as variations of a single theme. The idea of stacking autoencoders to produce even more condensed distributed representations is presented in detail, and the Python code for stacking and saving representations with autoencoders is provided with abundant explanations and illustrations. The chapter concludes with the recreation of a classical result in deep learning, an autoencoder that can learn to draw cats from watching unlabeled videos.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The expected value is actually the weighted sum, which can be calculated from a frequency table. If 3 out of five students got the grade ‘5’, and the other two got a grade ‘3’, \(\mathbb {E}(X)=0.6\cdot 5 + 0.4 \cdot 3\).
- 2.
We omit the proof but it can be found in any linear algebra textbook, such as e.g. [1].
- 3.
Numpy is the Python library for handling arrays and fast numerical computations.
- 4.
Try’adam’.
- 5.
Try’binary_crossentropy’.
References
S. Axler, Linear Algebra Done Right (Springer, New York, 2015)
R. Vidal, Y. Ma, S. Sastry, Generalized Principal Component Analysis (Springer, London, 2016)
I. Goodfellow, Y. Bengio, A. Courville, Deep Learning (MIT Press, Cambridge, 2016)
D.H. Ballard, Modular learning in neural networks, in AAAI-87 Proceedings (AAAI, 1987), pp. 279–284
Y. LeCun, Modeles connexionnistes de l’apprentissage (Connectionist Learning Models) (Université P. et M. Curie (Paris 6), 1987)
P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.-A. Manzagol, Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)
Q.V. Le, M.A. Ranzato, R. Monga, M. Devin, K. Chen, G.S. Corrado, J. Dean, A.Y. Ng, Building high-level features using large scale unsupervised learning, in Proceedings of the 29th International Conference on Machine Learning. ICML (2012)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this chapter
Cite this chapter
Skansi, S. (2018). Autoencoders. In: Introduction to Deep Learning. Undergraduate Topics in Computer Science. Springer, Cham. https://doi.org/10.1007/978-3-319-73004-2_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-73004-2_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-73003-5
Online ISBN: 978-3-319-73004-2
eBook Packages: Computer ScienceComputer Science (R0)