Abstract
An autoencoder (AE) is one of the important neural network methods for dimensionality reduction problems. Unfortunately, however, deep AEs have the drawback in trainability which often makes obtaining a good performance a difficult task owing to their model complexity. This paper proposes a simple weight initialization algorithm called the principal component initialization (PCI) method to improve and stabilize the generalization performance of deep AEs in one shot. PCI uses orthogonal bases of the original data space obtained with principal component analysis and transposed ones as initial weights of the AEs. The proposed method significantly outperforms the current de facto standard initialization method for image reconstruction tasks.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Akinduko, A.A., Mirkes, E.M., Gorban, A.N.: SOM: stochastic initialization versus principal components. Inf. Sci. 364, 213–221 (2016). https://doi.org/10.1016/j.ins.2015.10.013
Baldi, P., Hornik, K.: Neural networks and principal component analysis: learning from examples without local minima. Neural Netw. 2(1), 53–58 (1989). https://doi.org/10.1016/0893-6080(89)90014-2
Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. In: Advances in Neural Information Processing Systems, pp. 153–160 (2007)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015). https://doi.org/10.1109/ICCV.2015.123
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Karakida, R., Okada, M., Amari, S.i.: Dynamical analysis of contrastive divergence learning: restricted Boltzmann machines with Gaussian visible units. Neural Netw. 79, 78–87 (2016). https://doi.org/10.1016/j.neunet.2016.03.013
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Oja, E.: Simplified neuron model as a principal component analyzer. J. Math. Biol. 15(3), 267–273 (1982). https://doi.org/10.1007/BF00275687
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Suzuki, A., Sakanashi, H. (2019). PCI: Principal Component Initialization for Deep Autoencoders. In: Tetko, I., Kůrková, V., Karpov, P., Theis, F. (eds) Artificial Neural Networks and Machine Learning – ICANN 2019: Deep Learning. ICANN 2019. Lecture Notes in Computer Science(), vol 11728. Springer, Cham. https://doi.org/10.1007/978-3-030-30484-3_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-30484-3_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-30483-6
Online ISBN: 978-3-030-30484-3
eBook Packages: Computer ScienceComputer Science (R0)