Abstract
One of the challenges faced in the success of Deep Neural Network (DNN) implementation is setting the values for various hyper-parameters, one of which is network topology that is closely related to the number of hidden layers and neurons. Determining the number of hidden layers and neurons is very important and influential in DNN learning performance. However, up to now, there has been no guidance on it. Determining these two numbers manually (usually through trial and error methods) to find fairly optimal arrangement is a time-consuming process. In this study, we propose the method used for determining the number of hidden layers was through the number of components formed on the principal component analysis (PCA). By using Forest Type Mapping Data Set, based on PCA analysis, it was found out that the number of hidden layers that provide the best accuracy was three. This is in accordance with the number of components formed in the principal component analysis which gave a cumulative variance of around 70%.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
UCI Machine Learning Repository Homepage, https://archive.ics.uci.edu/ml/datasets/Forest+type+mapping, last accessed 2019/01/10.
References
Madhiarasan, M., Deepa, S.N.: A novel criterion to select hidden neuron numbers in improved back propagation networks for wind speed forecasting. Appl. Intell. 44(4), 878–893 (2016). https://doi.org/10.1007/s10489-015-0737-z
Madhiarasan, M., Deepa, S.N.: Comparative analysis on hidden neurons estimation in multi layer perceptron neural networks for wind speed forecasting. Artif. Intell. Rev. 48(4), 449–471 (2017). https://doi.org/10.1007/s10462-016-9506-6
Qolomany, B., Maabreh, M., Al-Fuqaha, A., Gupta, A., Benhaddou, D.: Parameters optimization of deep learning models using particle swarm optimization. In: 13th International Wireless Communications and Mobile Computing Conference (IWCMC), pp. 1285–1290 (2017). https://doi.org/10.1109/IWCMC.2017.7986470
Thomas, A.J., Petridis, M., Walters, S.D., Gheytassi, S.M., Morgan, R.E.: Eng. Appl. Neural Netw. 744, 279–290 (2017). https://doi.org/10.1007/978-3-319-65172-9
Thomas, A.J., Walters, S.D., Gheytassi, S.M., Morgan, R.E., Petridis, M.: On the optimal node ratio between hidden layers: a probabilistic study. Int. J. Mach. Learn. Comput. 6(5), 241–247 (2016). https://doi.org/10.18178/ijmlc.2016.6.5.605
Guliyev, N.J., Ismailov, V.E.: On the approximation by single hidden layer feedforward neural networks with fixed weights. Neural Netw. 98, 296–304 (2018). https://doi.org/10.1016/j.neunet.2017.12.007
Nitta, T.: Resolution of singularities introduced by hierarchical structure in deep neural networks. IEEE Trans. Neural Netw. Learn. Syst. 28(10), 2282–2293 (2017). https://doi.org/10.1109/TNNLS.2016.2580741
Koutsoukas, A., Monaghan, K.J., Li, X., Huan, J.: Deep-learning: investigating deep neural networks hyper-parameters and comparison of performance to shallow methods for modeling bioactivity data. J. Cheminform. 9(1), 1–13 (2017). https://doi.org/10.1186/s13321-017-0226-y
Bunjongjit, S., Ngaopitakkul, A., Pothisarn, C., Jettanasen, C.: Improvement to reduce training time of back-propagation neural networks for discrimination between external short circuit and internal winding fault. In: International Conference on Information Science, Electronics and Electrical Engineering, Sapporo, pp. 614–618 (2014)
Chhachhiya, D., Sharma, A., Gupta, M.: Designing optimal architecture of neural network with particle swarm optimization techniques specifically for educational dataset. In: 7th International Conference on Cloud Computing, Data Science and Engineering - Confluence, Noida, pp. 52–57 (2017)
Lee, S., Ha, J., Zokhirova, M., Moon, H., Lee, J.: Background information of deep learning for structural engineering. Arch. Comput. Methods Eng. 25(1), 121–129 (2018). https://doi.org/10.1007/s11831-017-9237-0
Tej, M.L., Holban, S.: Comparative study of clustering distance measures to determine neural network architectures. In: IEEE 12th International Symposium on Applied Computational Intelligence and Informatics (SACI), pp. 189–194 (2018)
Tej, M.L., Holban, S.: Determining optimal neural network architecture using regression methods. In: International Conference on Development and Application Systems (DAS), pp. 180–189 (2018)
Tej, M.L.: Neural network architecture through data mining techniques. In: Proceedings of the IIER International Conference, Bucharest, Romania (2017)
Lorenzo, P.R., Nalepa, J., Kawulok, M., Ramos, L.S., Pastor, J.R.: Particle swarm optimization for hyper-parameter selection in deep neural networks. In: Proceedings of the 2017 on Genetic and Evolutionary Computation Conference - GECCO 2017, vol. 8, pp. 481–488 (2017). https://doi.org/10.1145/3071178.3071208
Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13, 281–305 (2012)
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., Bengio, Y.: An empirical evaluation of deep architectures on problems with many factors of variation. In: ICML (2007)
Bengio, Y., Goodfellow, I., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)
Mitchell, T.M.: Machine Learning. McGraw-Hill, Boston (1997)
Jollife, I.T.: Principal Component Analysis. Springer, New York (2002)
Di, W., Bhardwaj, A., Wei, J.: Deep Learning Essentials. Packt Publishing, Birmingham (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Ibnu Choldun R., M., Santoso, J., Surendro, K. (2020). Determining the Number of Hidden Layers in Neural Network by Using Principal Component Analysis. In: Bi, Y., Bhatia, R., Kapoor, S. (eds) Intelligent Systems and Applications. IntelliSys 2019. Advances in Intelligent Systems and Computing, vol 1038. Springer, Cham. https://doi.org/10.1007/978-3-030-29513-4_36
Download citation
DOI: https://doi.org/10.1007/978-3-030-29513-4_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-29512-7
Online ISBN: 978-3-030-29513-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)