Abstract
In this paper, we introduce a mechanism for designing the architecture of a Sparse Multi-Layer Perceptron network, for classification, called ForestNet. Networks built using our approach are capable of handling high-dimensional data and learning representations of both visual and non-visual data. The proposed approach first builds an ensemble of randomized trees in order to gather information on the hierarchy of features and their separability among the classes. Subsequently, such information is used to design the architecture of a sparse network, for a specific data set and application. The number of neurons is automatically adapted to the dataset. The proposed approach was evaluated using two non-visual and two visual datasets. For each dataset, 4 ensembles of randomized trees with different sizes were built. In turn, per ensemble, a sparse network architecture was designed using our approach and a fully connected network with same architecture was also constructed. The sparse networks defined using our approach consistently outperformed their respective tree ensembles, achieving statistically significant improvements in classification accuracy. While we do not beat state-of-art results with our network size and the lack of data augmentation techniques, our method exhibits very promising results, as the sparse networks performed similarly to their fully connected counterparts with a reduction of more than 98% of connections in the visual tasks.
This research was partially funded by The German Academic Exchange Service (DAAD—German: Deutscher Akademischer Austauschdienst) and the Emerging Field Initiative project of FAU Erlangen-Nürnberg: Big-Thera.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Biau, G., Scornet, E., Welbl, J.: Neural random forests. Sankhya A 81, 347–386 (2019)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001). https://doi.org/10.1023/A:1010933404324
Dheeru, D., Karra Taniskidou, E.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml
Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=rJl-b3RcF7
Frosst, N., Hinton, G.: Distilling a neural network into a soft decision tree. arXiv preprint arXiv:1711.09784 (2017)
Geurts, P., Ernst, D., Wehenkel, L.: Extremely randomized trees. Mach. Learn. 63(1), 3–42 (2006). https://doi.org/10.1007/s10994-006-6226-1
Humbird, K.D., Peterson, J.L., Mcclarren, R.G.: Deep neural network initialization with decision trees. IEEE Trans. Neural Netw. Learn. Syst. 30(5), 1286–1295 (2019). https://doi.org/10.1109/TNNLS.2018.2869694
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436 (2015)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
LeCun, Y., Cortes, C.: MNIST handwritten digit database (2010) . http://yann.lecun.com/exdb/mnist/
Miikkulainen, R., et al.: Evolving deep neural networks. In: Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293–312. Elsevier (2019)
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning (2011)
Paszke, A., et al.: Automatic differentiation in pytorch (2017)
Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Rodríguez-Salas, D., Gómez-Gil, P., Olvera-López, A.: Designing partially-connected, multilayer perceptron neural nets through information gain. In: The 2013 International Joint Conference on Neural Networks (IJCNN), pp. 1–5, August 2013. https://doi.org/10.1109/IJCNN.2013.6706991
Sethi, I.K.: Entropy nets: from decision trees to neural networks. Proc. IEEE 78(10), 1605–1613 (1990)
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. comput. 10(2), 99–127 (2002)
Steinberg, D., Colla, P.: CART: classification and regression trees. In: The Top Ten Algorithms in Data Mining, vol. 9, p. 179 (2009)
Utgoff, P.E.: Perceptron trees: a case study in hybrid concept representations. Connect. Sci. 1(4), 377–391 (1989)
Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bull. 1(6), 80–83 (1945)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Rodríguez-Salas, D., Ravikumar, N., Seuret, M., Maier, A. (2020). ForestNet – Automatic Design of Sparse Multilayer Perceptron Network Architectures Using Ensembles of Randomized Trees. In: Palaiahnakote, S., Sanniti di Baja, G., Wang, L., Yan, W. (eds) Pattern Recognition. ACPR 2019. Lecture Notes in Computer Science(), vol 12046. Springer, Cham. https://doi.org/10.1007/978-3-030-41404-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-41404-7_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-41403-0
Online ISBN: 978-3-030-41404-7
eBook Packages: Computer ScienceComputer Science (R0)