Abstract
Capsule Networks are an emerging extension of the traditional multilayer perceptron model, providing classification and estimated pose parameters through extra computational data models such as vectors and matrices. With extra data comes extra processing and meticulous data manipulation and this paper presents a scalable GPU optimization for the training and evaluation of such networks. A novel data abstraction maps the individual values of this tensor to one-dimensional arrays to 2D grids of multi-dimensional elements, enabling the removal of redundant data movement and other structural overhead found in common machine learning libraries. Assuming a single loss function, adapted for vector-based forward and backward propagation, this optimization involves combining successive operations found in processing into kernel calls, albeit at the expense of higher memory resource utilization. Despite that, this GPU based approach reached 33 times speed up in forward propagation and 130 times speed up in back propagation against optimal single threaded CPU calculations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abadi, M., Agarwal, A., Barham, P., et al.: Tensorflow: large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467 (2016)
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., Shelhamer, E.: cudnn: efficient primitives for deep learning. CoRR, abs/1410.0759 (2014)
Coates, A., Huval, B., Wang, T., Wu, D.J., Ng, A.Y., Catanzaro, B.: Deep learning with cots hpc systems. In: Proceedings of the 30th International Conference on International Conference on Machine Learning – Volume 28, ICML’13, pp. III–1337–III–1345. JMLR.org (2013)
Di, W., Bhardwaj, A., Wei, J.: Deep Learning Essentials: Your Hands-on Guide to the Fundamentals of Deep Learning and Neural Network Modeling, 1 edn. Packt Publishing, Birmingham (2018)
Hecht-Neilsen, R.: Iii.3 – theory of the backpropagation neural network*. In: Wechsler, H. (ed.) Neural Networks for Perception, pp. 65–93. Academic Press, Inc., San Diego, CA (1992). https://doi.org/10.1016/B978-0-12-741252-8.50010-8
Hinton, G.E., Sabour, S., Frosst, N.: Matrix capsules with EM routing. In: International Conference on Learning Representations (2018)
Kanungo, T., Mount, D.M., Netanyahu, N.S., Piatko, C.D., Silverman, R., Wu, A.Y.: An efficient k-means clustering algorithm: analysis and implementation. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 881–892 (2002)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR, abs/1412.6980 (2014)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105. Curran Associates, Inc., Red Hook, NY (2012)
LeCun, Y., Cortes, C.: MNIST handwritten digit database. http://yann.lecun.com/exdb/mnist/ (2010). Last Accessed 01 Sept 2019
Minai, A.A.: Acceleration of backpropagation through learning rate and momentum adaptation. In: Proceedings of the International Joint Conference on Neural Networks, pp. 676–679 (1990)
Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. CoRR, abs/1710.09829 (2017)
Sanders, J., Kandrot, E.: CUDA by Example: An Introduction to General-Purpose GPU Programming, 1st edn. Addison-Wesley Professional, Boston, Mass (2010)
Sanderson, C.: Armadillo C++ Linear Algebra Library. https://doi.org/10.5281/zenodo.55251, June 2016. Last Accessed Date 01 Sept 2019
Acknowledgements
This material is based in part upon work supported by the National Science Foundation under grant number IIA-1301726. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Lopez, D.A., Wu, R., Barford, L., Harris, F.C. (2019). A Memory Layout for Dynamically Routed Capsule Layers. In: Latifi, S. (eds) 16th International Conference on Information Technology-New Generations (ITNG 2019). Advances in Intelligent Systems and Computing, vol 800. Springer, Cham. https://doi.org/10.1007/978-3-030-14070-0_43
Download citation
DOI: https://doi.org/10.1007/978-3-030-14070-0_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-14069-4
Online ISBN: 978-3-030-14070-0
eBook Packages: EngineeringEngineering (R0)