Abstract
Winograd convolution is widely used in deep neural networks (DNNs). Existing work for DNNs considers only the subset Winograd algorithms that are equivalent to Toom-Cook convolution. We investigate a wider range of Winograd algorithms for DNNs and show that these additional algorithms can significantly improve floating point (FP) accuracy in many cases. We present results for three FP formats: fp32, fp16 and bf16 (a truncated form of fp32) using 2000 inputs from the ImageNet dataset. We found that in fp16 this approach gives us up to 6.5 times better image recognition accuracy in one important case while maintaining the same number of elementwise multiplication operations in the innermost loop. In bf16 the convolution can be computed using \(5\%\) fewer innermost loop multiplications than with currently used Winograd algorithms while keeping the accuracy of image recognition the same as for direct convolution method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Barabasz, B., Anderson, A., Soodhalter, K.M., Gregg, D.: Error analysis and improving the accuracy of winograd convolution for DNNs. CoRR abs/1803.10986 (2018). http://arxiv.org/abs/1803.10986
Biggs, N.L.: Discrete Mathematics, 2nd edn. Oxford University Press, New York (2002)
Blahut, R.E.: Fast Algorithms for Signal Processing. Cambridge University Press, New York (2010)
Cook, S.A.: On the minimum computation time of functions. Ph.D. thesis, Harvard University, Cambridge, Massachusetts (1966)
Lavin, A., Gray, S.: Fast algorithms for convolutional neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4013–4021. IEEE, Las Vegas (2016)
Meng, L., Brothers, J.: Efficient winograd convolution via integer arithmetic. CoRR abs/1901.01965 (2019)
Selesnick, I.W., Burrus, C.S.: Extending winograd’s small convolution algorithm to longer lengths. In: 1994 IEEE International Symposium on Circuits and Systems, ISCAS 1994, London, England, UK, 30 May–2 June 1994, pp. 449–452 (1994)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)
Tolimieri, R., An, M., Lu, C.: Algorithms For Discrete Fourier Transform and Convolution, 2nd edn. Springer, New York (1997). https://doi.org/10.1007/978-1-4757-2767-8
Toom, A.L.: The complexity of a scheme of functional elements realizing multiplication of integers. Sov. Math. Dokl. 3, 714–716 (1963)
Vincent, K., Stephano, K., Frumkin, M., Ginsburg, B., Demouth, J.: On improving the numerical stability of winograd convolutions. In: Proceedings of the 5th International Conference on Learning Representations, Toulon, France, p. 4 (2017). https://openreview.net/forum?id=H1ZaRZVKg
Winograd, S.: Arithmetic Complexity Computations. SIAM Publications, Bristol (1980)
Zhao, Y., Wang, D., Wang, L., Liu, P.: A faster algorithm for reducing the computational complexity of convolutional neural networks. Algorithms 11(10) (2018). https://doi.org/10.3390/a11100159, http://www.mdpi.com/1999-4893/11/10/159
Acknowledgements
This work was supported by Science Foundation Ireland grant 12/IA/1381. We also extend our thanks to Andrew Mundy from Arm ML Research Lab for his contribution.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Barabasz, B., Gregg, D. (2019). Winograd Convolution for DNNs: Beyond Linear Polynomials. In: Alviano, M., Greco, G., Scarcello, F. (eds) AI*IA 2019 – Advances in Artificial Intelligence. AI*IA 2019. Lecture Notes in Computer Science(), vol 11946. Springer, Cham. https://doi.org/10.1007/978-3-030-35166-3_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-35166-3_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-35165-6
Online ISBN: 978-3-030-35166-3
eBook Packages: Computer ScienceComputer Science (R0)