Abstract
In convolution neural networks, digital multiplication operation is the arithmetic operation of the most space-consuming and power consumption. This paper trains convolutional neural network with three different data formats (float point, fixed point and dynamic fixed point) on two different datasets (MNIST, CIFAR-10). For each data set and each data format, the paper assesses the impact of the multiplication accuracy to the error rate at the end of the training. The results show that the network error rate which is trained with low accuracy fixed point has small difference with the network training error rate which is trained with floating point, and this phenomenon shows that the use of low precision can fully meet the training requirements in the process of training the network.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Williamson, D.: Dynamic scaling; iteration stages; digital filters; overflow probability; fixed point arithmetic; fixed-point filter. In: Dynamically Scaled Fixed Point Arithmetic, pp. 315–318. New York, NY, USA (1991)
Simard, P., Graf, H.P.: Backpropagation without multiplication. In: Advances in Neural Information Processing Systems, pp. 232–239 (1994)
Pham, P.-H., Jelaca, D., Farabet, C., Martini, B., LeCun, Y., Culurciello, E.: NeuFlow: dataflow vision processing system-on-a-chip. In: 2012 IEEE 55th International Midwest Symposium on Circuits and Systems (MWSCAS), pp. 1044–1047. IEEE (2012)
Vanhoucke, V., Senior, A., Mao, M.Z.: Improving the speed of neural networks on cpus. In: Proceedings of Deep Learning and Unsupervised Feature Learning NIPS Workshop (2011)
Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Le, Q., Mao, M., Ranzato, M., Senior, A. Tucker, P., Yang, K., Ng, A.Y.: Large scale distributed deep networks. In: NIPS 2012 (2012)
David, J., Kalach, K., Tittley, N.: Hardware complexity of modular multiplication andexponentiation. IEEE Trans. Comput. 56(10), 1308–1319 (2007)
Coates, A., Baumstarck, P., Le, Q., Ng, A. Y.: Scalable learning for object detection with gpu hardware. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009. IROS 2009, pp. 4287–4293. IEEE (2009)
Kim, S.K., McAfee, L.C., McMahon, P.L., Olukotun, K.: A highly scalable restricted Boltzmann machine FPGA implementation. In: International Conference on Field Programmable Logic and Applications, 2009. FPL 2009, pp. 367–372. IEEE (2009)
Gupta, S., Agrawal, A., Gopalakrishnan, K.: Deep Learing with Limited Numberical Precision (2015)
Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. Technical Report Arxiv Report 1302.4389, Universite de Montréal (2013)
Jarrett, K., Kavukcuoglu, K., Ranzato, M., LeCun, Y.: What is the best multi-stage architecture for object recognition? In: Proceedings of International Conference on Computer Vision (ICCV 2009), pp. 2146–2153. IEEE (2009)
Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., Temam, O. Diannao: a small footprint high-throughput accelerator for ubiquitous machine-learning. In: Proceedings of the 19th international conference on Architectural support for programming languages and operating systems, pp. 269–284. ACM (2014)
Holt, J.L., Baker, T.E.: Back propagation simulations using limited precision calculations. In: IJCNN-91-Seattle International Joint Conference on Neural Networks, 1991, vol. 2, pp. 121–126. IEEE (1991)
Savich, A.W., Moussa, M., Areibi, S.: The impact of arithmetic representation on implementing mlp-bp on fpgas: a study. Neural Netw. IEEE Trans. 18(1), 240–252 (2007)
Presley, R.K., Haggard, R.L.: A fixed point implementation of the backpropagation learning algorithm. In: Southeastcon 1994. Creative Technology Transfer-A Global Affair. Proceedings of the 1994 IEEE, pp. 136–138. IEEE (1994)
Nair, V., Hinton, G.: Rectified linear units improve restricted Boltzmann machines. In: ICML 2010 (2010)
Farabet, C., Martini, B., Corda, B., Akselrod, P., Culurciello, E., LeCun, Y. NeuFlow: Aruntime reconfigurable dataflow processor for vision. In: 2011 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 109–116. IEEE (2011)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical Report, University of Toronto (2009)
Wawrzynek, J., Asanovic, K., Kingsbury, B., Johnson, D., Beck, J., Morgan, N.: Spert-ii: a vector microprocessor system. Computer 29(3), 79–86 (1996)
Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: AISTATS 2011 (2011)
Acknowledgements
The authors of this paper are members of Shanghai Engineering Research Center of Intelligent Video Surveillance. In part by Technology Research Program of Ministry of Public Security of China under Grant 2015JSYJB26.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Cai, Y., Liang, C., Tang, Z., Li, H., Gong, S. (2018). Deep Neural Network with Limited Numerical Precision. In: Abawajy, J., Choo, KK., Islam, R. (eds) International Conference on Applications and Techniques in Cyber Security and Intelligence. ATCI 2017. Advances in Intelligent Systems and Computing, vol 580. Edizioni della Normale, Cham. https://doi.org/10.1007/978-3-319-67071-3_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-67071-3_8
Published:
Publisher Name: Edizioni della Normale, Cham
Print ISBN: 978-3-319-67070-6
Online ISBN: 978-3-319-67071-3
eBook Packages: EngineeringEngineering (R0)