Skip to main content
SpringerLink
Log in

Neural Network Precision Tuning Using Stochastic Arithmetic

  • Conference paper
  • First Online:
Software Verification and Formal Methods for ML-Enabled Autonomous Systems (NSV 2022, FoMLAS 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13466))

Abstract

Neural networks can be costly in terms of memory and execution time. Reducing their cost has become an objective, especially when integrated in an embedded system with limited resources. A possible solution consists in reducing the precision of their neurons parameters. In this article, we present how to use auto-tuning on neural networks to lower their precision while keeping an accurate output. To do so, we use a floating-point auto-tuning tool on different kinds of neural networks. We show that, to some extent, we can lower the precision of several neural network parameters without compromising the accuracy requirement.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    http://cadna.lip6.fr.

  2. 2.

    http://promise.lip6.fr.

  3. 3.

    http://half.sourceforge.net.

  4. 4.

    https://keras.io.

  5. 5.

    https://pytorch.org.

  6. 6.

    https://www.hdfgroup.org.

  7. 7.

    https://pandas.pydata.org.

  8. 8.

    https://f0uriest.github.io/keras2c/.

  9. 9.

    http://yann.lecun.com/exdb/mnist.

  10. 10.

    https://www.cs.toronto.edu/~kriz/cifar.html.

References

  1. IEEE Standard for Floating-Point Arithmetic, IEEE Std 754-2019 (Revision of IEEE 754-2008), pp. 1–84 (2019)

    Google Scholar 

  2. Adjé, A., Khalifa, D.B., Martel, M.: Fast and efficient bit-level precision tuning. arXiv:2103.05241 [cs] (2021)

  3. Benmaghnia, H., Martel, M., Seladji, Y.: Fixed-point code synthesis for neural networks. Artificial Intelligence, Soft Computing and Applications, pp. 11–30 (2022). arXiv:2202.02095

  4. Chang, Y.-C., Roohi, N., Gao, S.: Neural Lyapunov control. In: 33rd Conference on Neural Information Processing Systems (NeurIPS 2019) (2020). arXiv:2005.00611

  5. Cherubin, S., Cattaneo, D., Chiari, M., Bello, A.D., Agosta, G.: TAFFO: tuning assistant for floating to fixed point optimization. IEEE Embed. Syst. Lett. 12, 5–8 (2020)

    Article  Google Scholar 

  6. Chesneaux, J.-M.: L’arithmétique stochastique et le logiciel CADNA, Habilitation à diriger des recherches. Université Pierre et Marie Curie, Paris, France (1995)

    Google Scholar 

  7. Chesneaux, J.-M., Graillat, S., Jezequel, F.: Numerical validation and assessment of numerical accuracy. Oxford e-Research Center (2009)

    Google Scholar 

  8. Chiang, W.-F., Baranowski, M., Briggs, I., Solovyev, A., Gopalakrishnan, G., Rakamarić, Z.: Rigorous floating-point mixed-precision tuning. In: Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages, POPL 2017, pp. 300–315. ACM, New York (2017)

    Google Scholar 

  9. Csendes, T.: Adversarial example free zones for specific inputs and neural networks. In: Proceedings of ICAI, pp. 76–84 (2020)

    Google Scholar 

  10. Damouche, N., Martel, M.: Mixed precision tuning with Salsa. In: Proceedings of the 8th International Joint Conference on Pervasive and Embedded Computing and Communication Systems, Porto, Portugal, pp. 47–56. SCITEPRESS - Science and Technology Publications (2018)

    Google Scholar 

  11. Darulova, E., Izycheva, A., Nasir, F., Ritter, F., Becker, H., Bastian, R.: Daisy - framework for analysis and optimization of numerical programs (tool paper). In: Beyer, D., Huisman, M. (eds.) TACAS 2018. LNCS, vol. 10805, pp. 270–287. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-89960-2_15

    Chapter  Google Scholar 

  12. Darulova, E., Kuncak, V.: Towards a compiler for reals. ACM Trans. Programm. Lang. Syst. (TOPLAS) 39, 8:1–8:28 (2017)

    Google Scholar 

  13. Dutta, S., Jha, S., Sankaranarayanan, S., Tiwari, A.: Output range analysis for deep feedforward neural networks. In: Dutle, A., Muñoz, C., Narkawicz, A. (eds.) NFM 2018. LNCS, vol. 10811, pp. 121–138. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77935-5_9

    Chapter  Google Scholar 

  14. Essam, M., Tang, T.B., Ho, E.T.W., Chen, H.: Dynamic point stochastic rounding algorithm for limited precision arithmetic in deep belief network training. In: 2017 8th International IEEE/EMBS Conference on Neural Engineering (NER), Shanghai, China, pp. 629–632. IEEE, May 2017

    Google Scholar 

  15. Graillat, S., Jézéquel, F., Picot, R., Févotte, F., Lathuilière, B.: Auto-tuning for floating-point precision with discrete stochastic arithmetic. J. Comput. Sci. 36, 101017 (2019)

    Article  MathSciNet  Google Scholar 

  16. Guo, H., Rubio-González, C.: Exploiting community structure for floating-point precision tuning. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, Amsterdam Netherlands, pp. 333–343. ACM, July 2018

    Google Scholar 

  17. Gupta, S., Agrawal, A., Gopalakrishnan, K., Narayanan, P.: Deep learning with limited numerical precision. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML 2015 (2015). arXiv:1502.02551

  18. Ho, N.-M., Silva, H.D., Wong, W.-F.: GRAM: a framework for dynamically mixing precisions in GPU applications. ACM Trans. Archit. Code Optim. 18, 1–24 (2021)

    Article  Google Scholar 

  19. Hodován, R., Kiss, Á.: Practical improvements to the minimizing delta debugging algorithm. In: Proceedings of the 11th International Joint Conference on Software Technologies, Lisbon, Portugal, pp. 241–248. SCITEPRESS - Science and Technology Publications (2016)

    Google Scholar 

  20. Ioualalen, A., Martel, M.: Neural network precision tuning. In: Parker, D., Wolf, V. (eds.) QEST 2019. LNCS, vol. 11785, pp. 129–143. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30281-8_8

    Chapter  Google Scholar 

  21. Jézéquel, F., Hoseininasab, S., Hilaire, T.: Numerical validation of half precision simulations. In: Rocha, Á., Adeli, H., Dzemyda, G., Moreira, F., Ramalho Correia, A.M. (eds.) WorldCIST 2021. AISC, vol. 1368, pp. 298–307. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72654-6_29

    Chapter  Google Scholar 

  22. Kotipalli, P.V., Singh, R., Wood, P., Laguna, I., Bagchi, S.: AMPT-GA: automatic mixed precision floating point tuning for GPU applications. In: Proceedings of the ACM International Conference on Supercomputing, Phoenix, Arizona, pp. 160–170. ACM, June 2019

    Google Scholar 

  23. Laguna, I., Wood, P.C., Singh, R., Bagchi, S.: GPUMixer: performance-driven floating-point tuning for GPU scientific applications. In: Weiland, M., Juckeland, G., Trinitis, C., Sadayappan, P. (eds.) ISC High Performance 2019. LNCS, vol. 11501, pp. 227–246. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20656-7_12

    Chapter  Google Scholar 

  24. Lam, M.O., Hollingsworth, J.K., de Supinski, B.R., Legendre, M.P.: Automatically adapting programs for mixed-precision floating-point computation. In: Proceedings of the 27th International ACM Conference on International Conference on Supercomputing, ICS 2013, pp. 369–378. ACM, New York (2013)

    Google Scholar 

  25. Lam, M.O., Vanderbruggen, T., Menon, H., Schordan, M.: Tool integration for source-level mixed precision. In: 2019 IEEE/ACM 3rd International Workshop on Software Correctness for HPC Applications (Correctness), pp. 27–35 (2019)

    Google Scholar 

  26. Lauter, C., Volkova, A.: A framework for semi-automatic precision and accuracy analysis for fast and rigorous deep learning. arXiv:2002.03869 [cs] (2020)

  27. Lin, W., et al.: Robustness verification of classification deep neural networks via linear programming. In: Conference on Computer Vision and Pattern Recognition (2019)

    Google Scholar 

  28. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: 6th International Conference on Learning Representations, ICLR (2019). arXiv:1706.06083

  29. Mellempudi, N., Srinivasan, S., Das, D., Kaul, B.: Mixed precision training with 8-bit floating point. arXiv:1905.12334 [cs, stat] (2019)

  30. Menon, H., et al.: ADAPT: algorithmic differentiation applied to floating-point precision tuning. In: SC18: International Conference for High Performance Computing, Networking, Storage and Analysis, Dallas, TX, USA, pp. 614–626. IEEE, November 2018

    Google Scholar 

  31. Na, T., Ko, J.H., Kung, J., Mukhopadhyay, S.: On-chip training of recurrent neural networks with limited numerical precision. In: 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA, pp. 3716–3723. IEEE, May 2017

    Google Scholar 

  32. Rakin, A.S. et al.: RA-BNN: constructing robust & accurate binary neural network to simultaneously defend adversarial bit-flip attack and improve accuracy. arXiv:2103.13813 [cs, eess] (2021)

  33. Rubio-González, C.et al.: Precimonious: tuning assistant for floating-point precision. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC 2013, pp. 27:1–27:12. ACM, New York (2013)

    Google Scholar 

  34. Singh, G., Gehr, T., Mirman, M., Püschel, M., Vechev, M.: Fast and effective robustness certification. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, NeurIPS, pp. 10825–10836 (2018)

    Google Scholar 

  35. Su, C., Zhou, S., Feng, L.. Zhang, W.: Towards high performance low bitwidth training for deep neural networks, J. Semicond. 41, 022404 (2020). https://iopscience.iop.org/article/10.1088/1674-4926/41/2/022404

  36. Tjeng, V., Xiao, K., Tedrake, R.: Evaluating robustness of neural networks with mixed integer programming. arXiv:1711.07356 [cs] (2019)

  37. Vignes, J.: A stochastic arithmetic for reliable scientific computation. Math. Comput. Simul. 35, 233–261 (1993)

    Article  MathSciNet  Google Scholar 

  38. Vignes, J.: Discrete stochastic arithmetic for validating results of numerical software. Numer. Algorithms 37, 377–390 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  39. Wang, N., Choi, J., Brand, D., Chen, C.-Y., Gopalakrishnan, K.: Training deep neural networks with 8-bit floating point numbers. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31, pp. 7686–7695. Curran Associates Inc. (2018). arXiv:1812.08011. http://papers.nips.cc/paper/7994-training-deep-neural-networks-with-8-bit-floating-point-numbers.pdf

  40. Zeller, A., Hildebrandt, R.: Simplifying and isolating failure-inducing input. IEEE Trans. Softw. Eng. 28, 183–200 (2002)

    Article  Google Scholar 

  41. Zombori, D.: Verification of artificial neural networks via MIPVerify and SCIP, SCAN (2020)

    Google Scholar 

  42. Zombori, D., Bánhelyi, B., Csendes, T., Megyeri, I., Jelasity, M.: Fooling a complete neural network verifier. In: The 9th International Conference on Learning Representations (ICLR) (2021)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the InterFLOP (ANR-20-CE46-0009) project of the French National Agency for Research (ANR).

Author information

Authors and Affiliations

  1. Sorbonne Université, CNRS, LIP6, 75005, Paris, France

    Quentin Ferro, Stef Graillat, Thibault Hilaire, Fabienne Jézéquel & Basile Lewandowski

  2. Université Paris-Panthéon-Assas, 75005, Paris, France

    Fabienne Jézéquel

Authors
  1. Quentin Ferro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stef Graillat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thibault Hilaire
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fabienne Jézéquel
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Basile Lewandowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Quentin Ferro .

Editor information

Editors and Affiliations

  1. The Hebrew University of Jerusalem, Jerusalem, Israel

    Omri Isac

  2. Rensselaer Polytechnic Institute, New York, NY, USA

    Radoslav Ivanov

  3. The Hebrew University of Jerusalem, Jerusalem, Israel

    Guy Katz

  4. Palo Alto, CA, USA

    Nina Narodytska

  5. University of Trieste, Trieste, Italy

    Laura Nenzi

Appendices

Appendix A Sine Neural Network

1.1 Appendix A.1 Type Distribution for Sine Approximation with Input Value 2.37 Considering One Type per Neuron

See Fig. 10.

Fig. 10.
figure 10

Number of variables of each type and computation time for the sine neural network with input value 2.37

Full size image

1.2 Appendix A.2 Type Distribution for Sine Approximation with Input Value 2.37 Considering One Type per Layer

See Fig. 11.

Fig. 11.
figure 11

Precision of each layer for the sine neural network with input value 2.37

Full size image

Appendix B MNIST Neural Network

1.1 Appendix B.1 Type Distribution for MNIST with Test_data[61] Input Considering One Type per Layer

See Fig. 12.

Fig. 12.
figure 12

Precision of each layer for MNIST neural network with test_data[61] input

Full size image

1.2 Appendix B.2 Type Distribution for MNIST with Test_data[91] Input Considering one type per neuron

See Fig. 13.

Fig. 13.
figure 13

Number of variables of each type and computation time for MNIST neural network with test_data[91] input

Full size image

1.3 Appendix B.3 Type Distribution for MNIST with Test_data[91] Input Considering One Type per Layer

See Fig. 14.

Fig. 14.
figure 14

Precision of each layer for MNIST neural network with test_data[91] input

Full size image

Appendix C CIFAR Neural Network

1.1 Appendix C.1 Type Distribution for CIFAR with Test_data[386] Input Considering One Type per Layer

See Fig. 15.

Fig. 15.
figure 15

Precision of each layer for CIFAR neural network with test_data[386] input

Full size image

1.2 Appendix C.2 Type Distribution for CIFAR with Test_data[731] Input Considering One Type per Neuron

See Fig. 16.

Fig. 16.
figure 16

Number of variables of each type and computation time for CIFAR neural network with test_data[731] input

Full size image

1.3 Appendix C.3 Type Distribution for CIFAR with Test_data[731] Input Considering One Type per Layer

See Fig. 17.

Fig. 17.
figure 17

Precision of each layer for CIFAR neural network with test_data[731] input

Full size image

Appendix D Inverted pendulum

1.1 Appendix D.1 Type Distribution for the Inverted Pendulum with Input (0.5,0.5) Considering One Type per Layer

See Fig. 18.

Fig. 18.
figure 18

Precision of each layer for the pendulum neural network with input (0.5, 0.5)

Full size image

1.2 Appendix D.2 Type Distribution for the Inverted Pendulum with Input \((-3,-6)\) Considering One Type per Neuron

See Fig. 19.

Fig. 19.
figure 19

Number of variables of each type and computation time for the pendulum neural network with input \({(-3,-6)}\)

Full size image

1.3 Appendix D.3 Type Distribution for the Inverted Pendulum with Input \((-3,-6)\) Considering One Type per Layer

See Fig. 20.

Fig. 20.
figure 20

Precision of each layer for the pendulum neural network with input \((-3,-6)\)

Full size image

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ferro, Q., Graillat, S., Hilaire, T., Jézéquel, F., Lewandowski, B. (2022). Neural Network Precision Tuning Using Stochastic Arithmetic. In: Isac, O., Ivanov, R., Katz, G., Narodytska, N., Nenzi, L. (eds) Software Verification and Formal Methods for ML-Enabled Autonomous Systems. NSV FoMLAS 2022 2022. Lecture Notes in Computer Science, vol 13466. Springer, Cham. https://doi.org/10.1007/978-3-031-21222-2_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-21222-2_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-21221-5

  • Online ISBN: 978-3-031-21222-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation