Skip to main content
SpringerLink
Log in

Lower Bounds on Complexity of Shallow Perceptron Networks

  • Conference paper
  • First Online:
Book cover Engineering Applications of Neural Networks (EANN 2016)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 629))

Abstract

Model complexity of shallow (one-hidden-layer) perceptron networks computing multivariable functions on finite domains is investigated. Lower bounds are derived on growth of the number of network units or sizes of output weights in terms of variations of functions to be computed. A concrete construction of a class of functions which cannot be computed by percetron networks with considerably smaller numbers of units and output weights than the sizes of the function’s domains is presented. In particular, functions on Boolean d-dimensional cubes are constructed which cannot be computed by shallow perceptron networks with numbers of hidden units and sizes of output weights depending on d polynomially. A subclass of these functions is described whose elements can be computed by two-hidden-layer networks with the number of units depending on d linearly.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Fine, T.L.: Feedforward Neural Network Methodology. Springer, Heidelberg (1999)

    MATH  Google Scholar 

  2. Kecman, V.: Learning and Soft Computing. MIT Press, Cambridge (2001)

    MATH  Google Scholar 

  3. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)

    Article  Google Scholar 

  4. Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2, 1–127 (2009)

    Article  MATH  Google Scholar 

  6. LeCunn, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)

    Article  Google Scholar 

  7. Ba, L.J., Caruana, R.: Do deep networks really need to be deep?. In: Ghahrani, Z., et al. (eds.) Advances in Neural Information Processing Systems, vol. 27, pp. 1–9 (2014)

    Google Scholar 

  8. Kainen, P.C., Kůrková, V., Sanguineti, M.: Dependence of computational models on input dimension: tractability of approximation and optimization tasks. IEEE Trans. Inf. Theory 58, 1203–1214 (2012)

    Article  MathSciNet  Google Scholar 

  9. Maiorov, V., Pinkus, A.: Lower bounds for approximation by MLP neural networks. Neurocomputing 25, 81–91 (1999)

    Article  MATH  Google Scholar 

  10. Bianchini, M., Scarselli, F.: On the complexity of neural network classifiers: a comparison between shallow and deep architectures. IEEE Trans. Neural Netw. Learn. Syst. 25(8), 1553–1565 (2014)

    Article  Google Scholar 

  11. Bengio, Y., Delalleau, O., Roux, N.L.: The curse of highly variable functions for local kernel machines. In: Advances in Neural Information Processing Systems, vol. 18, pp. 107–114. MIT Press (2006)

    Google Scholar 

  12. Kůrková, V., Sanguineti, M.: Model complexities of shallow networks representing highly varying functions. Neurocomputing 171, 598–604 (2016)

    Article  Google Scholar 

  13. Ito, Y.: Finite mapping by neural networks and truth functions. Math. Sci. 17, 69–77 (1992)

    MathSciNet  MATH  Google Scholar 

  14. Barron, A.R.: Neural net approximation. In: Narendra, K. (ed.) Proceedings of the 7th Yale Workshop on Adaptive and Learning Systems, pp. 69–72. Yale University Press (1992)

    Google Scholar 

  15. Kůrková, V.: Dimension-independent rates of approximation by neural networks. In: Warwick, K., Kárný, M. (eds.) Computer-Intensive Methods in Control and Signal Processing: The Curse of Dimensionality, pp. 261–270. Birkhäuser, Boston (1997)

    Chapter  Google Scholar 

  16. Kůrková, V., Sanguineti, M.: Comparison of worst-case errors in linear and neural network approximation. IEEE Trans. Inf. Theory 48, 264–275 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  17. Kainen, P.C., Kůrková, V., Vogt, A.: A Sobolev-type upper bound for rates of approximation by linear combinations of heaviside plane waves. J. Approximation Theory 147, 1–10 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  18. Kůrková, V.: Minimization of error functionals over perceptron networks. Neural Comput. 20, 250–270 (2008)

    MathSciNet  MATH  Google Scholar 

  19. Kůrková, V.: Complexity estimates based on integral transforms induced by computational units. Neural Netw. 33, 160–167 (2012)

    Article  MATH  Google Scholar 

  20. Gnecco, G., Sanguineti, M.: On a variational norm tailored to variable-basis approximation schemes. IEEE Trans. Inf. Theory 57, 549–558 (2011)

    Article  MathSciNet  Google Scholar 

  21. Kůrková, V., Savický, P., Hlaváčková, K.: Representations and rates of approximation of real-valued Boolean functions by neural networks. Neural Netw. 11, 651–659 (1998)

    Article  Google Scholar 

  22. Cover, T.: Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Trans. Electron. Comput. 14, 326–334 (1965)

    Article  MATH  Google Scholar 

  23. Erdös, P., Spencer, J.H.: Probabilistic Methods in Combinatorics. Academic Press, New York (1974)

    MATH  Google Scholar 

  24. Sloane, N.J.A.: A library of Hadamard matrices. http://www.research.att.com/~njas/hadamard/

  25. MacWilliams, F., Sloane, N.J.A.: The Theory of Error-Correcting Codes. North-Holland, Asterdam (1977)

    MATH  Google Scholar 

Download references

Acknowledgments

This work was partially supported by the Czech Grant Agency grant 15-18108S and institutional support of the Institute of Computer Science RVO 67985807.

Author information

Authors and Affiliations

  1. Institute of Computer Science, Czech Academy of Sciences, Pod Vodárenskou věží  2, 18207, Prague, Czech Republic

    Věra Kůrková

Authors
  1. Věra Kůrková
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Věra Kůrková .

Editor information

Editors and Affiliations

  1. Robert Gordon University, Aberdeen, United Kingdom

    Chrisina Jayne

  2. Lab of Forest Informatics (FiLAB), Democritus University of Thrace Lab of Forest Informatics (FiLAB), Orestiada, Greece

    Lazaros Iliadis

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Kůrková, V. (2016). Lower Bounds on Complexity of Shallow Perceptron Networks. In: Jayne, C., Iliadis, L. (eds) Engineering Applications of Neural Networks. EANN 2016. Communications in Computer and Information Science, vol 629. Springer, Cham. https://doi.org/10.1007/978-3-319-44188-7_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-44188-7_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-44187-0

  • Online ISBN: 978-3-319-44188-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation