Skip to main content
SpringerLink
Log in
Book cover

International Conference on Artificial Neural Networks

ICANN 2014: Artificial Neural Networks and Machine Learning – ICANN 2014 pp 539–546Cite as

Sampling Hidden Parameters from Oracle Distribution

  • Conference paper

  • 4276 Accesses

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 8681))

Abstract

A new sampling learning method for neural networks is proposed. Derived from an integral representation of neural networks, an oracle probability distribution of hidden parameters is introduced. In general rigorous sampling from the oracle distribution holds numerical difficulty, a linear-time sampling algorithm is also developed. Numerical experiments showed that when hidden parameters were initialized by the oracle distribution, following backpropagation converged faster to better parameters than when parameters were initialized by a normal distribution.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bengio, Y., Courville, A., Vincent, P.: Representation learning: A review and new perspectives. IEEE Trans. on PAMI 35(8), 1798–1828 (2013)

    Article  Google Scholar 

  2. Denoeux, T., Lengellé, R.: Initializing back propagation networks with prototypes. Neural Networks 6(3), 351–363 (1993)

    Article  Google Scholar 

  3. De Freitas, J.F.G., Niranjan, M., Gee, A.H., Doucet, A.: Sequential Monte Carlo methods to train neural network models. Neural Computation 12(4), 955–993 (2000)

    Article  Google Scholar 

  4. Kůrková, V.: Complexity estimates based on integral transforms induced by computational units. Neural Networks 33, 160–167 (2012)

    Article  MATH  Google Scholar 

  5. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 11, 2278–2324 (1998)

    Article  Google Scholar 

  6. LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.-R.: Efficient Backprop. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) NN: Tricks of the Trade, 2nd edn. LNCS, vol. 7700, pp. 9–48. Springer, Heidelberg (2012)

    Google Scholar 

  7. LeCun, Y., Cortes, C.: The MNIST database of handwritten digits, http://yann.lecun.com/exdb/mnist/

  8. Murata, N.: An integral representation of functions using three-layered networks and their approximation bounds. Neural Networks 9(6), 947–956 (1996)

    Article  Google Scholar 

  9. Sprecher, D.A.: A numerical implementation of Kolmogorov’s superpositions. Neural Networks 9(5), 765–772 (1996)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Schools of Advanced Science and Engineering, Waseda University, 3–4–1 Ohkubo, Shinjuku-ku, Tokyo, 169–8555, Japan

    Sho Sonoda & Noboru Murata

Authors
  1. Sho Sonoda
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Noboru Murata
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Informatics, University of Hamburg, Vogt-Kölln-Straße 30, 22527, Hamburg, Germany

    Stefan Wermter , Cornelius Weber  & Sven Magg ,  & 

  2. Department of Informatics, Nicolaus Compernicus University, ul. Grudziądzka 5, 87-100, Torun, Poland

    Włodzisław Duch

  3. Department of Modern Languages, University of Helsinki, P.O. Box 24, 00014, Helsinki, Finland

    Timo Honkela

  4. Institute of Information and Communication Technologies, Bulgarian Academy of Sciences, Acad. G. Bonchev str. bl. 25A, 1113, Sofia, Bulgaria

    Petia Koprinkova-Hristova

  5. Institute of Neural Information Processing, University of Ulm, 89069, Oberer Eselsberg, Ulm, Germany

    Günther Palm

  6. Department of Information Systems, Quartier UNIL-Dorigny, Bâtiment Internef, University of Lausanne, 1015, Lausanne, Switzerland

    Alessandro E. P. Villa

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this paper

Cite this paper

Sonoda, S., Murata, N. (2014). Sampling Hidden Parameters from Oracle Distribution. In: Wermter, S., et al. Artificial Neural Networks and Machine Learning – ICANN 2014. ICANN 2014. Lecture Notes in Computer Science, vol 8681. Springer, Cham. https://doi.org/10.1007/978-3-319-11179-7_68

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-11179-7_68

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-11178-0

  • Online ISBN: 978-3-319-11179-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation