Skip to main content

The α-EM algorithm: A block connectable generalized leaning tool for neural networks

Plasticity Phenomena (Maturing, Learning and Memory)

Part of the Lecture Notes in Computer Science book series (LNCS,volume 1240)

Abstract

The α-divergence is utilized to derive a generalized expectation and maximization algorithm (EM algorithm). This algorithm has a wide range of applications. In this paper, neural network learning for mixture probabilities is focused. The α-EM algorithm includes the existing EM algorithm as a special case since that corresponds to α = −1. The parameter α specifies a probability weight for the learning. This number affects learning speed and local optimality. In the discussions of update equations of neural nets, extensions of basic statistics such as Fisher's efficient score, his measure of information and Cramér-Rao's inequality are also given. Besides, this paper unveils another new idea. It is found that the cyclic EM structure can be used as a building block to generate a learning systolic array. Attaching monitors to this systolic array makes it possible to create a functionally distributed learning system.

Keywords

  • Efficient Score
  • Probability Weight
  • Systolic Array
  • Expert Network
  • Differentiable Convex Function

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/BFb0032507
  • Chapter length: 10 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   74.99
Price excludes VAT (USA)
  • ISBN: 978-3-540-69074-0
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   99.00
Price excludes VAT (USA)

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm, J. R. Stat. Soc. Sr. B. 39 (1978) 1–38

    Google Scholar 

  2. Jordan, M.J., Jacobs, R.A.: Hierarchical mixtures of experts and the EM algorithm, Neural Computation 6 (1994) 181–214

    Google Scholar 

  3. Jordan, M.J., Xu, L.: Convergence results for the EM approach to mixtures of experts architecture, Neural Networks 8 (1995) 1409–1431

    Google Scholar 

  4. Amari, S.: Information geometry of the EM and em algorithms for neural nets, Neural Networks 8 (1995) 1379–1408

    Google Scholar 

  5. Amari, S.: Information, geometry of neural networks — New Bayesian duality theory-, Proc. ICONIP'96 1 (1996) 3–6

    Google Scholar 

  6. Xu, L.: Bayesian-Kullback Ying-Yang Machine: Reviews and new results, Proc. ICONIP'96, 1 (1996) 59–67

    Google Scholar 

  7. Rényi, A.: On measures of entropy and information, Proc. 4th Berkeley Symp. Math. Stat. and Pr., 1 (1960) 547–561

    Google Scholar 

  8. Havrda, J.H., Chavat, F.: Qualification methods of classification processes: Concepts of structural α. entropy, Kybernetica, 3 (1967) 30–35

    Google Scholar 

  9. Amari, S., Nagaoka, H.: Methods of Information Geometry (in Japanese), Iwanami Publishing Co. (1993)

    Google Scholar 

  10. Csiszár, I: A class of measure of informativity of observation channels, Periodica Mathematica Hungarica 2 (1972) 191–213

    Google Scholar 

  11. Matsuyama, Y.: The α-EM algorithm and its block connections, Tech. Report, CML-96-1 (1996) Computation Mechanism Lab., Dept. of EECE, Waseda Univ.

    Google Scholar 

  12. Matsuyama, Y.: The weighted EM learning and monitoring structure, Info. Proc. Soc. of Japan, 54th Convention Record (1997) 6G-04

    Google Scholar 

  13. Matsuyama, Y.: The Weighted EM algorithm and Block Monitoring, Proc. ICNN, (1997)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 1997 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Matsuyama, Y. (1997). The α-EM algorithm: A block connectable generalized leaning tool for neural networks. In: Mira, J., Moreno-Díaz, R., Cabestany, J. (eds) Biological and Artificial Computation: From Neuroscience to Technology. IWANN 1997. Lecture Notes in Computer Science, vol 1240. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0032507

Download citation

  • DOI: https://doi.org/10.1007/BFb0032507

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-63047-0

  • Online ISBN: 978-3-540-69074-0

  • eBook Packages: Springer Book Archive