Layered Neural Networks

  • Eytan Domany
  • Ronny Meir
Part of the Physics of Neural Networks book series (NEURAL NETWORKS)


Some of the recent work done on layered feed-forward networks is reviewed. First we describe exact solutions for the dynamics of such networks, which are expected to respond to an input by going through a sequence of preassigned states on the various layers. The family of networks considered has a variety of interlayer couplings: linear and nonlinear Hebbian, Hebbian with Gaussian synaptic noise and with various kinds of dilution, and the pseudoinverse (projector) matrix of couplings. In all cases our solutions take the form of layer-to-layer recursions for the mean overlap with a (random) key pattern and for the width of the embedding field distribution. Dynamics is governed by the fixed points of these recursions. For all cases nontrivial domains of attraction of the memory states are found. Next we review studies of unsupervised leaming in such networks and the emergence of orientation-selective cells. Finally the main ideas of three supervised leaming procedures, recendy introduced for layered networks, are oudined. All three procedures are based on a search in the space of intemal representations; one is designed for leaming in networks with fixed architecture and has no associated convergence theorem, whereas the other two are guaranteed to converge but may require expansion of the network by an uncontrolled number of hidden units.


Hide Layer Internal Representation Layered Network Hide Unit Iterate Learning 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 9.1
    For a recent review of physicists’ contributions see W. Kinzel, Physica Scripta T25, 144 (1989)ADSCrossRefGoogle Scholar
  2. 9.2
    E. Domany, R. Meir and W. Kinzel: Europhys. Lett. 2, 175 (1986)ADSCrossRefGoogle Scholar
  3. 9.3
    R. Meir and E. Domany: Phys. Rev. Lett. 59, 359 (1987)ADSCrossRefGoogle Scholar
  4. R. Meir and E. Domany: Europhys. Lett. 4, 645 (1988)ADSCrossRefGoogle Scholar
  5. R. Meir and E. Domany Phys. Rev. A 37, 608 (1988)ADSCrossRefGoogle Scholar
  6. 9.4
    R. Meir: J. Phys. (Paris) 49, 201 (1988)MathSciNetCrossRefGoogle Scholar
  7. 9.5
    B. Derrida and R. Meir: Phys. Rev. A 38, 3116 (1988)MathSciNetADSCrossRefGoogle Scholar
  8. 9.6
    E. Domany, W. Kinzel and R. Meir: J. Phys. A 22, 2081 (1989)MathSciNetADSCrossRefGoogle Scholar
  9. 9.7
    R. Linsker: Proc. Nat. Acad. Sci. USA 83, 7508–7512 (1986)ADSCrossRefGoogle Scholar
  10. 9.8
    R. Linsker: Proc. Nat. Acad. Sci. USA 83, 8390–8394 (1986)ADSCrossRefGoogle Scholar
  11. 9.9
    R. Linsker: Proc. Nat. Acad. Sci. USA 83, 8779–8783 (1986)ADSCrossRefGoogle Scholar
  12. 9.10
    R. Linsker, in:Computer Simulation in Brain Science, edited by R. Coltrili (Cambridge University Press, Cambridge 1988)Google Scholar
  13. R. Linsker: IEEE Computer (March 1988) 105–117Google Scholar
  14. 9.12R. Linsker: to be published in the proceedings of the 1988 Denver Conference on Neural Information Processing Systems (Morgan Kauffman)Google Scholar
  15. 9.13
    R. Meir and E. Domany: Phys. Rev. A 37, 2660 (1988)ADSCrossRefGoogle Scholar
  16. 9.14
    T. Grossman, R. Meir, and E. Domany: Complex Systems, 2, 555 (1988)MathSciNetMATHGoogle Scholar
  17. 9.15
    P. Rujan and M. Marchand: Complex Systems 3, 229 (1989)MATHGoogle Scholar
  18. 9.16
    M. Mézard and J.P. Nadal: J. Phys. A 22, 2191 (1989)MathSciNetADSCrossRefGoogle Scholar
  19. 9.17
    See for example J.E. Hopcroft and R.L. Mattson, Synthesis of Minimal Threshold Logic Networks, IEEE Trans. Electronic Computers, EC-14, 552 (1965)CrossRefGoogle Scholar
  20. 9.18
    J.J. Hopfield: Proc. Natl. Acad. USA 79, 2554 (1982)MathSciNetADSCrossRefGoogle Scholar
  21. 9.19
    DJ. Amit, H. Gutfreund, and H. Sompolinsky: Ann. Phys. 173, 30 (1987)ADSCrossRefGoogle Scholar
  22. 9.20
    H. Sompolinsky, in: Heidelberg Colloquium on Glassy Dynamics edited by J.L. van Hemmen and I. Morgenstem, Lecture Notes in Physics Vol. 275 (Springer, Berlin, Heidelberg 1987)CrossRefGoogle Scholar
  23. 9.21
    B. Derrida, E. Gardner, and A. Zippelius: Europhys. Lett. 4, 167 (1987)ADSCrossRefGoogle Scholar
  24. 9.22
    See for example E. Domany, J. Stat. Phys. 51, 743 (1988)MathSciNetADSMATHCrossRefGoogle Scholar
  25. 9.23See H. Ritter, K. Obermayer, K. Schulten, and J. Rubner, this volume, Chap. 8Google Scholar
  26. 9.24
    E. Gardner: J. Phys. A 21, 257 (1988)MathSciNetADSCrossRefGoogle Scholar
  27. E. Gardner and B. Derrida, J. Phys. A 21, 271 (1988)MathSciNetADSCrossRefGoogle Scholar
  28. 9.25
    W. Krauth and M. Opper: J. Phys. A 22, L519 (1989)ADSCrossRefGoogle Scholar
  29. 9.26
    L Kanter and H. Sompolinsky: Phys. Rev. A 35, 380 (1987)ADSCrossRefGoogle Scholar
  30. 9.27
    A.C.C. Coolen, J.J. Denier van der Gon, and Th.W. Ruijgrok: Proc. nEuro88Google Scholar
  31. A.C.C. Coolen, H J.J. Jonker, and Th.W. Ruijgrok, Utrecht preprint (1989)Google Scholar
  32. 9.28
    M. Opper, J. Kleinz, H. Köhler, and W. Kinzel: J. Phys. A 22, L407 (1989)ADSCrossRefGoogle Scholar
  33. 9.29
    T.B. Kepler and L.F. Abbott: J. Phys. (Paris) 49, 1657 (1988)CrossRefGoogle Scholar
  34. 9.30
    H. Homer, D. Bormann, M. Frick, H. Kinzelbach, and A. Schmidt: Z. Phys. B 76,381 (1989)ADSCrossRefGoogle Scholar
  35. 9.31
    I. Kanter: Phys. Rev. A 40, 2611 (1989)ADSCrossRefGoogle Scholar
  36. 9.32
    S. Diedrich and M. Opper: Phys. Rev. Lett. 58, 949 (1987)MathSciNetADSCrossRefGoogle Scholar
  37. E. Gardner, N. Stroud, and DJ. Wallace: J. Phys. A 22, 2019 (1989)MathSciNetADSCrossRefGoogle Scholar
  38. 9.33
    W. Krauth and M. Mézard: J. Phys. A 21, L745 (1987)CrossRefGoogle Scholar
  39. 9.34
    P. Peretto: Neural Networks 1, 309–322 (1988)CrossRefGoogle Scholar
  40. 9.35
    J.F. Fontanari and R. Meir: Caltech preprints (1989)Google Scholar
  41. 9.36
    L.F. Abbott and T.B. Kepler: J. Phys. A 22, L711 (1989)MathSciNetADSCrossRefGoogle Scholar
  42. 9.37
    FJ. Pineda: Phys. Rev. Lett. 59, 2229 (1987)MathSciNetADSCrossRefGoogle Scholar
  43. 9.38
    M. Opper: Europhys. Lett. 8, 389 (1989)ADSCrossRefGoogle Scholar
  44. 9.39
    J.A. Hertz, G.L Thorbergson, and A. Krogh: Physica Scripta T25, 149 (1989)ADSCrossRefGoogle Scholar
  45. W. Kinzel and M. Opper: this volume. Chap.4Google Scholar
  46. 9.41
    For reviews see: J.D. Cowan and D.H. Sharp, Quarterly Reviews of Biophysics, 21 365 (1988)CrossRefGoogle Scholar
  47. R.P. Lippmann, IEEE ASSP Magazine, 4, 4 (1987)CrossRefGoogle Scholar
  48. 9.42
    T. Kohonen: Self Organization and Associative Memory (Springer, Beriin, Heidelberg 1984)Google Scholar
  49. 9.43
    D.E. Rumelhart and J.L. McClelland: Parallel Distributed Processing: Explorations in the Microstructure of Cognition, 2 vols. (MTT Press, Cambridge, Mass. 1986)Google Scholar
  50. 9.44
    M. Minsky and S. Papert:Perceptrons, expanded edition (MIT Press, Cambridge, Mass. 1988)MATHGoogle Scholar
  51. 9.45
    D.E. Rumelhart, G.E. Hinton, and R J. Williams: in Parallel Distributed Processing: Explorations in the Microstructure of Cognition edited by D.E. Rumelhart and J.L. McClelland, (MIT Press, Cambridge, Mass. 1986) Vol. 1, p. 318Google Scholar
  52. 9.46
    Y. Le Cun: Proc. Cognitiva, 85, 593 (1985)Google Scholar
  53. PJ. Werbos, Ph.D. thesis. Harvard University (1974)Google Scholar
  54. 9.47
    D.B. Parker, MIT Technical Report TR-47 (1985)Google Scholar
  55. 9.48
    M. Abeles: Local Cortical Circuits (Springer, Berlin, Heidelberg 1982)CrossRefGoogle Scholar
  56. 9.49
    H. Sompolinsky: Phys. Rev. A 34, 2571 (1986)ADSCrossRefGoogle Scholar
  57. J.L. van Hemmen and R. Kühn, Phys. Rev. Lett. 57, 913 (1986)ADSCrossRefGoogle Scholar
  58. J.L. van Hemmen, Phys. Rev. A 36, 1959 (1987)ADSCrossRefGoogle Scholar
  59. 9.50
    W. Krauth, M. Mézard, and J.P. Nadal: Complex Systems, 2, 387 (1988)MathSciNetMATHGoogle Scholar
  60. 9.51
    W.A. Litüe: Math. Biosci. 19, 101 (1975)Google Scholar
  61. 9.52
    S. Amari and K. Maginu: Neural Networks, 1, 63 (1988)CrossRefGoogle Scholar
  62. See for example W. Feller, An Introduction to Probability Theory and its Applications (Wiley, New York 1966) Vol. 4 p. 256MATHGoogle Scholar
  63. 9.54It is very important to realize that the embedding fields Hi and Hj are not independent. Their correlation is M N. This correlation gives rise to the layer-to-layer recursive variation of the width parameter A which in tum, causes the appearance of non trivial domains of attractionGoogle Scholar
  64. 9.55
    B. Derrida: J. Phys. A 20, L72I (1987)CrossRefGoogle Scholar
  65. 9.56
    W. Kinzel: Z. Physik B 60, 205 (1985)MathSciNetADSGoogle Scholar
  66. 9.57
    W. Krauth, J.P. Nadal, and M. Mézard: J. Phys. A: Math. Gen. 21, 2995 (1988)ADSMATHCrossRefGoogle Scholar
  67. 9.58
    J. Kleinz: diploma thesis, Justus-Liebig University Glessen (1988)Google Scholar
  68. 9.59
    For a recent review see D. H. Hubel, Los Alamos Science 16, 14 (1988)Google Scholar
  69. 9.60
    D. Kämmen and A. Yuille: Biol. Cybem. 59, 23 (1988)CrossRefGoogle Scholar
  70. 9.61
    P. Huben Ann. Statistics 13, 435 (1985)CrossRefGoogle Scholar
  71. 9.62
    E. Oja: J. Math. Biol. 15, 267 (1982)MathSciNetMATHCrossRefGoogle Scholar
  72. 9.63
    R. E. Blahut: Principles and Applications of Information Theory, (Addison-Wesley, 1987)Google Scholar
  73. 9.64
    B. Widrow and R. Winter: Computer 21, 25 (1988)CrossRefGoogle Scholar
  74. 9.65
    F. Rosenblatt: Psych. Rev. 62, 386 (1958)CrossRefGoogle Scholar
  75. F. RosenblattPrinciples of Neurodynamics (Spartan, New York 1962)MATHGoogle Scholar
  76. 9.66
    P.M. Lewis and C.L. Coates: Threshold Logic (Wiley, New York 1967)MATHGoogle Scholar
  77. 9.67
    J. Denker, D. Schwartz, B. Wittner, S. Solla, J.J. Hopfield, R. Howard, and L. Jackel: Complex Systems 1, 877–922 (1987)MathSciNetMATHGoogle Scholar
  78. 9.68
    T. Grossman, Complex Systems 3, 407 (1989)MathSciNetGoogle Scholar
  79. 9.69
    T. Grossman, in Advances in Neural Information Processing Systems 2, edited by D. Touretzky (Morgan Kaufman, San Mateo 1990) p. 516Google Scholar
  80. 9.70
    R. Rohwer, in Advances in Neural Information Processing Systems 2, edited by D. Touretzky (Morgan Kaufman, San Mateo 1990) p. 538)Google Scholar
  81. A. Krogh, G.I. Thorbergsson, and J.A. Hertz, ibid, p. 773Google Scholar
  82. D. Saad and E. Marom, Complex Systems 4, 107 (1990)MathSciNetMATHGoogle Scholar
  83. 9.71
    D. Nabutovsky, T. Grossman, and E. Domany: Complex Systems 4, 519 (1990)MathSciNetMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1995

Authors and Affiliations

  • Eytan Domany
  • Ronny Meir

There are no affiliations available

Personalised recommendations