Skip to main content
SpringerLink
Log in

Approaches to Combining Local and Evolutionary Search for Training Neural Networks: A Review and Some New Results

  • Chapter
Advances in Evolutionary Computing

Part of the book series: Natural Computing Series ((NCS))

Abstract

Training of neural networks by local search such as gradient-based algorithms could be difficult. This calls for the development of alternative training algorithms such as evolutionary search. However, training by evolutionary search often requires long computation time. In this chapter, we investigate the possibilities of reducing the time taken by combining the efforts of local search and evolutionary search. There are a number of attempts to combine these search strategies, but not all of them are successful. This chapter provides a critical review of these attempts. Moreover, different approaches to combining evolutionary search and local search are compared. Experimental results indicate that while the Baldwinian and the two-phase approaches are inefficient in improving the evolution process for difficult problems, the Lamarckian approach is able to speed up the training process and to improve the solution quality. In this chapter, the strength and weakness of these approaches are illustrated, and the factors affecting their efficiency and applicability are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ackley, D. H., Littman, M. L. (1992) Interactions between learning and evolution. In C. G. Langton, C. Taylor, J. D. Farmer, and S. Rasmussen, editors, Artificial Life 2, 487–509. Redwood City, CA: Addison-Wesley

    Google Scholar 

  2. Ackley, D. H., Littman, M. L. (1994) A case for Lamarckian evolution. In C. G. Langton, editor, Artificial Life 3, 3–10. Reading, MA: Addison-Wesley

    Google Scholar 

  3. Angeline, P. J., Saunders, G. M., Pollack, J. B. (1994) An evolutionary algorithm that constructs recurrent neural networks. IEEE Transactions on Neural Networks, 5(1):54–65

    Article  Google Scholar 

  4. Back, T., Hammel, U., Schwefel, H.-P. (1997) Evolutionary computation: Comments on the history and current state. IEEE Transactions on Evolutionary Computation, 1(1):3–17

    Article  Google Scholar 

  5. Baldwin, J. M. (1896) A new factor in evolution. American Naturalist, 30:441–451

    Article  Google Scholar 

  6. Belew, R. K., McInerney, J., Schraudolph, N. N. (1992) Evolving networks: Using the genetic algorithm with connectionist learning. In C. G. Langton, C. Taylor, J. D. Farmer, and S. Rasmussen, editors, Artificial Life 2, 511–547. Redwood City, CA: Addison-Wesley

    Google Scholar 

  7. Bengio, Y., Simard, P., Frasconi, P. (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2): 157–166

    Article  Google Scholar 

  8. Braun, H., Zagorski, P. (1994) ENZO-M — a hybrid approach for optimizing neural networks by evolution and learning. In Y. Davidor, H.-P. Schwefel, and R. Manner, editors, Parallel Problem Solving from Nature — PPSN III, 440–451. Berlin: Springer-Verlag

    Google Scholar 

  9. Chalmers, D. J. (1990) The evolution of learning: An experiment in genetic connectionism. In D. S. Touretzky, editor, Proceedings of the 1990 Connectionist Models Summer School, 81–90. San Mateo, CA: Morgan Kaufmann

    Google Scholar 

  10. Collins, R. J., Jefferson, D. R. (1991) Selection in massively parallel genetic algorithms. In Proceedings of the Fourth International Conference on Genetic Algorithms, 249–256

    Google Scholar 

  11. Crosher, D. (1993) The artificial evolution of a generalized class of adaptive processes. In AI’93 Workshop on Evolutionary Computation, 18–36

    Google Scholar 

  12. Dantzig, G. B. (1963) Linear Programming and Extensions. Princeton, NJ: Princeton University Press

    MATH  Google Scholar 

  13. Davidor, Y. (1991) A naturally occurring niche& species phenomenon: The model and first results. In Proceedings of the Fourth International Conference on Genetic Algorithms, 257–262

    Google Scholar 

  14. De Garis, H. (1991) GenNets: Genetically programmed neural networks — using the genetic algorithm to train neural nets whose inputs and/or outputs vary in time. In Proceedings of the IEEE International Joint Conference on Neural Networks, 1391–1396

    Google Scholar 

  15. Elman, J. L. (1988) Finding structure in time. Technical Report CRL 8801, Center for Research in Language, University of California, San Diego

    Google Scholar 

  16. Erkmen, I., Ozdogan, A. (1997) Short term load forecasting using genetically optimized neural network cascaded with a modified kohonen clustering process. In Proceedings of the IEEE International Symposium on Intelligent Control, 107–112

    Google Scholar 

  17. Fogel, D. B. (1994) An introduction to simulated evolutionary optimization. IEEE Transactions on Neural Networks, 5(1):3–14

    Article  Google Scholar 

  18. Fogel, D. B. (1995) Evolutionary computation: toward a new philosophy of machine intelligence. Piscataway, NJ: IEEE Press

    Google Scholar 

  19. Fogel, D. B., Fogel, L. J., Porto, V. W. (1990) Evolving neural networks. Biological Cybernetics, 63:487–493

    Article  Google Scholar 

  20. Fogel, L. J., Owens, A. J., Walsh, M. J. (1966) Artificial Intelligence Through Simulated Evolution. New York: Wiley

    MATH  Google Scholar 

  21. Fontanari, J. F., Meir, R. (1991) Evolving a learning algorithm for the binary perceptron. Network, 2(4):353–359

    Article  Google Scholar 

  22. French, R. M., Messinger, A. (1994) Genes, phenes and the Baldwin effect: Learning and evolution in a simulated population. In A. B. Rodney and M. Pattie, editors, Artificial Life 4, 277–282. Cambridge, MA: MIT Press

    Google Scholar 

  23. Goldberg, D. E. (1989) Genetic Algorithms in Search, Optimization, and Machine Learning. Reading, MA: Addison-Wesley

    MATH  Google Scholar 

  24. Greenwood, G. W. (1997) Training partially recurrent neural networks using evolutionary strategies. IEEE Transactions on Speech Audio Processing, 5(2):192–194

    Article  Google Scholar 

  25. Gruau, F., Whitley, D. (1993) Adding learning to the cellular development of neural networks: Evolution and the Baldwin effect. Evolutionary Computation, 1(3):213–233

    Article  Google Scholar 

  26. Hanes, M. D., Ahalt, S. C., Krishnamurthy, A. K. (1994) Acoustic-to-phonetic mapping using recurrent neural networks. IEEE Transactions on Neural Networks, 5(4):659–662

    Article  Google Scholar 

  27. Harp, S. A., Samad, T., Guha, A. (1989) Towards the genetic synthesis of neural networks. In J. D. Schaffer, editor, Proceedings of the Third International Conference on Genetic Algorithms, 360–369. San Mateo, CA: Morgan Kaufmann

    Google Scholar 

  28. Harvey, I. (1997) Is there another new factor in evolution? Evolutionary Computation, 4(3):313–329

    Article  Google Scholar 

  29. Hebb, D. O. (1949) The Organization of Behavior. New York: Wiley

    Google Scholar 

  30. Hinton, G. E., Nowlan, S. J. (1987) How learning can guide evolution. Complex Systems, 1:495–502

    MATH  Google Scholar 

  31. Hornik, K. (1990) Approximation capabilities of multilayer feedforward neural networks. Neural Networks, 4:251–257

    Article  Google Scholar 

  32. Huang, W. M., Lippmann, R. P. (1988) Neural net and traditional classifiers. In D. Anderson, editor, Neural Information Processing Systems, 387–396. New York: American Institute of Physics

    Google Scholar 

  33. Ichimura, T., Takano, T., Tazaki, E. (1995) Reasoning and learning method for fuzzy rules using neural networks with adaptive structured genetic algorithm. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, 3269–3274

    Google Scholar 

  34. Janson, D. J., Frenzel, J. F. (1992) Application of genetic algorithms to the training of higher order neural networks. Journal of Systems Engineering, 2(4):272–276

    Google Scholar 

  35. Janson, D. J., Frenzel, J. F. (1993) Training product unit neural networks with genetic algorithms. IEEE Expert, 8(5):26–33

    Article  Google Scholar 

  36. Jordan, M. I. (1986) Attractor dynamics and parallelism in a connectionist sequential machine. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, 531–546

    Google Scholar 

  37. Keesing, R., Stork, D. G. (1991) Evolution and learning in neural networks: The number and distribution of learning trial affect the rate of evolution. In R. P. Lippmann, J. E. Moody, and D. S. Touretzky, editors, Advances in Neural Information Processing Systems 3, 804–810. San Mateo, CA: Morgan Kaufmann

    Google Scholar 

  38. Kim, H. B., Jung, S. H., Kim, T. G., Park, K. H. (1996) Fast learning method for back-propagation neural network by evolutionary adaptation of learning rates. Neurocomputating, 11(1):101–106

    Article  MATH  Google Scholar 

  39. Kitano, H. (1990) Empirical studies on the speed of convergence of neural network training using genetic algorithms. In Proceedings of the Eighth National Conference on Artificial Intelligence, 789–795

    Google Scholar 

  40. Kolen, J. F., Pollack, J. B. (1990) Back propagation is sensitive to initial conditions. Complex Systems, 4:269–280

    MATH  Google Scholar 

  41. Korning, P. G. (1995) Training neural networks by means of genetic algorithms working on very long chromosomes. International Journal of Neural Systems, 6(3):299–316

    Article  Google Scholar 

  42. Ku, K. W. C. (1999) On the Combination of Local and Evolutionary Search for Training Recurrent Neural Networks. PhD thesis, The Hong Kong Polytechnic University, Hong Kong

    Google Scholar 

  43. Ku, K. W. C., Mak, M. W. (1997) Exploring the effects of Lamarckian and Baldwinian learning in evolving recurrent neural networks. In Proceedings of the IEEE International Conference on Evolutionary Computation, 617–621

    Google Scholar 

  44. Ku, K. W. C., Mak, M. W. (1998) Empirical analysis of the factors that affect the Baldwin effect. In A. E. Eiben, T. Back, M. Schoenauer, and H.-P. Schwefel, editors, Parallel Problem Solving from Nature — PPSN V, 481–490. Berlin: Springer-Verlag

    Chapter  Google Scholar 

  45. Ku, K. W. C., Mak, M. W., Siu, W. C. (2000) A study of the Lamarckian evolution of recurrent neural networks. IEEE Transactions on Evolutionary Computation, 4(1):31–42

    Article  Google Scholar 

  46. Ku, K. W. C., Mak, M. W., Siu, W. C. (1999) Adding learning to cellular genetic algorithms for training recurrent neural networks. IEEE Transactions on Neural Networks, 10(2):239–252

    Article  Google Scholar 

  47. Lee, S. W. (1996) Off-line recognition of totally unconstrained handwritten numerals using multilayer cluster neural network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(6):648–652

    Article  Google Scholar 

  48. Lippmann, R. P. (1987) An introduction to computing with neural nets. IEEE Acoustics, Speech, and Signal Processing Magazine, 4–22

    Google Scholar 

  49. Maniezzo, V. (1994) Genetic evolution of the topology and weight distribution of neural networks. IEEE Transactions on Neural Networks, 5(1):39–53

    Article  Google Scholar 

  50. Mayley, G. (1997) Landscapes, learning costs, and genetic assimilation. Evolutionary Computation, 4(3):213–234

    Article  Google Scholar 

  51. McDonnell, J. R., Waagen, D. (1994) Evolving recurrent perceptions for time-series modelling. IEEE Transactions on Neural Networks, 5(1):24–38

    Article  Google Scholar 

  52. Menczer, F., Parisi, D. (1992) Evidence of hyperplanes in the genetic learning of neural networks. Biological Cybernetics, 66:283–289

    Article  Google Scholar 

  53. Merelo, J. J., Paton, M., Canas, A., Prieto, A., Moran, F. (1993) Optimization of a competitive learning neural network by genetic algorithms. In Proceedings of the International Workshop on Artificial Neural Networks, 185–192

    Google Scholar 

  54. Michalewicz, Z. (1996) Genetic Algorithms + Data Structures — Evolution Programs. Berlin: Springer-Verlag

    Book  MATH  Google Scholar 

  55. Miller, G. F., Todd, P. M., Hegde, S. U. (1989) Designing neural networks using genetic algorithms. In J. D. Schaffer, editor, Proceedings of the Third International Conference on Genetic Algorithms, 379–384. San Meteo, CA: Morgan Kaufmann

    Google Scholar 

  56. Mitchell, M. (1996) An Introduction to Genetic Algorithms. Cambridge, MA: MIT Press

    Google Scholar 

  57. Montana, D. J., Davis, L. (1989) Training feedforward neural network using genetic algorithms. In Proceedings of the Eleventh International Joint Conference on Artificial Intelligence, 762–767

    Google Scholar 

  58. Mozer, M. C. (1992) Induction of multiscale temporal structure. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems 4, 275–282. San Mateo, CA: Morgan Kaufmann

    Google Scholar 

  59. Nolfi, S., Elman, J. L., Parisi, D. (1994) Learning and evolution in neural networks. Adaptive Behavior, 3:5–28

    Article  Google Scholar 

  60. Omatu, S., Yoshioka, M. (1997) Self-tuning neuro-PID control and applications. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, 1985–1989

    Google Scholar 

  61. Omlin, C. W., Giles, C. L. (1996) Rule revision with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering, 8(1): 183–188

    Article  Google Scholar 

  62. Paredis, J. (1996) Coevolutionary life-time learning. In H.-M. Voigt, W. Ebeling, I. Rechenberg, and H.-P. Schwefel, editors, Parallel Problem Solving from Nature — PPSN IV, 72–80. Berlin: Springer-Verlag

    Chapter  Google Scholar 

  63. Parisi, D., Nolfi, S. (1996) The influence of learning on evolution. In R. K. Belew and M. Mitchell, editors, Adaptive Individuals in Evolving Populations: Models and Algorithms, 419–428. Reading, MA: Addison-Wesley

    Google Scholar 

  64. Pineda, F. J. (1987) Generalization of backpropagation to recurrent neural networks. Physical Review Letters, 59:2229–2232

    Article  MathSciNet  Google Scholar 

  65. Port, R. F. (1990) Representation and recognition of temporal patterns. Connection Science, 2:151–176

    Article  Google Scholar 

  66. Porto, V. W., Fogel, D. B., Fogel, L. J. (1995) Alternative neural networks training methods. IEEE Expert, 10(3):16–22

    Article  Google Scholar 

  67. Rechenberg, I. (1989) Evolution strategy: Nature’s way of optimization. In Optimization: Methods and Applications, Possibilities and Limitations, volume 47 of Lecture Notes in Engineering. Berlin: Springer-Verlag

    Google Scholar 

  68. Riedmiller, M., Braun, H. (1993) A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In Proceedings of the International Conference on Neural Networks, 586–591

    Google Scholar 

  69. Rudolph, G. (1991) Global optimization by means of distributed evolution strategies. In H. P. Schwefel and R. Männer, editors, Parallel Problem Solving from Nature — PPSN I, 209–213. Berlin: Springer-Verlag

    Google Scholar 

  70. Rumelhart, D. E., Hinton, G. E., Williams, R. J. (1986) Learning internal representations by error propagation. In D. E. Rumelhart, J. L. McClelland, and the PDP Research Group, editors, Parallel Distribution Processing: Explorations in the Micro structure of Cognition. Vol. 1: Foundation. Cambridge, MA: MIT Press

    Google Scholar 

  71. Saravanan, N., Fogel, D. B. (1995) Evolving neural control systems. IEEE Expert, 10(3):23–27

    Article  Google Scholar 

  72. Sasaki, T., Tokoro, M. (1998) Adaptation under changing environments with various rates of inheritance of acquired characters: Comparison between Darwinian and Lamarckian evolution. In Proceedings of the Second Asia-Pacific Conference on Simulated Evolution and Learning, 34–41

    Google Scholar 

  73. Schwefel, H.-P. (1995) Evolution and Optimum Seeking. New York: Wiley

    Google Scholar 

  74. Sejnowski, T. J., Rosenberg, C. R. (1987) Parallel networks that learn to pronounce English text. Complex Systems, 1:145–168

    MATH  Google Scholar 

  75. Skinner, A. J., Broughton, J. Q. (1995) Neural networks in computational materials science: Training algorithms. Modelling and Simulation in Materials Science and Engineering, 3:371–389

    Article  Google Scholar 

  76. Solis, F. J., Wets, R. J-B. (1981) Minimization by random search techniques. Mathematics of Operations Research, 6(1):19–30

    Article  MathSciNet  MATH  Google Scholar 

  77. Turney, P. (1996) Myths and legends of the Baldwin effect. In Proceedings of the Workshop on Evolutionary Computing and Machine Learning at the 13th International Conference on Machine Learning, 135–142

    Google Scholar 

  78. Waibel, A. (1989) Modular construction of time-delay neural networks for speech recognition. Neural Computation, 1:39–46

    Article  Google Scholar 

  79. Whitley, D. (1994) A genetic algorithm tutorial. Statistics& Computing, 4(2):65–85

    Google Scholar 

  80. Whitley, D., Gordon, V. S., Mathias, K. (1994) Lamarckian evolution, the Baldwin effect and function optimization. In Y. Davidor, H.-P. Schwefel, and R. Manner, editors, Parallel Problem Solving from Nature — PPSN HI, 6–15. Berlin: Springer-Verlag

    Google Scholar 

  81. Whitley, D., Starkweather, T., Bogart, C. (1990) Genetic algorithms and neural networks: Optimizing connections and connectivity. Parallel Computing, 14:347–361

    Article  Google Scholar 

  82. Wieland, A. (1991) Evolving neural network controllers for unstable systems. In Proceedings of the International Joint Conference on Neural Networks, 667–673

    Google Scholar 

  83. Williams, R. J., Zipser, D. (1989) Experimental analysis of the real-time recurrent learning algorithm. Connection Science, 1:87–111

    Article  Google Scholar 

  84. Williams, R. J., Zipser, D. (1989) A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1:270–280

    Article  Google Scholar 

  85. Wu, K. H., Chen, C. H., Lee, J. D. (1996) Cache-genetic-based modular fuzzy neural networks for robot path planning. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, 3089–3094

    Google Scholar 

  86. Yao, X. (1999) Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423–1447

    Article  Google Scholar 

  87. Yao, X., Liu, Y. (1997) A new evolutionary system for evolving artificial neural networks. IEEE Transactions on Neural Networks, 8(3):694–713

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, City University of Hong Kong, Hong Kong

    Kim W. C. Ku

  2. Center for Multimedia Signal Processing, Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hong Kong

    M. W. Mak & W. C. Siu

Authors
  1. Kim W. C. Ku
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. W. Mak
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. W. C. Siu
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Machine Intelligence Unit, Indian Statistical Institute, 203 B.T. Road, Kolkata, 700 035, India

    Ashish Ghosh

  2. Department of Management Information, Hannan University, 5-4-33 Amamhigigashi, Matsubara, Osaka, 580-8502, Japan

    Shigeyoshi Tsutsui

Rights and permissions

Reprints and permissions

Copyright information

© 2003 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Ku, K.W.C., Mak, M.W., Siu, W.C. (2003). Approaches to Combining Local and Evolutionary Search for Training Neural Networks: A Review and Some New Results. In: Ghosh, A., Tsutsui, S. (eds) Advances in Evolutionary Computing. Natural Computing Series. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-18965-4_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-18965-4_24

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-62386-8

  • Online ISBN: 978-3-642-18965-4

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation