Global Optimisation of Neural Networks Using a Deterministic Hybrid Approach

  • Gleb Beliakov
  • Ajith Abraham
Part of the Advances in Soft Computing book series (AINSC, volume 14)


Selection of the topology of a neural network and correct parameters for the learning algorithm is a tedious task for designing an optimal artificial neural network, which is smaller, faster and with a better generalization performance. In this paper we introduce a recently developed cutting angle method (a deterministic technique) for global optimization of connection weights. Neural networks are initially trained using the cutting angle method and later the learning is fine-tuned (meta-learning) using conventional gradient descent or other optimization techniques. Experiments were carried out on three time series benchmarks and a comparison was done using evolutionary neural networks. Our preliminary experimentation results show that the proposed deterministic approach could provide near optimal results much faster than the evolutionary approach.


Connection Weight Output Weight Hide Layer Neuron Chaotic Time Series Unit Simplex 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    A. Abraham and B. Nath, ALEC -An Adaptive Learning Framework for Optimizing Artificial Neural Networks, in V. N. Alexandrov, ed., Computational Science, Springer-Verlag, San Francisco, USA, 2001, pp. 171–180.Google Scholar
  2. [2]
    A. Abraham and B. Nath, Optimal Design of Neural Nets Using Hybrid Algorithms, 6th Pacific Rim International Conference on Artificial Intelligence, Springer Verlag, Melbourne, 2000, pp. 510–520.Google Scholar
  3. [3]
    M. Andramonov, A. Rubinov and B. Glover, Cutting angle methods in global optimization, Applied Mathematics Letters, 12 (1999), pp. 95–100.MathSciNetMATHCrossRefGoogle Scholar
  4. [4]
    A. Bagirov and A. Rubinov, Global minimization of increasing positively homogeneous function over the unit simplex, Annals of Operations Research, 98 (2000), pp. 171–187.MathSciNetMATHCrossRefGoogle Scholar
  5. [5]
    L. M. Batten and G. Beliakov, Fast algorithm for the cutting angle method of global optimization, Journal of Global Optimization, accepted (2001).Google Scholar
  6. [6]
    G. Beliakov, Fuzzy clustering of non-convex patterns using global optimisation, FUZZ-IEEE, Melbourne, 2001.Google Scholar
  7. [7]
    G. Beliakov, Least squares splines with free knots: global optimisation approach, IMA Journal of Numerical Analysis, submitted (2001).Google Scholar
  8. [8]
    M. Bianchini, M. Gori and M. Maggini, On the problem of local minima in recurrent neural networks, IEEE Transactions on Neural Networks, 5 (1994), pp. p167(11).Google Scholar
  9. [9]
    C. M. Bishop, Neural Networks for Pattern Recognition, Oxford Press, 1995.Google Scholar
  10. [10]
    G. E. P. Box and G. M. Jenkins, Time Series Analysis, Forecasting and Control, Holden Day, San Francisco, 1970.MATHGoogle Scholar
  11. [11]
    M. Brown and C. Harris, Neurofuzzy adaptive modelling and control, Prentice Hall, London, 1994.Google Scholar
  12. [12]
    P. A. Castillo, J. J. Merelo, A. Prieto, V. Rivas and G. Romero, G-Prop: Global optimization of multilayer perceptrons using GAs, Neurocomputing, 35 (2000), pp. 149–163.MATHGoogle Scholar
  13. [13]
    F. M. Coetzee and V. L. Stonick, On the uniqueness of weights in single-layer perceptrons, IEEE Transactions on Neural Networks, 7 (1996), pp. p318(8).Google Scholar
  14. [14]
    W. Duch and J. Korczak, Optimization and global minimization methods suitable for neural networks, Neural computing surveys (1999).Google Scholar
  15. [15]
    S. Ergezinger and E. Thomson, An accelerated learning algorithm for multilayer perceptron: optimizing layer by layer, IEEE Transactions on Neural Networks, 6 (1995), pp. 31–42.CrossRefGoogle Scholar
  16. [16]
    D. Fogel, Evolutionary Computation: Towards a New Philosophy of Machine Intelligence, IEEE press, 1999.Google Scholar
  17. [17]
    M. Forti, A note on neural networks with multiple equilibrium points, IEEE Transactions on Circuits and Systems-I: Fundamental Theory, 43 (1996), pp. p487(5).Google Scholar
  18. [18]
    H. V. Gupta, K. Hsu and S. Sorooshian, Superior training of artificial neural networks using weight-space partitioning, International conference on neural networks, Houston, USA, 1997, pp. 1919–1923.Google Scholar
  19. [19]
    R. Horst and P. M. Pardalos, Handbook of global optimization, Kluwer Academic Publishers, Dordrecht; Boston, 1995.Google Scholar
  20. [20]
    N. Kasabov, Foundations of Neural Networks, Fuzzy Systems and Knowledge Engineering, The MIT Press, 1996.Google Scholar
  21. [21]
    M. C. Mackey and L. Glass, Oscillation and Chaos in Physiological Control Systems, Science, 197 (1977), pp. 287–289.CrossRefGoogle Scholar
  22. [22]
    O. L. Mangasarian, Mathematical programming in neural networks, ORSA Journal on Computing, 5 (1993), pp. 349–360.MATHCrossRefGoogle Scholar
  23. [23]
    T. Masters, Advanced algorithms for neural networks: a C++ sourcebook, Wiley, New York, 1995.Google Scholar
  24. [24]
    T. Masters, Practical neural network recipes in C++, Academic Press, Boston, 1993.Google Scholar
  25. [25]
    V. V. Phansalkar and M. A. L. Thathachar, Local and Global Optimization Algorithms for Generalized Learning Automata, Neural Computation, 7 (1995), pp. 950–973.CrossRefGoogle Scholar
  26. [26]
    J. Pintér, Global optimization in action: continuous and Lipschitz optimization-algorithms, implementations, and applications, Kluwer Academic Publishers, Dordrecht; Boston, 1996.Google Scholar
  27. [27]
    A. M. Rubinov, Abstract convexity and global optimization, Kluwer Academic Publishers, Dordrecht; Boston, 2000.Google Scholar
  28. [28]
    R. Sexton, R. Dorsey and J. Johnson, Optimization of neural networks: A comparative analysis of the genetic algorithm and simulated annealing, European Journal of Operational Research, 114 (1999), pp. 589–601.MATHCrossRefGoogle Scholar
  29. [29]
    R. Sexton, R. Dorsey and J. Johnson, Toward global optimization of neural networks: A comparison of the genetic algorithm and backpropagation, Decision Support Systems, 22 (1998), pp. 171–185.CrossRefGoogle Scholar
  30. [30]
    R. S. Sexton, R. E. Dorsey and J. D. Johnson, Beyond Backpropagation: Using Simulated Annealing for Training Neural Networks, Journal of End User Computing, 11 (1999), pp. 3.Google Scholar
  31. [31]
    Y. Shang and B. W. Wah, Global optimization for neural network training,Computer, 29 (1996), pp. p45(10).Google Scholar
  32. [32]
    K. K. Shukla and Raghunath, An efficient global algorithm for supervised training of neural networks,Computers and Electrical Engineering, 25 (1999), pp. 195(2).Google Scholar
  33. [33]
    A. Törn and A. Zhilinskas, Global optimization, Springer-Verlag, Berlin; New York, 1989.Google Scholar
  34. [34]
    X. Yao, Evolving Artificial Neural Networks, Proceedings of the IEEE, 87 (1999), pp. 1423–1447.CrossRefGoogle Scholar
  35. [35]
    X. M. Zhang and Y. Q. Chen, Ray-guided global optimization method for training neural networks, Neurocomputing, 30 (2000), pp. 333–337.CrossRefGoogle Scholar
  36. [36]
    X.-S. Zhang, Neural networks in optimization, Kluwer Academic Publishers, Boston, Mass., 2000.MATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Gleb Beliakov
    • 1
  • Ajith Abraham
    • 2
  1. 1.School of Computing and MathematicsDeakin UniversityClayton, MelbourneAustralia
  2. 2.School of Computing and Information TechnologyMonash UniversityChurchillAustralia

Personalised recommendations