Advertisement

A Comparative Study of Harmony Search Algorithms for Improving the Training Process of Extreme Learning Machines

  • Daniel Pusil
  • Carlos Cobos
  • Martha Mendoza
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10632)

Abstract

Extreme Learning Machine (ELM) is a model for the training of single-hidden feedforward networks and is used extensively in problems of classification and regression for its good results and reduced training time compared to backpropagation. Previous work has shown that to define randomly the input weights and biases of the hidden layer reduces quality (accuracy) of ELM. This paper makes a comparative study of three variants of harmony search (the original Harmony Search, Global-best Harmony Search, and New Global Harmony Search), a memetic algorithm from the state of the art denominated M-ELM and a random walk algorithm named RW-ELM on 20 classical classification datasets available in the UCI repository. The results show that the best algorithm for training ELMs is Harmony Search and that the other two variants of this algorithm are better than M-ELM and RW-ELM when cross-validation is used. The experiments were performed at first using separate archives for training and testing, then using cross-validation with 5 folders. Based on literature, authors recommend the use of cross-validation technique because more realistic accuracy results can be obtained.

Keywords

Extreme learning machine Harmony search Global-best harmony search Hill climbing Memetic algorithm Simulated annealing 

References

  1. 1.
    Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986)CrossRefGoogle Scholar
  2. 2.
    Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme learning machine: theory and applications. Neurocomputing 70, 489–501 (2006)CrossRefGoogle Scholar
  3. 3.
    van Heeswijk, M.: Advances in Extreme Learning Machines (2015). http://urn.fi/URN:ISBN:978-952-60-6149-8
  4. 4.
    Zhu, Q.Y., Qin, A.K., Suganthan, P.N., Huang, G.: Bin: evolutionary extreme learning machine. Pattern Recognit. 38, 1759–1763 (2005)CrossRefGoogle Scholar
  5. 5.
    Han, F., Yao, H.F., Ling, Q.H.: An improved evolutionary extreme learning machine based on particle swarm optimization. Neurocomputing 116, 87–93 (2013)CrossRefGoogle Scholar
  6. 6.
    Saraswathi, S., Sundaram, S., Sundararajan, N., Zimmermann, M., Nilsen-Hamilton, M.: ICGA-PSO-ELM approach for accurate multiclass cancer classification resulting in reduced gene sets in which genes encoding secreted proteins are highly represented. IEEE/ACM Trans. Comput. Biol. Bioinform. 8, 452–463 (2011)CrossRefGoogle Scholar
  7. 7.
    Silva, D.N.G., Pacifico, L.D.S., Ludermir, T.B.: An evolutionary extreme learning machine based on group search optimization. In: 2011 IEEE Congress of Evolutionary Computation (CEC), pp. 574–580 (2011)Google Scholar
  8. 8.
    Cao, J., Lin, Z., Huang, G.: Bin: self-adaptive evolutionary extreme learning machine. Neural Process. Lett. 36, 285–305 (2012)CrossRefGoogle Scholar
  9. 9.
    Valença, I., Valença, M.: Optimizing the extreme learning machine using harmony search for hydrologic time series forecasting. Intell. Data Eng. Autom. Learn. IDEAL 2012, 261–269 (2012)Google Scholar
  10. 10.
    Kong, H.: Evolving extreme learning machine paradigm with adaptive operator selection and parameter control. Int. J. Uncertain. Fuzziness Knowl. Base Syst. 21, 143–154 (2013)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Learning, A.E., Elm, M.: Evolutionary extreme learning machine based on particle swarm optimization and clustering strategies. In: Proceedings of the 2013 International Joint Conference Neural Networks, vol. 560, pp. 305–310 (2013)Google Scholar
  12. 12.
    Bazi, Y., Alajlan, N., Melgani, F., AlHichri, H., Malek, S., Yager, R.R.: Differential evolution extreme learning machine for the classification of hyperspectral images. IEEE Geosci. Remote Sens. Lett. 11, 1066–1070 (2014)CrossRefGoogle Scholar
  13. 13.
    Dash, R., Dash, P.K., Bisoi, R.: A self adaptive differential harmony search based optimized extreme learning machine for financial time series prediction. Swarm Evol. Comput. 19, 25–42 (2014)CrossRefGoogle Scholar
  14. 14.
    Liu, N., Wang, H., Member, S.: Ensemble based extreme learning machine. IEEE Sig. Process. Lett. 17, 754–757 (2010)CrossRefGoogle Scholar
  15. 15.
    Cao, J., Lin, Z., Huang, G., Liu, N.: Voting based extreme learning machine. Inf. Sci. (Ny) 185, 66–77 (2012)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Zhang, Y., Wu, J., Cai, Z., Zhang, P., Chen, L.: Memetic extreme learning machine. Pattern Recogn. 58, 135–148 (2016)CrossRefGoogle Scholar
  17. 17.
    Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1, 67–82 (1997)CrossRefGoogle Scholar
  18. 18.
    Wong, W.K., Guo, Z.X.: A hybrid intelligent model for medium-term sales forecasting in fashion retail supply chains using extreme learning machine and harmony search algorithm. Int. J. Prod. Econ. 128, 614–624 (2010)CrossRefGoogle Scholar
  19. 19.
    Huang, G., Zhu, Q., Siew, C.: Extreme learning machine: a new learning scheme of feedforward neural networks. Neurocomputing 70, 489–501 (2006)CrossRefGoogle Scholar
  20. 20.
    Huang, G.B., Chen, L., Siew, C.K.: Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans. Neural Netw. 17, 879–892 (2006)CrossRefGoogle Scholar
  21. 21.
    Geem, Z.W., Kim, J.H., Loganathan, G.V.: A new heuristic optimization algorithm: harmony search. Simulation 76, 60–68 (2001)CrossRefGoogle Scholar
  22. 22.
    Mahdavia, M., Fesangharyb, M., Damangirb, E.: An improved harmony search algorithm for solving optimization problems. Appl. Math. Comput. 188, 1567–1579 (2007)MathSciNetGoogle Scholar
  23. 23.
    Lee, K.S., Geem, Z.W.: A new meta-heuristic algorithm for continuous engineering optimization: harmony search theory and practice. Comput. Methods Appl. Mech. Eng. 194, 3902–3933 (2005)CrossRefGoogle Scholar
  24. 24.
    Manjarres, D., et al.: A survey on applications of the harmony search algorithm. Eng. Appl. Artif. Intell. 26, 1818–1831 (2013)CrossRefGoogle Scholar
  25. 25.
    Cobos, C., Pérez, J., Estupiñan, D.: A survey of harmony search. Av. en Sist. e Inform. 8, 67–80 (2011)zbMATHGoogle Scholar
  26. 26.
    Omran, M.G.H., Mahdavi, M.: Global-best harmony search. Appl. Math. Comput. 198, 643–656 (2008)MathSciNetzbMATHGoogle Scholar
  27. 27.
    Zou, D., Gao, L., Wu, J., Li, S., Li, Y.: A novel global harmony search algorithm for reliability problems. Comput. Ind. Eng. 58, 307–316 (2010)CrossRefGoogle Scholar
  28. 28.
    Cobos, C., Estupiñán, D., Pérez, J.: GHS + LEM: global-best harmony search using learnable evolution models. Appl. Math. Comput. 218, 2558–2578 (2011)MathSciNetzbMATHGoogle Scholar
  29. 29.
    Eberhart, R., Kennedy, J.: A new optimizer using particle swarm theory. In: Proceedings of the Sixth International Symposium Micro Machine and Human Science, pp. 39–43 (1995)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Information Technology Research Group (GTI)Universidad del CaucaPopayánColombia

Personalised recommendations