Advertisement

Advances in Machine Learning Modeling Reviewing Hybrid and Ensemble Methods

  • Sina Ardabili
  • Amir MosaviEmail author
  • Annamária R. Várkonyi-Kóczy
Conference paper
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 101)

Abstract

The conventional machine learning (ML) algorithms are continuously advancing and evolving at a fast-paced by introducing the novel learning algorithms. ML models are continually improving using hybridization and ensemble techniques to empower computation, functionality, robustness, and accuracy aspects of modeling. Currently, numerous hybrid and ensemble ML models have been introduced. However, they have not been surveyed in a comprehensive manner. This paper presents the state of the art of novel ML models and their performance and application domains through a novel taxonomy.

Keywords

Machine learning Deep learning Ensemble models 

Nomenclatures

ANN

Artificial neural network

ELM

Extreme learning machine

ML

Machine learning

SVM

Support vector machine

WNN

Wavelet neural networks

DL

Deep learning

ARIMA

Autoregressive integrated moving average

EE-ANT

Ensemble empirical with adaptive noise technology

DA-KF

Data assimilation Kalman filter-based

OSELM

Online sequential extreme learning machine

BAGNBT

Bagging-based naïve bayes trees

EEMD

Ensemble empirical mode decomposition

GOA

Grasshopper optimization algorithm

HybPAS

Hybrid of linear regression-deep neural network

TSM

Trauma Severity model

GBDT

Gradient boosting decision tree

EBFTM

Evidential belief function and tree-based models

DTFNN

Decision tree overfitting and neural network

ICEEMDMAN

Improved complete ensemble empirical mode decomposition method with adaptive noise

RF

Random forest

Notes

Acknowledgments

This publication has been supported by the Project: “Support of research and development activities of the J. Selye University in the field of Digital Slovakia and creative industry” of the Research & Innovation Operational Programme (ITMS code: NFP313010T504) co-funded by the European Regional Development Fund.

References

  1. 1.
    Choubin, B., et al.: Snow avalanche hazard prediction using machine learning methods. J. Hydrol. 577 (2019)CrossRefGoogle Scholar
  2. 2.
    Dehghani, M., et al.: Prediction of hydropower generation using Grey wolf optimization adaptive neuro-fuzzy inference system. Energies 12(2) (2019)CrossRefGoogle Scholar
  3. 3.
    Mosavi, A., Bathla, Y., Varkonyi-Koczy, A.: Predicting the future using web knowledge: state of the art survey. In: Luca, D., Sirghi, L., Costin, C. (eds.), pp. 341–349. Springer (2018)Google Scholar
  4. 4.
    Mosavi, A., Rabczuk, T.: Learning and intelligent optimization for material design innovation. In: Kvasov, D.E., et al. (eds.), pp. 358–363. Springer (2017)Google Scholar
  5. 5.
    Mosavi, A., Rabczuk, T., Várkonyi-Kóczy, A.R.: Reviewing the novel machine learning tools for materials design. In: Luca, D., Sirghi, L., Costin, C. (eds.), pp. 50–58. Springer (2018)Google Scholar
  6. 6.
    Qasem, S.N., et al.: Estimating daily dew point temperature using machine learning algorithms. Water (Switzerland) 11(3) (2019)CrossRefGoogle Scholar
  7. 7.
    Taherei Ghazvinei, P., et al.: Sugarcane growth prediction based on meteorological parameters using extreme learning machine and artificial neural network. Eng. Appl. Comput. Fluid Mech. 12(1), 738–749 (2018)Google Scholar
  8. 8.
    Torabi, M., et al.: A Hybrid clustering and classification technique for forecasting short-term energy consumption. Environ. Prog. Sustain. Energy 38(1), 66–76 (2019)CrossRefGoogle Scholar
  9. 9.
    Torabi, M., et al.: A hybrid machine learning approach for daily prediction of solar radiation. In: Lecture Notes in Networks and Systems, pp. 266–274. Springer (2019)Google Scholar
  10. 10.
    Dineva, A., et al.: Review of soft computing models in design and control of rotating electrical machines. Energies 12(6) (2019)CrossRefGoogle Scholar
  11. 11.
    Mosavi, A., Ozturk, P., Chau, K.W.: Flood prediction using machine learning models: literature review. Water (Switzerland) 10(11) (2018)CrossRefGoogle Scholar
  12. 12.
    Mosavi, A., et al.: State of the art of machine learning models in energy systems, a systematic review. Energies 12(7) (2019)CrossRefGoogle Scholar
  13. 13.
    Mosavi, A., Várkonyi-Kóczy, A.R.: Integration of machine learning and optimization for robot learning. In: Jablonski, R., Szewczyk, R. (eds.), pp. 349–355. Springer (2017)Google Scholar
  14. 14.
    Cheng, L., Yu, T.: A new generation of AI: A review and perspective on machine learning technologies applied to smart energy and electric power systems. Int. J. Energy Res. 43(6), 1928–1973 (2019)CrossRefGoogle Scholar
  15. 15.
    Cheng, L., et al.: Machine Learning for Energy and Electric Power Systems: State of the Art and Prospects. Dianli Xitong Zidonghua/Autom. Electric Power Syst. 43(1), 15–31 (2019)Google Scholar
  16. 16.
    Chou, J.S., Tran, D.S.: Forecasting energy consumption time series using machine learning techniques based on usage patterns of residential householders. Energy 709–726 (2018)CrossRefGoogle Scholar
  17. 17.
    Chatterjee, B., et al.: RF-PUF: enhancing IoT security through authentication of wireless nodes using in-situ machine learning. IEEE Internet of Things J. 6(1), 388–398 (2019)CrossRefGoogle Scholar
  18. 18.
    Panesar, S.S., et al.: Machine learning versus logistic regression methods for 2-year mortality prognostication in a small, heterogeneous glioma database. World Neurosurg X 2 (2019)CrossRefGoogle Scholar
  19. 19.
    Thomas, P.B.M., et al.: Feasibility of simple machine learning approaches to support detection of non-glaucomatous visual fields in future automated glaucoma clinics. Eye (Basingstoke) 33(7), 1133–1139 (2019)Google Scholar
  20. 20.
    Alhajri, M.I., Ali, N.T., Shubair, R.M.: Classification of indoor environments for IoT applications: a machine learning approach. IEEE Antennas Wirel. Propag. Lett. 17(12), 2164–2168 (2018)CrossRefGoogle Scholar
  21. 21.
    Jamil, A., Bayram, B.: The delineation of tea gardens from high resolution digital orthoimages using mean-shift and supervised machine learning methods. Geocarto Int. (2019)Google Scholar
  22. 22.
    Maxwell, A.E., Warner, T.A., Fang, F.: Implementation of machine-learning classification in remote sensing: an applied review. Int. J. Remote Sens. 39(9), 2784–2817 (2018)CrossRefGoogle Scholar
  23. 23.
    Sehgal, V., et al.: Machine learning creates a simple endoscopic classification system that improves dysplasia detection in barrett’s oesophagus amongst non-expert endoscopists. Gastroenterol. Res. Pract. 2018 (2018)Google Scholar
  24. 24.
    Azeem, M.I., et al.: Machine learning techniques for code smell detection: a systematic literature review and meta-analysis. Inf. Softw. Technol. 108, 115–138 (2019)CrossRefGoogle Scholar
  25. 25.
    Jabeen, A., Ranganathan, S.: Applications of machine learning in GPCR bioactive ligand discovery. Curr. Opin. Struct. Biol. 55, 66–76 (2019)CrossRefGoogle Scholar
  26. 26.
    Xu, C., Jackson, S.A.: Machine learning and complex biological data. Genome Biol. 20(1) (2019)Google Scholar
  27. 27.
    Zhang, Z., Sejdić, E.: Radiological images and machine learning: trends, perspectives, and prospects. Comput. Biol. Med. 108, 354–370 (2019)CrossRefGoogle Scholar
  28. 28.
    Bock, F.E., et al., A review of the application of machine learning and data mining approaches in continuum materials mechanics. Front. Mater. 6 (2019)Google Scholar
  29. 29.
    Ekins, S., et al.: Exploiting machine learning for end-to-end drug discovery and development. Nat. Mater. 18(5), 435–441 (2019)CrossRefGoogle Scholar
  30. 30.
    Woldaregay, A.Z., et al.: Data-driven blood glucose pattern classification and anomalies detection: Machine-learning applications in type 1 diabetes. J. Med. Internet Res. 21(5) (2019)CrossRefGoogle Scholar
  31. 31.
    Najafzadeh, M., Ghaemi, A.: Prediction of the five-day biochemical oxygen demand and chemical oxygen demand in natural streams using machine learning methods. Environ. Monit. Assess. 191(6) (2019)Google Scholar
  32. 32.
    Singh, H., Rana, P.S., Singh, U.: Prediction of drug synergy in cancer using ensemble-based machine learning techniques. Mod. Phys. Lett. B 32(11) (2018)CrossRefGoogle Scholar
  33. 33.
    Choubin, B., et al.: River suspended sediment modelling using the CART model: A comparative study of machine learning techniques. Sci. Total Environ. 615, 272–281 (2018)CrossRefGoogle Scholar
  34. 34.
    Zarkogianni, K., Athanasiou, M., Thanopoulou, A.C.: Comparison of machine learning approaches toward assessing the risk of developing cardiovascular disease as a long-term diabetes complication. IEEE J. Biomed. Health Inf. 22(5), 1637–1647 (2018)CrossRefGoogle Scholar
  35. 35.
    Aram, F., et al.: Design and validation of a computational program for analysing mental maps: aram mental map analyzer. Sustainability (Switzerland) 11(14) (2019)CrossRefGoogle Scholar
  36. 36.
    Asadi, E., et al.: Groundwater quality assessment for drinking and agricultural purposes in Tabriz Aquifer, Iran (2019)Google Scholar
  37. 37.
    Asghar, M.Z., Subhan, F., Imran, M., Kundi, F.M., Shamshirband, S., Mosavi, A., Csiba, P., Várkonyi-Kóczy, A.R.: Performance evaluation of supervised machine learning techniques for efficient detection of emotions from online content (2019), 2019080019.  https://doi.org/10.20944/preprints201908.0019.v1
  38. 38.
    Bemani, A., Baghban, A., Shamshirband, S., Mosavi, A., Csiba, P., Várkonyi-Kóczy, A.R.: Applying ANN, ANFIS, and LSSVM models for estimation of acid solvent solubility in supercritical CO2 (2019), 2019060055.  https://doi.org/10.20944/preprints201906.0055.v2
  39. 39.
    Mosavi, A., et al.: Prediction of multi-inputs bubble column reactor using a novel hybrid model of computational fluid dynamics and machine learning. Eng. Appl. Comput. Fluid Mech. 13(1), 482–492 (2019)Google Scholar
  40. 40.
    Nosratabadi, S., et al.: Sustainable business models: a review. Sustainability (Switzerland) 11(6) (2019)CrossRefGoogle Scholar
  41. 41.
    Rezakazemi, M., Mosavi, A., Shirazian, S.: ANFIS pattern for molecular membranes separation optimization. J. Mol. Liq. 274, 470–476 (2019)CrossRefGoogle Scholar
  42. 42.
    Riahi-Madvar, H., et al.: Comparative analysis of soft computing techniques RBF, MLP, and ANFIS with MLR and MNLR for predicting grade-control scour hole geometry. Eng. Appl. Comput. Fluid Mech. 13(1), 529–550 (2019)Google Scholar
  43. 43.
    Shabani, S., Samadianfard, S., Taghi Sattari, M., Shamshirband, S., Mosavi, A., Kmet, T., Várkonyi-Kóczy, A.R.: Modeling daily pan evaporation in humid climates using gaussian process regression (2019), 2019070351. ( https://doi.org/10.20944/preprints201907.0351.v1)
  44. 44.
    Shamshirband, S., Hadipoor, M., Baghban, A., Mosavi, A., Bukor J., Várkonyi-Kóczy, A.R.: Developing an ANFIS-PSO model to predict mercury emissions in combustion flue gases (2019), 2019070165.  https://doi.org/10.20944/preprints201907.0165.v1
  45. 45.
    Shamshirband, S., et al.: Ensemble models with uncertainty analysis for multi-day ahead forecasting of chlorophyll a concentration in coastal waters. Eng. Appl. Comput. Fluid Mech. 13(1), 91–101 (2019)Google Scholar
  46. 46.
    Shamshirband, S., Mosavi, A., Rabczuk, T.: Particle swarm optimization model to predict scour depth around bridge pier (2019). arXiv:1906.08863
  47. 47.
    Choubin, B., et al.: An ensemble prediction of flood susceptibility using multivariate discriminant analysis, classification and regression trees, and support vector machines. Sci. Total Environ. 651, 2087–2096 (2019)CrossRefGoogle Scholar
  48. 48.
    Dineva, A., et al.: Multi-label classification for fault diagnosis of rotating electrical machines (2019)Google Scholar
  49. 49.
    Farzaneh-Gord, M., et al.: Numerical simulation of pressure pulsation effects of a snubber in a CNG station for increasing measurement accuracy. Eng. Appl. Comput. Fluid Mech. 13(1), 642–663 (2019)Google Scholar
  50. 50.
    Ghalandari, M., et al.: Investigation of submerged structures’ flexibility on sloshing frequency using a boundary element method and finite element analysis. Eng. Appl. Comput. Fluid Mech. 13(1), 519–528 (2019)Google Scholar
  51. 51.
    Ghalandari, M., et al.: Flutter speed estimation using presented differential quadrature method formulation. Eng. Appl. Comput. Fluid Mech. 13(1), 804–810 (2019)Google Scholar
  52. 52.
    Karballaeezadeh, N., et al.: Prediction of remaining service life of pavement using an optimized support vector machine (case study of Semnan-Firuzkuh road). Eng. Appl. Comput. Fluid Mech. 13(1), 188–198 (2019)Google Scholar
  53. 53.
    Menad, N.A., et al.: Modeling temperature dependency of oil-water relative permeability in thermal enhanced oil recovery processes using group method of data handling and gene expression programming. Eng. Appl. Comput. Fluid Mech. 13(1), 724–743 (2019)MathSciNetGoogle Scholar
  54. 54.
    Mohammadzadeh, S., et al.: Prediction of compression index of fine-grained soils using a gene expression programming model. Infrastructures 4(2), 26 (2019)CrossRefGoogle Scholar
  55. 55.
    Mosavi, A., Edalatifar, M.: A hybrid neuro-fuzzy algorithm for prediction of reference evapotranspiration. In: Lecture Notes in Networks and Systems, pp. 235–243. Springer (2019)Google Scholar
  56. 56.
    Mosavi, A., Lopez, A., Várkonyi-Kóczy, A.R.: Industrial applications of big data: state of the art survey. Luca, D., Sirghi, L., Costin, C. (eds.), pp. 225–232. Springer (2018)Google Scholar
  57. 57.
    Bui, D.T., et al.: Shallow landslide prediction using a novel hybrid functional machine learning algorithm. Remote Sens. 11(8) (2019)Google Scholar
  58. 58.
    Pham, B.T., et al.: Landslide susceptibility modeling using Reduced Error Pruning Trees and different ensemble techniques: hybrid machine learning approaches. Catena 175, 203–218 (2019)CrossRefGoogle Scholar
  59. 59.
    Zhang, X., Mahadevan, S.: Ensemble machine learning models for aviation incident risk prediction. Decis. Support Syst. 116, 48–63 (2019)CrossRefGoogle Scholar
  60. 60.
    Jaiswal, A., Malhotra, R.: Software reliability prediction using machine learning techniques. Int. J. Syst. Assur. Eng. Manag. 9(1), 230–244 (2018)CrossRefGoogle Scholar
  61. 61.
    Khagi, B., Kwon, G.R., Lama, R.: Comparative analysis of Alzheimer’s disease classification by CDR level using CNN, feature selection, and machine-learning techniques. Int. J. Imaging Syst. Technol. (2019)Google Scholar
  62. 62.
    Kumari, M., et al.: Comparative analysis of machine learning based QSAR models and molecular docking studies to screen potential anti-tubercular inhibitors against InhA of mycobacterium tuberculosis. Int. J. Comput. Biol. Drug Des. 11(3), 209–235 (2018)CrossRefGoogle Scholar
  63. 63.
    Bataineh, A.A.: A comparative analysis of nonlinear machine learning algorithms for breast cancer detection. Int. J. Mach. Learn. Comput. 9(3), 248–254 (2019)MathSciNetCrossRefGoogle Scholar
  64. 64.
    Manzoor, S.I., Singla, J.: A comparative analysis of machine learning techniques for spam detection. Int. J. Adv. Trends Comput. Sci. Eng. 8(3), 810–814 (2019)CrossRefGoogle Scholar
  65. 65.
    Odugu, K., Rajasekar, B.: Comparative analysis on supervised machine learning models for future wireless communication networks. Int. J. Innov. Technol. Explor. Eng. 8(6), 721–723 (2019)Google Scholar
  66. 66.
    Hou Q, et al.: An adaptive hybrid model for short-term urban traffic flow prediction. Phys. A Stat. Mech. Appl. 527 (2019)CrossRefGoogle Scholar
  67. 67.
    Du, P., et al.: A novel hybrid model for short-term wind power forecasting. Appl. Soft Comput. J. 80, 93–106 (2019)CrossRefGoogle Scholar
  68. 68.
    Zhang, W., He, H., Zhang, S.: A novel multi-stage hybrid model with enhanced multi-population niche genetic algorithm: An application in credit scoring. Expert Syst. Appl. 121, 221–232 (2019)CrossRefGoogle Scholar
  69. 69.
    Pham, B.T., Prakash, I.: A novel hybrid model of Bagging-based Naïve Bayes Trees for landslide susceptibility assessment. Bull. Eng. Geol. Env. 78(3), 1911–1925 (2019)CrossRefGoogle Scholar
  70. 70.
    Wu, J., et al.: A new hybrid model to predict the electrical load in five states of Australia. Energy, 598–609 (2019)Google Scholar
  71. 71.
    Albalawi, F., et al.: Hybrid model for efficient prediction of poly(A) signals in human genomic DNA. Methods (2019)Google Scholar
  72. 72.
    Gorczyca, M.T., Toscano, N.C., Cheng, J.D.: The trauma severity model: an ensemble machine learning approach to risk prediction. Comput. Biol. Med. 108, 9–19 (2019)CrossRefGoogle Scholar
  73. 73.
    Wang, Q.F., Xu, M., Hussain, A.: Large-scale ensemble model for customer churn prediction in search ads. Cogn. Comput. 11(2), 262–270 (2019)CrossRefGoogle Scholar
  74. 74.
    Naghibi, S.A., et al.: Application of rotation forest with decision trees as base classifier and a novel ensemble model in spatial modeling of groundwater potential. Environ. Monit. Assess. 191(4) (2019)Google Scholar
  75. 75.
    Ali, M., Prasad, R.: Significant wave height forecasting via an extreme learning machine model integrated with improved complete ensemble empirical mode decomposition. Renew. Sustain. Energy Rev. 281–295 (2019)CrossRefGoogle Scholar
  76. 76.
    Yamanaka, A., Maeda, Y., Sasaki, K.: Ensemble Kalman filter-based data assimilation for three-dimensional multi-phase-field model: Estimation of anisotropic grain boundary properties. Mater. Des. 165 (2019)CrossRefGoogle Scholar
  77. 77.
    Yadav, D.C., Pal, S.: To generate an ensemble model for women thyroid prediction using data mining techniques. Asian Pac. J. Cancer Prev. 20(4), 1275–1281 (2019)CrossRefGoogle Scholar
  78. 78.
    Ardabili, S., Mosavi, A., Mahmoudi, A., Mesri Gundoshmian, T., Nosratabadi, S., Var-konyi-Koczy, A.: Modelling temperature variation of mushroom growing hall using artificial neural networks (2019)Google Scholar
  79. 79.
    Mesri Gundoshmian, T., Ardabili, S., Mosavi, A., Varkonyi-Koczy, A.: Prediction of combine harvester performance using hybrid machine learning modeling and response surface methodology (2019)Google Scholar
  80. 80.
    Ardabili, S., Mosavi, A., Varkonyi-Koczy, A.: Systematic review of deep learning and machine learning models in biofuels research (2019)Google Scholar
  81. 81.
    Ardabili, S., Mosavi, A., Varkonyi-Koczy, A.: Advances in machine learning model-ing reviewing hybrid and ensemble methods (2019)Google Scholar
  82. 82.
    Ardabili, S., Mosavi, A., Varkonyi-Koczy, A.: Building Energy information: demand and consumption prediction with Machine Learning models for sustainable and smart cities (2019)Google Scholar
  83. 83.
    Ardabili, S., Mosavi, A., Dehghani, M., Varkonyi-Koczy, A.: Deep learning and machine learning in hydrological processes climate change and earth systems a systematic review (2019)Google Scholar
  84. 84.
    Mohammadzadeh D., Karballaeezadeh, N., Mohemmi, M., Mosavi, A., Várkonyi-Kóczy A.: Urban train soil-structure interaction modeling and analysis (2019)Google Scholar
  85. 85.
    Mosavi, A., Ardabili, S., Varkonyi-Koczy, A.: List of deep learning models (2019)Google Scholar
  86. 86.
    Nosratabadi, S., Mosavi, A., Keivani, R., Ardabili, S., Aram, F., State of the art survey of deep learning and machine learning models for smart cities and urban sustainability (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Institute of Advanced Studies KoszegKoszegHungary
  2. 2.Kalman Kando Faculty of Electrical EngineeringObuda UniversityBudapestHungary
  3. 3.School of the Built EnvironmentOxford Brookes UniversityOxfordUK
  4. 4.Department of Mathematics and InformaticsJ. Selye UniversityKomarnoSlovakia

Personalised recommendations