Skip to main content
SpringerLink
Log in

Foundations of Ensemble Learning

  • Chapter
  • First Online:
Recent Advances in Ensembles for Feature Selection

Part of the book series: Intelligent Systems Reference Library ((ISRL,volume 147))

Abstract

This chapter describes the basic ideas under the ensemble approach, together with the classical methods that have being used in the field of Machine Learning. Section 3.1 states the rationale under the approach, while in Sect. 3.2 the most popular methods are briefly described. Finally, Sect. 3.3 summarizes and discusses the contents of this chapter.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://scikit-learn.org/stable/index.html.

  2. 2.

    https://es.mathworks.com/help/stats/classification-ensembles.html?requestedDomain=true.

  3. 3.

    https://machinelearningmastery.com/use-ensemble-machine-learning-algorithms-weka/.

References

  1. Rokach, L.: Ensemble-based classifiers. Artif. Intell. Rev. 33(1), 1–39 (2010)

    Article  MathSciNet  Google Scholar 

  2. Polikar, R.: Ensemble based systems in decision making. IEEE Circuits Syst. Mag. 6(3), 21–45 (2006)

    Article  Google Scholar 

  3. Brown, G.: Ensemble learning. In: Sammut, C., Webb, G.I. (eds.) Encyclopedia of Machine Learning. Springer, Berlin (2010)

    Google Scholar 

  4. Caruana, R., Niculescu-Mizil, A.: An empirical comparison of supervised learning algorithms. In: Machine Learning: Proceedings of the 23rd International Conference, pp. 161–168. ACM (2006)

    Google Scholar 

  5. Tukey, J.W.: Exploratory Data Analysis. Addison-Wesley, Reading (1977)

    Google Scholar 

  6. Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, New York (2004)

    Google Scholar 

  7. Freund, Y.: Boosting a weak learning algorithm by majority. Inf. Comput. 121(82), 256–285 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  8. Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Machine Learning: Proceedings of the Thirteenth International Conference, pp. 325–332 (1996)

    Google Scholar 

  9. Dietterich, T.G., Bakiri, G.: Solving multiclass learning problems via error-correcting output codes. J. Artif. Intell. Res. 2, 263–286 (1995)

    Article  MATH  Google Scholar 

  10. Mardani, A., Jusoh, A., Nor, K., Khalifah, Z., Zakwan, N., Valipour, A.: Multiple criteria decision-making techniques and their applications -a review of the literature from 2000 to 2014. Econ. Res. Ekon. Istraivanja 28(1), 516–571 (2015)

    Article  Google Scholar 

  11. Saaty, T.: What is the analytic hierarchy process? In: Mitra, G., Greenberg, H., Lootsma, F., Rijkaert, M., Zimmermann, H. (eds.) Mathematical Models for Decision Support, vol. 48, pp. 109–121. Springer, Berlin (1988)

    Google Scholar 

  12. Ilangkumaran, M., Karthikeyan, M., Ramachandran, T., Boopathiraja, M., Kirubakaran, B.: Risk analysis and warning rate of hot environment for foundry industry using hybrid MCDM technique. Saf. Sci. 72, 133–143 (2015)

    Article  Google Scholar 

  13. Hashemkhani Zolfani, S., Esfahani, M.H., Bitarafan, M., Zavadskas, E.K., Arefi, S.L.: Developing a new hybrid MCDM method for selection of the optimal alternative of mechanical longitudinal ventilation of tunnel pollutants during automobile accidents. Transport 28, 89–96 (2013)

    Article  Google Scholar 

  14. Hu, S.-K., Lu, M.-T., Tzeng, G.-H.: Exploring smart phone improvements based on a hybrid MCDM model. Expert Syst. Appl. 41, 4401–4413 (2014)

    Article  Google Scholar 

  15. Brown, G., Wyatt, J., Harris, R., Yao, X.: Diversity creation methods: a survey and categorization. Inf. Fusion 6(1), 5–20 (2005)

    Article  Google Scholar 

  16. Bramer, M.: Principles of Data Mining. Springer, Berlin (2013)

    Book  MATH  Google Scholar 

  17. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Berlin (2006)

    Google Scholar 

  18. Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)

    MATH  MathSciNet  Google Scholar 

  19. Schapire, R.E.: The strength of weak learnability. Mach. Learn. 5(2), 197–227 (1990)

    Google Scholar 

  20. Vega-Pons, S., Ruiz-Shulcloper, J.: A survey of clustering ensemble algorithms. Int. J. Pattern Recognit. Artif. Intell. 25(3), 337–372 (2011)

    Article  MathSciNet  Google Scholar 

  21. Hu, J., Li, T., Luo, C., Fujita, H., Yang, Y.: Incremental fuzzy cluster ensemble learning based on rough set theory. Knowl. Based Syst. 132, 144–155 (2017). https://doi.org/10.1016/j.knosys.2017.06.020

    Article  Google Scholar 

  22. Dimitriadou, E., Weingessel, A., Hornik, K.: A cluster ensembles framework. Design and Application of Hybrid Intelligent Systems. IOS Press, Amsterdam (2003)

    Google Scholar 

  23. Strehl, A., Ghosh, J.: Cluster ensembles—a knowledge reuse framework for combining multiple partitions. J. Mach. Learn. Res. 3, 583–617 (2002)

    MathSciNet  MATH  Google Scholar 

  24. Hore, P., Hall, L.O., Goldgof, D.B.: A scalable framework for cluster ensembles. Pattern Recognit. 42(5), 676–688 (2009)

    Article  MATH  Google Scholar 

  25. Hore, P., Hall, L., Goldgof, D.: A cluster ensemble framework for large data sets. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics, SMC’06, vol. 4, pp. 3342–3347 (2006)

    Google Scholar 

  26. Pérez-Gállego, P., Quevedo, J.R., del Coz, J.J.: Using ensembles for problems with characterizable changes in data distribution: a case study on quantification. Inf. Fusion 34, 87–100 (2017)

    Article  Google Scholar 

  27. Windeatt, T., Duangsoithong, R., Smith, R.: Embedded feature ranking for ensemble MLP classifiers. IEEE Trans. Neural Netw. 2286, 988–994 (2011)

    Article  Google Scholar 

  28. Attik, M.: Using ensemble feature selection approach in selecting subset with relevant features. In: Advances in Neural Networks-ISNN, pp. 1359–1366. Springer (2006)

    Chapter  Google Scholar 

  29. Saeys, Y., Abeel, T., Van de Peer, Y.: Robust feature selection using ensemble feature selection techniques. Machine Learning and Knowledge Discovery in Databases. Springer, Berlin (2008)

    Chapter  Google Scholar 

  30. Bolón-Canedo, V., Sánchez-Maroño, N., Alonso-Betanzos, A.: An ensemble of filters and classifiers for microarray data classification. Pattern Recognit. 45(1), 531–539 (2012)

    Article  Google Scholar 

  31. Yang, F., Mao, K.Z.: Robust feature selection for microarray data based on multicriterion fusion. IEEE/ACM Trans. Comput. Biol. Bioinform. (TCBB) 8(4), 1080–1092 (2011)

    Article  Google Scholar 

  32. Seijo-Pardo, B., Porto-Díaz, I., Bolón-Canedo, V., Alonso-Betanzos, A.: Ensemble feature selection: homogeneous and heterogeneous approaches. Knowl. Based Syst. https://doi.org/10.1016/j.knosys.2016.11.017

  33. Kuncheva, L., Whitaker, C.: Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach. Learn. 51, 181–207 (2003)

    Article  MATH  Google Scholar 

  34. Zhihua, Z.: Ensemble Methods: Foundations and Algorithms. Chapman and Hall/CRC. CRC Press, Boca Raton (2012)

    Google Scholar 

  35. Fernández-Delgado, M., Cernadas, E., Barro, S., Amorim, D.: Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res. 15, 3133–3181 (2014)

    MathSciNet  MATH  Google Scholar 

  36. Wainberg, M., Alipanahi, B., Frey, B.J.: Are random forests truly the best classifiers? J. Mach. Learn. Res. 17, 1–5 (2016)

    MathSciNet  Google Scholar 

  37. Schapire, R.E., Freund, Y.: Boosting. Foundations and Algorithms. The MIT Press, Cambridge (2012)

    Google Scholar 

  38. Schapire, R.E.: The strength of weak learnability. Mach. Learn. 5, 197–227 (1990)

    Google Scholar 

  39. Wu, X., Kumar, V., Quinlan, J.R., Ghosh, J., Yang, Q., Motoda, H., et al.: Top 10 algorithms in data mining. Knowl. Inf. Syst. 14(1), 1–37 (2008)

    Article  Google Scholar 

  40. Quinlan, J.R.: Bagging, Boosting and C4.5. In: Proceedings of the 13th National Conference on Artificial Intelligence, pp. 725–730 (1996)

    Google Scholar 

  41. Merler, S., Caprile, B., Furlanello, C.: Parallelizing AdaBoost by weights dynamics. Comput. Stat. Data Anal. 51(5), 2487–2498 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  42. Zhang, C.X., Zhang, J.S.: A local boosting algorithm for solving classification problems. Comput. Stat. Data Anal. 52(4), 1928–1941 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  43. Buhlmann, P., Yu, B.: Boosting With the L2 Loss. J. Am. Stat. Assoc. 98(462), 324–339 (2003)

    Article  MATH  Google Scholar 

  44. Dettling, M., Bühlmann, P.: Boosting for tumor classification with gene expression data. Bioinformatics 19(9), 1061–1069 (2003)

    Article  Google Scholar 

  45. Buhlmann, P.: Boosting for high-dimensional linear models. Ann. Stat. 34(2), 559–583 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  46. Landesa-Vzquez, I., Alba-Castro, J.L.: Double-base asymmetric AdaBoost. Neurocomputing 118, 101–114 (2013)

    Article  Google Scholar 

  47. Ting, K. M.: A comparative study of cost-sensitive boosting algorithms. In: ICML, pp. 983–990 (2000)

    Google Scholar 

  48. Viola, P., Jones, M.: Fast and robust classification using asymmetric AdaBoost and a detector cascade. In: NIPS (2002)

    Google Scholar 

  49. Nikolaou, N., Edakunni, N., Kull, M., Flatch, P., Brown, G.: Cost-sensitive boosting algorithms: do we really need them? Mach. Learn. 104(2–3), 359–384 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  50. Breiman, M.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)

    MATH  MathSciNet  Google Scholar 

  51. Ho, T.K.: Random decision forests. In: Proceedings of 3rd International Conference on Document Analysis and Recognition, vol. 1, pp. 278–282 (1995)

    Google Scholar 

  52. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)

    Article  MATH  Google Scholar 

  53. Flach, P.: Machine Learning. The Art and Science of Algorithms that Make Sense of Data. Cambridge University Press, Cambridge (2012)

    Book  MATH  Google Scholar 

  54. Bolón-Canedo, V., Sánchez-Maroño, N., Alonso-Betanzos, A.: Data classification using an ensemble of filters. Neurocomputing 135, 13–20 (2014)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Facultad de Informática, Universidade da Coruña, A Coruña, Spain

    Verónica Bolón-Canedo & Amparo Alonso-Betanzos

Authors
  1. Verónica Bolón-Canedo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amparo Alonso-Betanzos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Verónica Bolón-Canedo .

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Bolón-Canedo, V., Alonso-Betanzos, A. (2018). Foundations of Ensemble Learning. In: Recent Advances in Ensembles for Feature Selection. Intelligent Systems Reference Library, vol 147. Springer, Cham. https://doi.org/10.1007/978-3-319-90080-3_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-90080-3_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-90079-7

  • Online ISBN: 978-3-319-90080-3

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation