Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Searching Parsimonious Solutions with GA-PARSIMONY and XGBoost in High-Dimensional Databases

  • Conference paper
  • First Online:
International Joint Conference SOCO’16-CISIS’16-ICEUTE’16 (SOCO 2016, CISIS 2016, ICEUTE 2016)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 527))

Abstract

EXtreme Gradient Boosting (XGBoost) has become one of the most successful techniques in machine learning competitions. It is computationally efficient and scalable, it supports a wide variety of objective functions and it includes different mechanisms to avoid over-fitting and improve accuracy. Having so many tuning parameters, soft computing (SC) is an alternative to search precise and robust models against classical hyper-tuning methods. In this context, we present a preliminary study in which a SC methodology, named GA-PARSIMONY, is used to find accurate and parsimonious XGBoost solutions. The methodology was designed to optimize the search of parsimonious models by feature selection, parameter tuning and model selection. In this work, different experiments are conducted with four complexity metrics in six high dimensional datasets. Although XGBoost performs well with high-dimensional databases, preliminary results indicated that GA-PARSIMONY with feature selection slightly improved the testing error. Therefore, the choice of solutions with fewer inputs, between those with similar cross-validation errors, can help to obtain more robust solutions with better generalization capabilities.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  1. Ahila, R., Sadasivam, V., Manimala, K.: An integrated PSO for parameter determination and feature selection of ELM and its application in classification of power system disturbances. Appl. Soft Comput. 32, 23–37 (2015)

    Article  Google Scholar 

  2. Antonanzas-Torres, F., Urraca, R., Antonanzas, J., Fernandez-Ceniceros, J., de Pison, F.M.: Generation of daily global solar irradiation with support vector machines for regression. Energy Convers. Manage. 96, 277–286 (2015)

    Article  Google Scholar 

  3. Caamaño, P., Bellas, F., Becerra, J.A., Duro, R.J.: Evolutionary algorithm characterization in real parameter optimization problems. Appl. Soft Comput. 13(4), 1902–1921 (2013)

    Article  Google Scholar 

  4. Chen, N., Ribeiro, B., Vieira, A., Duarte, J., Neves, J.C.: A genetic algorithm-based approach to cost-sensitive bankruptcy prediction. Expert Syst. Appl. 38(10), 12939–12945 (2011)

    Article  Google Scholar 

  5. Chen, T., He, T., Benesty, M.: xgboost: Extreme Gradient Boosting (2015). https://github.com/dmlc/xgboost, rpackageversion 0.4-3

  6. Corchado, E., Wozniak, M., Abraham, A., de Carvalho, A.C.P.L.F., Snásel, V.: Recent trends in intelligent data analysis. Neurocomputing 126, 1–2 (2014)

    Article  Google Scholar 

  7. Dhiman, R., Saini, J., Priyanka: Genetic algorithms tuned expert model for detection of epileptic seizures from EEG signatures. Appl. Soft Comput. 19, 8–17 (2014)

    Google Scholar 

  8. Ding, S.: Spectral and wavelet-based feature selection with particle swarm optimization for hyperspectral classification. J. Softw. 6(7), 1248–1256 (2011)

    Article  Google Scholar 

  9. Fernandez-Ceniceros, J., Sanz-Garcia, A., Antonanzas-Torres, F., de Pison, F.M.: A numerical-informational approach for characterising the ductile behaviour of the t-stub component. part 2: parsimonious soft-computing-based metamodel. Eng. Struct. 82, 249–260 (2015)

    Article  Google Scholar 

  10. Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat. 29(5), 1189–1232 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  11. Huang, H.L., Chang, F.L.: ESVM: evolutionary support vector machine for automatic feature selection and classification of microarray data. Biosystems 90(2), 516–528 (2007)

    Article  Google Scholar 

  12. Kaggle: The home of data science. https://www.kaggle.com/

  13. KDD-CUP: Annual data mining and knowledge discovery competition organized by ACM. http://www.kdd.org/kdd-cup

  14. Michalewicz, Z., Janikow, C.Z.: Handling constraints in genetic algorithms. In: ICGA, pp. 151–157 (1991)

    Google Scholar 

  15. Oduguwa, V., Tiwari, A., Roy, R.: Evolutionary computing in manufacturing industry: an overview of recent applications. Appl. Soft Comput. 5(3), 281–299 (2005)

    Article  Google Scholar 

  16. Core Team, R.: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2013)

    Google Scholar 

  17. Reif, M., Shafait, F., Dengel, A.: Meta-learning for evolutionary parameter optimization of classifiers. Mach. Learn. 87(3), 357–380 (2012)

    Article  MathSciNet  Google Scholar 

  18. Sanz-Garcia, A., Fernandez-Ceniceros, J., Antonanzas-Torres, F., Pernia-Espinoza, A., Martinez-de Pison, F.J.: GA-PARSIMONY: a GA-SVR approach with feature selection and parameter optimization to obtain parsimonious solutions for predicting temperature settings in a continuous annealing furnace. Appl. Soft Comput. 35, 13–28 (2015)

    Article  Google Scholar 

  19. Sanz-Garcia, A., Fernández-Ceniceros, J., Fernández-Martínez, R., Martínez-de-Pisón, F.J.: Methodology based on genetic optimisation to develop overall parsimony models for predicting temperature settings on annealing furnace. Ironmaking Steelmaking 41(2), 87–98 (2014)

    Article  Google Scholar 

  20. Sanz-García, A., Fernández-Ceniceros, J., Antoñanzas-Torres, F., Martínez-de Pisón, F.J.: Parsimonious support vector machines modelling for set points in industrial processes based on genetic algorithm optimization. In: Herrero, Á., et al. (eds.) International Joint Conference SOCO13-CISIS13-ICEUTE13. Advances in Intelligent Systems and Computing, vol. 239, pp. 1–10. Springer International Publishing, Heidelberg (2014)

    Google Scholar 

  21. Seni, G., Elder, J.: Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions. Morgan and Claypool Publishers, Chicago (2010)

    Book  Google Scholar 

  22. Shaffer, J.P.: Modified sequentially rejective multiple test procedures. J. Am. Stat. Assoc. 81(395), 826–831 (1986)

    Article  MATH  Google Scholar 

  23. Urraca, R., Sanz-Garcia, A., Fernandez-Ceniceros, J., Sodupe-Ortega, E., Martinez-de-Pison, F.J.: Improving hotel room demand forecasting with a hybrid GA-SVR methodology based on skewed data transformation, feature selection and parsimony tuning. In: Onieva, E., Santos, I., Osaba, E., Quintián, H., Corchado, E. (eds.) HAIS 2015. LNCS (LNAI), vol. 9121, pp. 632–643. Springer, Heidelberg (2015). doi:10.1007/978-3-319-19644-2_52

    Chapter  Google Scholar 

  24. Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bull. 1(6), 80–83 (1945). http://dx.doi.org/10.2307/3001968

    Article  Google Scholar 

  25. Winkler, S.M., Affenzeller, M., Kronberger, G., Kommenda, M., Wagner, S., Jacak, W., Stekel, H.: Analysis of selected evolutionary algorithms in feature selection and parameter optimization for data based tumor marker modeling. In: Moreno-Díaz, R., Pichler, F., Quesada-Arencibia, A. (eds.) EUROCAST 2011. LNCS, vol. 6927, pp. 335–342. Springer, Heidelberg (2012). doi:10.1007/978-3-642-27549-4_43

    Chapter  Google Scholar 

  26. Xue, B., Zhang, M., Browne, W.N.: Particle swarm optimisation for feature selection in classification: novel initialisation and updating mechanisms. Appl. Soft Comput. 18, 261–276 (2014)

    Article  Google Scholar 

  27. Ye, J.: On measuring and correcting the effects of data mining and model selection. J. Am. Stat. Assoc. 93(441), 120–131 (1998)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

The authors would like to acknowledge the fellowship APPI15/05 granted by the Banco Santander and the University of La Rioja.

Author information

Authors and Affiliations

  1. EDMANS Group, University of La Rioja, Logroño, Spain

    Francisco Javier Martinez-de-Pison, Esteban Fraile-Garcia, Javier Ferreiro-Cabello, Rubén Gonzalez & Alpha Pernia

Authors
  1. Francisco Javier Martinez-de-Pison
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Esteban Fraile-Garcia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Javier Ferreiro-Cabello
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rubén Gonzalez
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alpha Pernia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Francisco Javier Martinez-de-Pison .

Editor information

Editors and Affiliations

  1. Computational Intelligence Group, University of the Basque Country (UPV/EHU), San Sebastian, Spain

    Manuel Graña

  2. University of the Basque Country, Vitoria, Spain

    José Manuel López-Guede

  3. Computational Intelligence Group, University of the Basque Country (UPV/EHU), San Sebastian, Spain

    Oier Etxaniz

  4. Department of Civil Engineering, University of Burgos, Burgos, Spain

    Álvaro Herrero

  5. University of Salamanca, Salamanca, Spain

    Héctor Quintián

  6. University of Salamanca, Salamanca, Spain

    Emilio Corchado

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Martinez-de-Pison, F.J., Fraile-Garcia, E., Ferreiro-Cabello, J., Gonzalez, R., Pernia, A. (2017). Searching Parsimonious Solutions with GA-PARSIMONY and XGBoost in High-Dimensional Databases. In: Graña, M., López-Guede, J.M., Etxaniz, O., Herrero, Á., Quintián, H., Corchado, E. (eds) International Joint Conference SOCO’16-CISIS’16-ICEUTE’16. SOCO CISIS ICEUTE 2016 2016 2016. Advances in Intelligent Systems and Computing, vol 527. Springer, Cham. https://doi.org/10.1007/978-3-319-47364-2_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-47364-2_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-47363-5

  • Online ISBN: 978-3-319-47364-2

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation