Skip to main content
SpringerLink
Log in

Local Data Characteristics in Learning Classifiers from Imbalanced Data

  • Chapter
  • First Online:
Advances in Data Analysis with Computational Intelligence Methods

Part of the book series: Studies in Computational Intelligence ((SCI,volume 738))

Abstract

Learning classifiers from imbalanced data is still one of challenging tasks in machine learning and data mining. Data difficulty factors referring to internal and local characteristics of class distributions deteriorate performance of standard classifiers. Many of these factors may be approximated by analyzing the neighbourhood of the learning examples and identifying different types of examples from the minority class. In this paper, we follow recent research on developing such methods for assessing the types of examples which exploit either k-nearest neighbours or kernels. We discuss the approaches to tune the size of both kinds of neighborhoods depending on the data set characteristics and evaluate their usefulness in series of experiments with real-world and synthetic data sets. Furthermore, we claim that the proper analysis of these neighborhoods could be the basis for developing new specialized algorithms for imbalanced data. To illustrate it, we study generalizations of over-sampling in pre-processing methods and neighbourhood based ensembles.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.ics.uci.edu/mlearn/MLRepository.html.

  2. 2.

    We are grateful to Prof. W. Michalowski and the MET Research Group from the University of Ottawa for providing us an access to scrotal-pain data set.

References

  1. Aggarwal, C.C. (Ed.): Data Classification: Algorithms and Applications. Chapman & Hall/CRC (2015)

    Google Scholar 

  2. Batista, G., Prati, R., Monard, M.: A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD Explor. Newslett. 6(1), 20–29 (2004)

    Article  Google Scholar 

  3. Blagus, R., Lusa, L.: Class prediction for high-dimensional class-imbalanced data. BMC Bioinf. 11, 523 (2010)

    Article  Google Scholar 

  4. Błaszczyński, J., Stefanowski, J., Idkowiak, L.: Extending bagging for imbalanced data. In: Proceedings of the 8th CORES 2013. Springer Series on Advances in Intelligent Systems and Computing, vol. 226, pp. 269–278 (2013)

    Google Scholar 

  5. Błaszczyński, J., Stefanowski, J.: Neighbourhood sampling in bagging for imbalanced data. Neurocomputing 150 A, 184–203 (2015)

    Google Scholar 

  6. Błaszczyński, J., Lango, M.: Diversity analysis on imbalanced data using neighbourhood and roughly balanced bagging ensembles. In: Proceedings of ICAISC 2016. Lecture Notes in Computer Science, vol. 9692, pp. 552–562 (2016)

    Google Scholar 

  7. Branco, P., Torgo, L., Ribeiro, R.: A survey of predictive modeling under imbalanced distributions. ACM Comput. Surv. (CSUR) 49(2), 31:1–31:50 (2016)

    Google Scholar 

  8. Chawla, N.: Data mining for imbalanced datasets: an overview. In: Maimon O., Rokach L. (eds.) The Data Mining and Knowledge Discovery Handbook, pp. 853–867. Springer (2005)

    Google Scholar 

  9. Chawla, N., Bowyer, K., Hall, L., Kegelmeyer, W.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 341–378 (2002)

    MATH  Google Scholar 

  10. Cost, S., Salzberg, S.: A weighted nearest neighbor algorithm for learning with symbolic features. Mach. Learn. J. 10(1), 1213–1228 (1993)

    Google Scholar 

  11. Dasarathy, B.V.: NN concepts and techniques: an introductory survey. In: Nearest Neighbor Norms, NN Pattern Classification Techniques, pp. 1–30. IEEE Press (1991)

    Google Scholar 

  12. Enas, G., Chai, S.: Choice of the smoothing parameter and efficiency of the k-nearest neighbour classification. Comput. Math. Appl. 12, 308–317 (1986)

    Article  Google Scholar 

  13. Fernandez, A., Lopez, V., Galar, M., Jesus, M., Herrera, F.: Analysis the classification of imbalanced data sets with multiple classes, binarization techniques and ad-hoc approaches. Knowl. Based Syst. 42, 97–110 (2013)

    Article  Google Scholar 

  14. Galar, M., Fernandez, A., Barrenechea, E., Bustince, H., Herrera, F.: A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev. 99, 1–22 (2011)

    Google Scholar 

  15. Garcia, V., Sanchez, J.S., Mollineda, R.A.: An empirical study of the behaviour of classifiers on imbalanced and overlapped data sets. In: Proceedings of Progress in Pattern Recognition, Image Analysis and Applications 2007. LNCS, vol. 4756, pp. 397–406. Springer (2007)

    Google Scholar 

  16. Gatnar, E.: Multimodel Approach to Discrimination and Regression Issues. PWN Warszawa (2008) (in Polish)

    Google Scholar 

  17. Goldstein, M.: \(K_n\)-nearest neighbour classification. IEEE Trans. Inf. Theory 627–630 (1972)

    Google Scholar 

  18. Guyon, I., Saffari, A., Dror, G., Cawley, G.: Model selection: beyond the Bayesian/frequentist divide. J. Mach. Learn. Res. 11, 61–87 (2010)

    MathSciNet  MATH  Google Scholar 

  19. Han, H., Wang, W., Mao, B.: Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning. In: Proceedings of ICIC. LNCS, vol. 3644, pp. 878–887. Springer (2005)

    Google Scholar 

  20. He, H., Yungian, M. (eds): Imbalanced Learning. Foundations, Algorithms and Applications. IEEE, Wiley (2013)

    Google Scholar 

  21. He, H., Garcia, E.: Learning from imbalanced data. IEEE Trans. Data Knowl. Eng. 21(9), 1263–1284 (2009)

    Article  Google Scholar 

  22. Hido S., Kashima H.: Roughly balanced bagging for imbalance data. In: Proceedings of the SIAM International Conference on Data Mining, pp. 143–152 (2008). An Extended Version in Statistical Analysis and Data Mining, vol. 2, no. 5–6, pp. 412–426 (2009)

    Google Scholar 

  23. Hoens, T., Chawla, N.: Generating diverse ensembles to counter the problem of class imbalance. Proc. PAKDD 2010, 488–499 (2010)

    Google Scholar 

  24. Japkowicz, N.: Concept-learning in the presence of between-class and within-class imbalances. In: Proceedings of Canadian Conference on AI, vol. 2001, pp. 67–77 (2001)

    Google Scholar 

  25. Japkowicz, N., Stephen, S.: Class imbalance problem: a systematic study. Intell. Data Anal. J. 6(5), 429–450 (2002)

    MATH  Google Scholar 

  26. Jo, T., Japkowicz, N.: Class imbalances versus small disjuncts. ACM SIGKDD Explor. Newslett. 6(1), 40–49 (2004)

    Article  Google Scholar 

  27. Jones, M.C., Marron, J.S., Sheather, S.J.: A brief survey of bandwidth selection for density estimation. J. Am. Stat. Assoc. 91(433), 401–407 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  28. Krawczyk, B.: Learning from imbalanced data: open challenges and future directions. Prog. Artif. Intell. 5(4), 221–232 (2016)

    Google Scholar 

  29. Kubat, M., Matwin, S.: Addressing the curse of imbalanced training sets: one-side selection. In: Proceedings of the 14th International Conference on Machine Learning ICML-97, pp. 179–186 (1997)

    Google Scholar 

  30. Lango, M., Stefanowski, J.: The usefulness of roughly balanced bagging for complex and high-dimensional imbalanced data. In: Proceedings of International ECML PKDD Workshop on New Frontiers in Mining Complex Patterns NFmCP 2015. LNAI 9607, pp. 93–107. Springer (2015)

    Google Scholar 

  31. Laurikkala, J.: Improving identification of difficult small classes by balancing class distribution. Tech. Report A-2001-2, University of Tampere (2001)

    Google Scholar 

  32. Lumijarvi, J., Laurikkala, J., Juhola, M.: A comparison of different heterogeneous proximity functions and Euclidean distance. Stud. Health Technol. Inform. 107(Part 2), 1362–1366 (2004)

    Google Scholar 

  33. Ledl, T.: Kernel density estimation: theory and application in discriminant analysis. Austrian J. Stat. 33(3), 267–279 (2004)

    Article  Google Scholar 

  34. Liu, B., Yang, Y., Webb, GT., Boughton, J.: A comparative study of bandwidth choice in kernel density estimation for Naive Bayesian classiffication. In: Proceedings of the 13th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, PAKDD ’09. LNCS, vol. 5476, pp. 302–313. Springer (2009)

    Google Scholar 

  35. Lin, W., Chen, J.: Class-imbalanced classifiers for high-dimensional data. Brief. Bioinform. 14(1), 13–26 (2013)

    Article  Google Scholar 

  36. Lopez, V., Fernandez, A., Garcia, S., Palade, V., Herrera, F.: An insight into classification with imbalanced data: empirical results and current trends on using data intrinsic characteristics. Inf. Sci. 257, 113–141 (2014)

    Article  Google Scholar 

  37. Maciejewski, T., Stefanowski, J.: Local neighbourhood extension of SMOTE for mining imbalanced data. In: Proceedings of IEEE Symposium on Computational Intelligence and Data Mining, pp. 104–111 (2011)

    Google Scholar 

  38. Napierala, K.: Improving rule classifiers for imbalanced data. Ph.D. Thesis. Poznan University of Technology (2013)

    Google Scholar 

  39. Napierala, K., Stefanowski, J., Wilk, Sz.: Learning from imbalanced data in presence of noisy and borderline examples. In: Proceedings of 7th International Conference on RSCTC 2010. LNAI, vol. 6086, pp. 158–167. Springer (2010)

    Google Scholar 

  40. Napierala, K., Stefanowski, J.: The influence of minority class distribution on learning from imbalance data. In: Proceedings of 7th Conference on HAIS 2012. LNAI, vol. 7209, pp. 139–150. Springer (2012)

    Google Scholar 

  41. Napierala, K., Stefanowski, J.: BRACID: a comprehensive approach to learning rules from imbalanced data. J. Intell. Inf. Syst. 39(2), 335–373 (2012)

    Article  Google Scholar 

  42. Napierala, K., Stefanowski, J.: Types of minority class examples and their influence on learning classifiers from imbalanced data. J. Intell. Inf. Syst. 46(3), 563–597 (2016)

    Article  Google Scholar 

  43. Napierala, K., Stefanowski, J., Trzcielinska, M.: Local characteristics of minority examples in pre-processing of imbalanced data. In: Andreasen, T., et al. (eds.) Proceedings of ISMIS 2014. LNAI, vol. 8502, pp. 123–132. Springer (2014)

    Google Scholar 

  44. Nickerson, A., Japkowicz, N., Milios, E.: Using unsupervised learning to guide re-sampling in imbalanced data sets. In: Proceedings of the 8th International Workshop on Artificial Intelligence and Statistics, pp. 261–265 (2001)

    Google Scholar 

  45. Niemann, U., Spiliopoulou, M., Volzke, H., Kuhn, J.P.: Subpopulation discovery in epidemiological data with subspace clustering. Found. Comput. Decis. Sci. 39(4), 271–300 (2014)

    MathSciNet  MATH  Google Scholar 

  46. Prati, R., Batista, G., Monard, M.: Class imbalance versus class overlapping: an analysis of a learning system behavior. In: Proceedings of 3rd Mexican International Conference on Artificial Intelligence, pp. 312–321 (2004)

    Google Scholar 

  47. Quinlan, R.: C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, San Mateo, CA (1993)

    Google Scholar 

  48. Saez, J., Luengo, J., Stefanowski, J., Herrera, F.: Addressing the noisy and borderline examples problem in classification with imbalanced datasets via a class noise filtering method-based re-sampling technique. Inf. Sci. 291, 184–203 (2015)

    Article  Google Scholar 

  49. Seaz, J., Krawczyk, B., Wozniak, M.: Analyzing the oversampling of different classes and types in multi-class imbalanced data. Pattern Recogn. 57, 164–178 (2016). doi:10.1016/j.atcog.2016.03.012

  50. Sheather, S.J.: Density estimation. Stat. Sci. 19(4), 588–597 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  51. Silverman, B.W.: Density Estimation for Statistics and Data Analysis. Chapman and Hall/CRC (1986)

    Google Scholar 

  52. Stefanowski, J.: Overlapping, rare examples and class decomposition in learning classifiers from imbalanced data. In: Ramanna, S., Jain, L.C., Howlett, R.J. (eds.) Emerging Paradigms in Machine Learning, pp. 277–306 (2013)

    Google Scholar 

  53. Stefanowski, J.: Dealing with data difficulty factors while learning from imbalanced data. In: Mielniczuk, J., Matwin, S. (eds.) Challenges in Computational Statistics and Data Mining, pp. 333–363. Springer (2016)

    Google Scholar 

  54. Stefanowski, J.: On properties of under-sampling bagging and its extensions for imbalanced data. In: Proceedings of the 9th International Conference on Computer Recognition Systems CORES 2015, pp. 407–417. Springer (2016)

    Google Scholar 

  55. Stefanowski, J., Wilk, Sz.: Selective pre-processing of imbalanced data for improving classification performance. In: Proceedings of the 10th International Conference on DaWaK 2008. LNCS, vol. 5182, pp. 283–292. Springer (2008)

    Google Scholar 

  56. Sun, Y., Wong, A., Kamel, M.: Classification of imbalanced data: a review. Int. J. Pattern Recogn. Artif. Intell. 23(4), 687–719 (2009)

    Article  Google Scholar 

  57. Tomasev, N., Mladenic, D.: Class imbalance and the curse of minority hubs. Knowl.-Based Syst. 53, 157–172 (2013)

    Article  Google Scholar 

  58. Wang, S., Yao, T.: Diversity analysis on imbalanced data sets by using ensemble models. In: Proceedings of IEEE Symposium on Computational Intelligence and Data Mining, pp. 324–331 (2009)

    Google Scholar 

  59. Wang, S., Yao, X.: Mutliclass imbalance problems: analysis and potential solutions. IEEE Trans. Syst. Man Cybern. Part B 42(4), 1119–1130 (2012)

    Article  Google Scholar 

  60. Weiss, G.M.: Mining with rarity: a unifying framework. ACM SIGKDD Explor. Newslett. 6(1), 7–19 (2004)

    Article  Google Scholar 

  61. Weiss, G.M., Provost, F.: Learning when training data are costly: the effect of class distribution on tree induction. J. Artif. Intell. Res. 19, 315–354 (2003)

    MATH  Google Scholar 

  62. Wilk, S., Stefanowski, J., Wojciechowski, S., Farion, K.J, Michalowski, W.: Application of preprocessing methods to imbalanced clinical data: an experimental study. In: Pietka E. (ed.) Information Technologies in Medicine, pp. 503–515. Springer (2016)

    Google Scholar 

  63. Wilson, D.R., Martinez, T.R.: Improved heterogeneous distance functions. J. Artif. Intell. Res. 6, 1–34 (1997)

    MathSciNet  MATH  Google Scholar 

  64. Wojciechowski, S., Wilk, Sz.: Difficulty Factors and Preprocessing in Imbalanced Data Sets: An Experimental Study on Artificial Data. Found. Comput. Decis. Sci. 42(2), 149–176 (2017)

    Google Scholar 

  65. Zliobaite, I., Pechenizkiy, M., Gama, J.: An overview of concept drift applications. In: Japkowicz, N., Stefanowski, J. (eds.) Big Data Analysis: New Algorithms for a New Society. Springer Studies in Big Data Series, pp. 91–11 (2016)

    Google Scholar 

Download references

Acknowledgements

The research was funded by the the Polish National Science Center, grant no. DEC-2013/11/B/ST6/00963. Close co-operation with Krystyna Napierala in research on modeling types of examples and with Mateusz Lango in research on ensemble models is also acknowledged.

Author information

Authors and Affiliations

  1. Institute of Computing Science, Poznań University of Technology, Piotrowo 2, 60-965, Poznań, Poland

    Jerzy Błaszczyński & Jerzy Stefanowski

Authors
  1. Jerzy Błaszczyński
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jerzy Stefanowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jerzy Stefanowski .

Editor information

Editors and Affiliations

  1. Department of Medicine, Division of Nephrology and Hypertension, University of Louisville, Louisville, Kentucky, USA

    Adam E Gawęda

  2. Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland

    Janusz Kacprzyk

  3. Institute of Computational Intelligence, Department of Mechanical Engineering and Computer Science, Częstochowa University of Technology, Częstochowa, Poland

    Leszek Rutkowski

  4. School of Electrical and Computer Engineering, Oklahoma State University, Stillwater, Oklahoma, USA

    Gary G. Yen

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this chapter

Cite this chapter

Błaszczyński, J., Stefanowski, J. (2018). Local Data Characteristics in Learning Classifiers from Imbalanced Data. In: Gawęda, A., Kacprzyk, J., Rutkowski, L., Yen, G. (eds) Advances in Data Analysis with Computational Intelligence Methods. Studies in Computational Intelligence, vol 738. Springer, Cham. https://doi.org/10.1007/978-3-319-67946-4_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-67946-4_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-67945-7

  • Online ISBN: 978-3-319-67946-4

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation