Addressing Local Class Imbalance in Balanced Datasets with Dynamic Impurity Decision Trees

  • Andriy Mulyar
  • Bartosz KrawczykEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11198)


Decision trees are among the most popular machine learning algorithms, due to their simplicity, versatility, and interpretability. Their underlying principle revolves around the recursive partitioning of the feature space into disjoint subsets, each of which should ideally contain only a single class. This is achieved by selecting features and conditions that allow for the most effective split of the tree structure. Traditionally, impurity metrics are used to measure the effectiveness of a split, as ideally in a given subset only instances from a single class should be present. In this paper, we discuss the underlying shortcoming of such an assumption and introduce the notion of local class imbalance. We show that traditional splitting criteria induce the emergence of increasing class imbalances as the tree structure grows. Therefore, even when dealing with initially balanced datasets, class imbalance will become a problem during decision tree induction. At the same time, we show that existing skew-insensitive split criteria return inferior performance when data is roughly balanced. To address this, we propose a simple, yet effective hybrid decision tree architecture that is capable of dynamically switching between standard and skew-insensitive splitting criterion during decision tree induction. Our experimental study depicts that local class imbalance is embedded in most standard classification problems and that the proposed hybrid approach is capable of alleviating its influence.


Machine learning Decision trees Splitting criteria Class imbalance 



This work is supported by the VCU College of Engineering Deans Undergraduate Research Initiative (DURI) program.


  1. 1.
    Alcalá-Fdez, J., Fernández, A., Luengo, J., Derrac, J., García, S.: KEEL data-mining software tool: data set repository, integration of algorithms and experimental analysis framework. Mult.-Valued Log. Soft Comput. 17(2–3), 255–287 (2011)Google Scholar
  2. 2.
    Boonchuay, K., Sinapiromsaran, K., Lursinsap, C.: Decision tree induction based on minority entropy for the class imbalance problem. Pattern Anal. Appl. 20(3), 769–782 (2017)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Breiman, L.: Technical note: some properties of splitting criteria. Mach. Learn. 24(1), 41–47 (1996)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and Regression Trees. Wadsworth (1984)Google Scholar
  5. 5.
    Cano, A.: A survey on graphic processing unit computing for large-scale data mining. Wiley Interdisc. Rew. Data Min. Knowl. Discov. 8(1) (2018)Google Scholar
  6. 6.
    Cieslak, D.A., Chawla, N.V.: Learning decision trees for unbalanced data. In: Daelemans, W., Goethals, B., Morik, K. (eds.) ECML PKDD 2008. LNCS (LNAI), vol. 5211, pp. 241–256. Springer, Heidelberg (2008). Scholar
  7. 7.
    Cieslak, D.A., Hoens, T.R., Chawla, N.V., Kegelmeyer, W.P.: Hellinger distance decision trees are robust and skew-insensitive. Data Min. Knowl. Discov. 24(1), 136–158 (2012)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Flach, P.A.: The geometry of roc space: understanding machine learning metrics through roc isometrics. In: Proceedings of the Twentieth International Conference on International Conference on Machine Learning, pp. 194–201. ICML’03, AAAI Press (2003).
  9. 9.
    García, S., Fernández, A., Luengo, J., Herrera, F.: Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power. Inf. Sci. 180(10), 2044–2064 (2010)CrossRefGoogle Scholar
  10. 10.
    Hapfelmeier, A., Pfahringer, B., Kramer, S.: Pruning incremental linear model trees with approximate lookahead. IEEE Trans. Knowl. Data Eng. 26(8), 2072–2076 (2014)CrossRefGoogle Scholar
  11. 11.
    He, H., Garcia, E.A.: Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 21(9), 1263–1284 (2009). Scholar
  12. 12.
    Jaworski, M., Duda, P., Rutkowski, L.: New splitting criteria for decision trees in stationary data streams. IEEE Trans. Neural Netw. Learn. Syst. 29(6), 2516–2529 (2018)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Kearns, M.J., Mansour, Y.: On the boosting ability of top-down decision tree learning algorithms. In: STOC, pp. 459–468. ACM (1996)Google Scholar
  14. 14.
    Krawczyk, B.: Learning from imbalanced data: open challenges and future directions. Prog. AI 5(4), 221–232 (2016)Google Scholar
  15. 15.
    Lango, M., Brzezinski, D., Firlik, S., Stefanowski, J.: Discovering minority sub-clusters and local difficulty factors from imbalanced data. In: Yamamoto, A., Kida, T., Uno, T., Kuboyama, T. (eds.) DS 2017. LNCS (LNAI), vol. 10558, pp. 324–339. Springer, Cham (2017). Scholar
  16. 16.
    Li, F., Zhang, X., Zhang, X., Du, C., Xu, Y., Tian, Y.: Cost-sensitive and hybrid-attribute measure multi-decision tree over imbalanced data sets. Inf. Sci. 422, 242–256 (2018)CrossRefGoogle Scholar
  17. 17.
    Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Smith, M.R., Martinez, T.R., Giraud-Carrier, C.G.: An instance level analysis of data complexity. Mach. Learn. 95(2), 225–256 (2014)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Weinberg, A.I., Last, M.: Interpretable decision-tree induction in a big data parallel framework. Appl. Math. Comput. Sci. 27(4), 737–748 (2017)MathSciNetzbMATHGoogle Scholar
  20. 20.
    Woźniak, M.: A hybrid decision tree training method using data streams. Knowl. Inf. Syst. 29(2), 335–347 (2011)CrossRefGoogle Scholar
  21. 21.
    Woźniak, M., Graña, M., Corchado, E.: A survey of multiple classifier systems as hybrid systems. Inf. Fusion 16, 3–17 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Department of Computer ScienceVirginia Commonwealth UniversityRichmondUSA

Personalised recommendations