Addressing Local Class Imbalance in Balanced Datasets with Dynamic Impurity Decision Trees
Decision trees are among the most popular machine learning algorithms, due to their simplicity, versatility, and interpretability. Their underlying principle revolves around the recursive partitioning of the feature space into disjoint subsets, each of which should ideally contain only a single class. This is achieved by selecting features and conditions that allow for the most effective split of the tree structure. Traditionally, impurity metrics are used to measure the effectiveness of a split, as ideally in a given subset only instances from a single class should be present. In this paper, we discuss the underlying shortcoming of such an assumption and introduce the notion of local class imbalance. We show that traditional splitting criteria induce the emergence of increasing class imbalances as the tree structure grows. Therefore, even when dealing with initially balanced datasets, class imbalance will become a problem during decision tree induction. At the same time, we show that existing skew-insensitive split criteria return inferior performance when data is roughly balanced. To address this, we propose a simple, yet effective hybrid decision tree architecture that is capable of dynamically switching between standard and skew-insensitive splitting criterion during decision tree induction. Our experimental study depicts that local class imbalance is embedded in most standard classification problems and that the proposed hybrid approach is capable of alleviating its influence.
KeywordsMachine learning Decision trees Splitting criteria Class imbalance
This work is supported by the VCU College of Engineering Deans Undergraduate Research Initiative (DURI) program.
- 1.Alcalá-Fdez, J., Fernández, A., Luengo, J., Derrac, J., García, S.: KEEL data-mining software tool: data set repository, integration of algorithms and experimental analysis framework. Mult.-Valued Log. Soft Comput. 17(2–3), 255–287 (2011)Google Scholar
- 4.Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and Regression Trees. Wadsworth (1984)Google Scholar
- 5.Cano, A.: A survey on graphic processing unit computing for large-scale data mining. Wiley Interdisc. Rew. Data Min. Knowl. Discov. 8(1) (2018)Google Scholar
- 8.Flach, P.A.: The geometry of roc space: understanding machine learning metrics through roc isometrics. In: Proceedings of the Twentieth International Conference on International Conference on Machine Learning, pp. 194–201. ICML’03, AAAI Press (2003). http://dl.acm.org/citation.cfm?id=3041838.3041863
- 13.Kearns, M.J., Mansour, Y.: On the boosting ability of top-down decision tree learning algorithms. In: STOC, pp. 459–468. ACM (1996)Google Scholar
- 14.Krawczyk, B.: Learning from imbalanced data: open challenges and future directions. Prog. AI 5(4), 221–232 (2016)Google Scholar
- 15.Lango, M., Brzezinski, D., Firlik, S., Stefanowski, J.: Discovering minority sub-clusters and local difficulty factors from imbalanced data. In: Yamamoto, A., Kida, T., Uno, T., Kuboyama, T. (eds.) DS 2017. LNCS (LNAI), vol. 10558, pp. 324–339. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67786-6_23CrossRefGoogle Scholar