Cost-sensitive hierarchical classification via multi-scale information entropy for data with an imbalanced distribution

Applied Intelligence (2021)Cite this article

Abstract

Imbalanced distributions present a great problem in machine learning classification tasks. Various algorithms based on cost-sensitive learning have been developed to address the imbalanced distribution problem. However, classes with a hierarchical tree structure create a new challenge for cost-sensitive learning. In this paper, we propose a cost-sensitive hierarchical classification method based on multi-scale information entropy. We construct an information entropy threshold for each level in the tree structure and assign cost-sensitive weights accordingly. First, we use the class hierarchy to divide a large hierarchical classification problem into several smaller sub-classification problems. In this way, a large-scale classification task can be decomposed into multiple, controllable, small-scale classification tasks. Second, we use a logistic regression algorithm to obtain the probabilities of classes at each level. Then, we consider the information entropy at each level as a threshold, which decreases inter-level error propagation in the tree structure. Finally, we design a cost-sensitive model based on the information of each class and use hierarchical information entropy weights as cost-sensitive weights. Information entropy measures the information of the majority and minority classes and allocates them different cost weights to solve imbalanced distribution problems. Experiments on four imbalanced distribution datasets demonstrate that the cost-sensitive hierarchical classification algorithm provides excellent efficiency and effectiveness.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

US$ 39.95

Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Notes

  1. 1.

    Datasets and code used in this research have been uploaded to GitHub. They are accessible at: https://github.com/fhqxa/CSHC.

References

  1. 1.

    Ahmadian S, Khanteymoori A (2015) Training back propagation neural networks using asexual reproduction optimization. In: The 7th conference on information and knowledge technology, pp 1–6

  2. 2.

    Braytee A, Wei L, Kennedy P (2016) A cost-sensitive learning strategy for feature extraction from imbalanced data. In: International conference on neural information processing

  3. 3.

    Cai Z, Zhu W (2018) Multi-label feature selection via feature manifold learning and sparsity regularization. Int J Mach Learn Cybern 9(8):1321–1334

    Article  Google Scholar 

  4. 4.

    Cao P, Zhao D, Zaiane O (2013) An optimized cost-sensitive SVM for imbalanced data learning. In: Pacific-Asia conference on knowledge discovery and data mining

  5. 5.

    Castellanos F, Valero-Mas J, Calvo-Zaragoza J (2018) Oversampling imbalanced data in the string space. Pattern Recognit Lett 103:32–38

    Article  Google Scholar 

  6. 6.

    Chen Y, Hu H, Tang K (2009) Constructing a decision tree from data with hierarchical class labels. Exp Syst Appl 36:4838–4847

    Article  Google Scholar 

  7. 7.

    Dekel O, Keshet J, Singer Y (2004) Large margin hierarchical classification. In: International conference on machine learning

  8. 8.

    Ding C, Dubchak I (2001) Multi-class protein fold recognition using support vector machines and neural networks. Bioinformatics 17(4):349–358

    Article  Google Scholar 

  9. 9.

    Duda R, Hart P, Stork D (2001) Pattern classification. Wiley

  10. 10.

    Fan J, Gao Y, Luo H, Jain R (2008) Mining multilevel image semantics via hierarchical classification. IEEE Trans Multimed 10(2):167–187

    Article  Google Scholar 

  11. 11.

    Fan J, Zhang J, Mei K, Peng J, Gao L (2015) Cost-sensitive learning of hierarchical tree classifiers for large-scale image classification and novel category detection. Pattern Recognit 48(5):1673–1687

    Article  Google Scholar 

  12. 12.

    Fawcett T, Provost F (1997) Adaptive fraud detection. Data Min Knowl Discov 1(3):291–316

    Article  Google Scholar 

  13. 13.

    Feng F, Li K, Shen J (2020) Using cost-sensitive learning and feature selection algorithms to improve the performance of imbalanced classification. IEEE Access 10(99):1–12

    Google Scholar 

  14. 14.

    Ghatasheh N, Faris H, Altaharwa I (2020) Business analytics in telemarketing: cost-sensitive analysis of bank campaigns using artificial neural networks. Appl Ences 10(7):2581–2592

    Google Scholar 

  15. 15.

    Grimaudo L, Mellia M, Baralis E (2012) Hierarchical learning for fine grained internet traffic classification. In: International wireless communications and mobile computing conference

  16. 16.

    Guo S, Zhao H (2020) Hierarchical classification with multi-path selection based on granular computing. Artif Intell Rev (1)1–23

  17. 17.

    Khan S, Hayat M, Bennamoun M, Sohel F, Togneri R (2018) Cost-sensitive learning of deep feature representations from imbalanced data. IEEE Trans Neural Netw Learn Syst 29(8):3573–3587

    Article  Google Scholar 

  18. 18.

    Kira K, Rendell L (1992) A practical approach to feature selection. In: International workshop on machine learning

  19. 19.

    Krause J, Stark M, Deng J (2013) Li, F: 3D object representations for fine-grained categorization. In: International IEEE workshop on 3D representation and recognition

  20. 20.

    Lin W, Tsai C, Hu Y, et al. (2017) Clustering-based undersampling in class-imbalanced data. Inf Sci 17(26):409–419

    Google Scholar 

  21. 21.

    Ling C, Sheng S, Qiang Y (2006) Simple test strategies for cost-sensitive decision trees. IEEE Trans Knowl Data Eng 8(18):1055–1067

    Article  Google Scholar 

  22. 22.

    Liu J, Hu Q, Yu D (2008) A weighted rough set based method developed for class imbalance learning. Inf Sci 178(4):1235– 1256

    MathSciNet  Article  Google Scholar 

  23. 23.

    Liu X, Wu J, Zhou Z (2009) Exploratory undersampling for class-imbalance learning. IEEE Trans Syst Man Cybern Part B 39(2):539–550

    Article  Google Scholar 

  24. 24.

    Lu J, Tan Y (2010) Cost-sensitive subspace learning for human age estimation. In: Proceedings of the international conference on image processing

  25. 25.

    Min F, He H, Qian Y et al (2011) Test-cost-sensitive attribute reduction. Information Sciences An International Journal 181(22):4928–4942

    Article  Google Scholar 

  26. 26.

    Nakano F, Pinto W, Pappa G, Cerri R (2017) Top-down strategies for hierarchical classification of transposable elements with neural networks. In: International joint conference on neural networks

  27. 27.

    Nie F, Huang H, Xiao C, Ding C (2010) Efficient and robust feature selection via joint l2,1-norms minimization. In: International conference on neural information processing systems

  28. 28.

    Peng H, Long F, Ding C (2005) Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell 27(8):1226–1238

    Article  Google Scholar 

  29. 29.

    Qing T, Wu G, Wang F (2005) Posterior probability support vector machines for unbalanced data. IEEE Trans Neural Netw 16(6):1561–1573

    Article  Google Scholar 

  30. 30.

    Sahin Y, Bulkan S, Duman E (2013) A cost-sensitive decision tree approach for fraud detection. Exp Syst Appl 40(15):5916– 5923

    Article  Google Scholar 

  31. 31.

    Sajad A, Ali K (2019) Evolving artificial neural networks using butterfly optimization algorithm for data classification. In: International conference on neural information processing, pp 596–609

  32. 32.

    Sandrine D, Jane F (2002) A prediction-based resampling method for estimating the number of clusters in a dataset. Genome Biol 3(7):1–21

    Google Scholar 

  33. 33.

    Sayed J, Sajad A, Abbas K, et al. (2020) Neuroevolution-based autonomous robot navigation: a comparative study. Cogn Syst Res 62:35–43

    Article  Google Scholar 

  34. 34.

    Sheng S, Ling C, Ni A, Zhang S (2006) Cost-sensitive test strategies. In: Conference on AAAI Press

  35. 35.

    Sun A, Lim E (2001) Hierarchical text classification and evaluation. In: IEEE international conference on data mining

  36. 36.

    Sun Y, Kamel M, Wong A, Wang Y (2007) Cost-sensitive boosting for classification of imbalanced data. Pattern Recognit 40(12):3358–3378

    Article  Google Scholar 

  37. 37.

    Thai-Nghe N, Gantner Z, Schmidt L (2010) Cost-sensitive learning methods for imbalanced data. In: International joint conference on neural networks

  38. 38.

    Tuo Q, Zhao H, Hu Q (2019) Hierarchical feature selection with subtree based graph regularization. Knowl-Based Syst 163:996–1008

    Article  Google Scholar 

  39. 39.

    Wang C, Wang Y, Shao M, Qian Y, Chen D (2009) Fuzzy rough attribute reduction for categorical data. IEEE Trans Fuzzy Syst pp(99):1–12

    Google Scholar 

  40. 40.

    Wang S, Zhu W (2018) Sparse graph embedding unsupervised feature selection. IEEE Trans Syst Man Cybern Syst 48(3):329–341

    Article  Google Scholar 

  41. 41.

    Wang C, Huang Y, Shao M, Hu Q, Chen D (2019) Feature selection based on neighborhood self-information. IEEE Trans Cybern pp(99):1–12

    Google Scholar 

  42. 42.

    Wei L, Liao M, Gao X, Zou Q (2015) An improved protein structural prediction method by incorporating both sequence and structure information. IEEE Trans Nanobiosci 14(4):339– 349

    Article  Google Scholar 

  43. 43.

    Xiao J, Hays J, Ehinger K, Oliva A, Torralba A (2010) Sun database: large-scale scene recognition from abbey to zoo. In: Proceedings of IEEE conference on computer vision and pattern recognition, vol 23, pp 3485–3492

  44. 44.

    Yu X, Liu J, Keung J (2020) Improving ranking-oriented defect prediction using a cost-sensitive ranking SVM. IEEE Trans Reliab 69(1):139–153

    Article  Google Scholar 

  45. 45.

    Yu W, Hu Q, Zhou Y, Hong Z, Qian Y, Liang J (2017) Local bayes risk minimization based stopping strategy for hierarchical classification. In: IEEE international conference on data mining

  46. 46.

    Zadrozny B, Langford J, Abe N (2003) Cost-sensitive learning by cost-proportionate example weighting. In: IEEE international conference on data mining

  47. 47.

    Zhang Y, Zhou Z (2010) Cost-sensitive face recognition. IEEE Trans Pattern Anal Mach Intell 10(32):1758–1769

    Article  Google Scholar 

  48. 48.

    Zhao H, Hu Q, Wang P (2017) Hierarchical feature selection with recursive regularization. In: International joint conference on artificial intelligence, pp 3483–3489

  49. 49.

    Zhao H, Hu Q, Zhu P, et al. (2019) A recursive regularization based feature selection framework for hierarchical classification. IEEE Trans Knowl Data Eng PP(99):10–23

    Google Scholar 

  50. 50.

    Zhou Y, Hu Q, Yu W (2018) Deep super-class learning for long-tail distributed image classification. Pattern Recognit 80:118– 128

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant No. 61703196, the Natural Science Foundation of Fujian Province under Grant No. 2018J01549, and the President’s Fund of Minnan Normal University under Grant No. KJ19021.

Author information

Affiliations

  1. Fujian Key Laboratory of Granular Computing and Application, Minnan Normal University, Zhangzhou, Fujian, 363000, China

    Weijie Zheng & Hong Zhao

  2. School of Computer Science, Minnan Normal University, Zhangzhou, Fujian, 363000, China

    Hong Zhao

Authors
  1. Weijie Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hong Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hong Zhao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zheng, W., Zhao, H. Cost-sensitive hierarchical classification via multi-scale information entropy for data with an imbalanced distribution. Appl Intell (2021). https://doi.org/10.1007/s10489-020-02089-1

Download citation

Keywords

  • Imbalanced distribution
  • Information entropy
  • Cost-sensitive
  • Hierarchical classification