Evaluation Measures for Multi-class Subgroup Discovery

  • Tarek Abudawood
  • Peter Flach
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5781)


Subgroup discovery aims at finding subsets of a population whose class distribution is significantly different from the overall distribution. It has previously predominantly been investigated in a two-class context. This paper investigates multi-class subgroup discovery methods. We consider six evaluation measures for multi-class subgroups, four of them new, and study their theoretical properties. We extend the two-class subgroup discovery algorithm CN2-SD to incorporate the new evaluation measures and a new weighting scheme inspired by AdaBoost. We demonstrate the usefulness of multi-class subgroup discovery experimentally, using discovered subgroups as features for a decision tree learner. Not only is the number of leaves of the decision tree reduced with a factor between 8 and 16 on average, but significant improvements in accuracy and AUC are achieved with particular evaluation measures and settings. Similar performance improvements can be observed when using naive Bayes.


Mutual Information Evaluation Measure Gini Index Target Concept Heuristic Function 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Lavrač, N., Kavšek, B., Flach, P., Todorovski, L.: Subgroup Discovery with CN2-SD. Journal of Machine Learning Research 5, 153–188 (2004)MathSciNetGoogle Scholar
  2. 2.
    Klösgen, W.: Explora: A multipattern and multistrategy discovery assistant. In: Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., Uthurusamy, R. (eds.) Advances in Knowledge Discovery and Data Mining, pp. 249–271. MIT Press, Cambridge (2004)Google Scholar
  3. 3.
    Klösgen, W.: Subgroup discovery. In: Klösgen, W., Zytkow, J.M. (eds.) Handbook of Data Mining and Knowledge Discovery, pp. 354–361. Oxford University Press, Oxford (2002)Google Scholar
  4. 4.
    Fürnkranz, J., Flach, P.: ROC ’n’ rule learning: Towards a better understanding of covering algorithms. Machine Learning 58, 39–77 (2005)CrossRefzbMATHGoogle Scholar
  5. 5.
    Clark, P., Niblett, T.: The CN2 Induction Algorithm. Machine Learning 3, 261–283 (1989)Google Scholar
  6. 6.
    Clark, P., Boswell, R.: Rule induction with CN2: Some recent improvements. In: Kodratoff, Y. (ed.) EWSL 1991. LNCS (LNAI), vol. 482, pp. 151–163. Springer, Heidelberg (1991)CrossRefGoogle Scholar
  7. 7.
    Schapire, R.E.: The boosting approach to machine learning: An overview. In: Nonlinear Estimation and Classification. Lecture Notes in Statistics. Springer, Heidelberg (2003)Google Scholar
  8. 8.
    Lavrač, N., Flach, P., Zupan, B.: Rule evaluation measures: A unifying view. In: Džeroski, S., Flach, P.A. (eds.) ILP 1999. LNCS (LNAI), vol. 1634, pp. 174–185. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  9. 9.
    Friedman, J.H.: Another approach to polychotomous classification. Technical report, Stanford University, Department of Statistics (1996)Google Scholar
  10. 10.
    Kijsirikul, B., Ussivakul, N., Meknavin, S.: Adaptive directed acyclic graphs for multiclass classification. In: Ishizuka, M., Sattar, A. (eds.) PRICAI 2002. LNCS (LNAI), vol. 2417, pp. 158–168. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  11. 11.
    Platt, J.C., Cristianini, N.: Large margin DAGs for multiclass classification. In: Advance in Neural Information Processing Systems, vol. 12. MIT Press, Cambridge (2000)Google Scholar
  12. 12.
    Hsu, C.W., Lin, C.J.: A comparison of methods for multiclass support vector machines. Neural Networks 13, 415–425 (2002)Google Scholar
  13. 13.
    Jin, X., Xu, A., Bie, R., Guo, P.: Machine Learning Techniques and Chi-Square Feature Selection for Cancer Classification Using SAGE Gene Expression Profiles. In: Li, J., Yang, Q., Tan, A.-H. (eds.) BioDM 2006. LNCS (LNBI), vol. 3916, pp. 106–115. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  14. 14.
    Newman, D., Hettich, S., Blake, C., Merz, C.: UCI repository of machine learning databases (1998)Google Scholar
  15. 15.
    Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)zbMATHGoogle Scholar
  16. 16.
    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7, 1–30 (2006)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Battiti, R.: Using mutual information for selecting features in supervised neural net learning. Transactions on Neural Networks 5(4) (1994)Google Scholar
  18. 18.
    Fleuret, F.: Fast binary feature selection with conditional mutual information. Journal of Machine Learning Research 5, 1531–1555 (2004)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Bostrom, H.: Covering vs. divide-and-conquer for top-down induction of logic programs. In: Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1194–1200. Morgan Kaufmann, San Francisco (1995)Google Scholar
  20. 20.
    Friedman, J.H., Fisher, N.I.: Bump hunting in high-dimensional data. Statistics and Computing 9, 123–143 (1999)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Tarek Abudawood
    • 1
  • Peter Flach
    • 1
  1. 1.Department of Computer ScienceUniversity of BristolUnited Kingdom

Personalised recommendations