Multi-objective evolutionary optimization using the relationship between F1 and accuracy metrics in classification tasks
- 29 Downloads
This work analyses the complementarity and contrast between two metrics commonly used for evaluating the quality of a binary classifier: the correct classification rate or accuracy, C, and the F1 metric, which is very popular when dealing with imbalanced datasets. Based on this analysis, a set of constraints relating C and F1 are defined as a function of the ratio of positive patterns in the dataset. We evaluate the possibility of using a multi-objective evolutionary algorithm guided by this pair of metrics to optimise binary classification models. To check the validity of the constraints, we perform an empirical analysis considering 26 benchmark datasets obtained from the UCI repository and an interesting liver transplant dataset. The results show that the relation is fulfilled and that the use of the algorithm for simultaneously optimising the pair (C,F1) leads to a generally balanced accuracy for both classes. The experiments also reveal that, in some cases, better results are obtained by using the majority class as the positive class instead of using the minority one, which is the most common approach with imbalanced datasets.
KeywordsBinary classification Evaluation metrics F1-metric Multi-objective evolutionary algorithms
This work has been partially subsidised by the TIN2014-54583-C2-1-R, TIN2017-85887-C2-1-P and TIN2017-90567-REDT projects of the Spanish Ministry of Economy and Competitiveness (MINECO), and FE726 DER funds of the European Union.
Compliance with Ethical Standards
Conflict of interests
All authors declare that they have no conflict of interest.
This article does not contain any studies with human participants performed by any of the authors.
- 1.Akbani R, Kwek S, Japkowicz N (2004) Applying support vector machines to imbalanced datasets. In: Proceedings of the 15th European conference on machine learning (ECML2004), pp 39–50Google Scholar
- 3.Asuncion A, Newman D (2007) UCI maching learning repository. http://www.ics.uci.edu/mlearn/MLRepository.html
- 6.Briceño J, Cruz-Ramírez M, Prieto M, Navasa M, Ortiz de Urbina J, Orti R, Gómez-Bravo M, Otero A, Varo E, Tomé S, Clemente G, Bañares R, Bárcena R, Cuervas-Mons V, Solórzano G, Vinaixa C, Rubín A, Colmenero J, Valdivieso A, Ciria R, Hervás-Martínez C, de la Mata M (2014) Use of artificial intelligence as an innovative donor-recipient matching model for liver transplantation: results from a multicenter Spanish study. J Hepatol 61(5):1020–1028CrossRefGoogle Scholar
- 8.Chawla N, Sylvester J (2007) Exploiting diversity in ensembles: improving the performance on unbalanced datasets. In: Multiple classifier systems, lecture notes in computer science, vol 4472, pp 397–406Google Scholar
- 10.Chinta P, Balamurugan P, Shevade S, Murty M (2013) Optimizing F-measure with non-convex loss and sparse linear classifiers. In: The 2013 international joint conference on neural networks (IJCNN), pp 1–8Google Scholar
- 16.Estabrooks A, Japkowicz N (2001) A mixture-of-experts framework for learning from imbalanced data sets. In: Advances in intelligent data analysis, lecture notes in computer science, vol 2189, pp 34–43Google Scholar
- 21.García V, Mollineda R, Sánchez J (2009) Index of balanced accuracy: a performance measure for skewed class distributions. In: Proceedings of the 4th Iberian conference on pattern recognition and image analysis (IbPRIA 2009), lecture notes in computer science, vol 5524, pp 441–448Google Scholar
- 27.Jansche M (2005) Maximum expected f-measure training of logistic regression models. In: Proceedings of the conference on human language technology and empirical methods in natural language processing, pp 692–699Google Scholar
- 29.Joachims T (2005) A support vector method for multivariate performance measures. In: Proceedings of the 22nd international conference on machine learning, pp 377–384Google Scholar
- 30.Joshi M (2002) On evaluating performance of classifiers for rare classes. In: Proceedings 2002 IEEE international conference on data mining, pp 641–644Google Scholar
- 31.Kubat M, Matwin S (1997) Addressing the curse of imbalanced training sets: one-sided selection. In: Proceedings of the fourteenth international conference on machine learning, pp 179– 186Google Scholar
- 33.Liu Z, Tan M, Jiang F (2009) Regularized F-measure maximization for feature selection and classification. J Biomed Biotechnol 2009:617946:8Google Scholar
- 39.Musicant DR, Kumar V, Ozgur A (2003) Optimizing F-measure with support vector machines. In: Proceedings of the international FLAIRS conference, pp 356–360Google Scholar