Abstract
Whereas benchmarking experiments are very frequently used to investigate the performance of statistical or machine learning algorithms for supervised and unsupervised learning tasks, overall analyses of such experiments are typically only carried out on a heuristic basis, if at all. We suggest to determine winners, and more generally, to derive a consensus ranking of the algorithms, as the linear order on the algorithms which minimizes average symmetric distance (Kemeny-Snell distance) to the performance relations on the individual benchmark data sets. This leads to binary programming problems which can typically be solved reasonably efficiently. We apply the approach to a medium-scale benchmarking experiment to assess the performance of Support Vector Machines in regression and classification problems, and compare the obtained consensus ranking with rankings obtained by simple scoring and Bradley-Terry modeling.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
BARTHÉLEMY, J.-P. and MONJARDET, B. (1981): The Median Procedure in Cluster Analysis and Social Choice Theory. Mathematical Social Sciences, 1, 235–267.
solve. Version 5.5.0.7.
BLAKE, C.L. and MERZ, C.J. (1998): UCI Repository of Machine Learning Databases.
BORDA, J.C. (1781): Mémoire sur les Élections au Scrutin. Histoire de l’Académie Royale des Sciences.
BRADLEY, R.A. and TERRY, M.E. (1952): Rank Analysis of Incomplete Block Designs I: The Method of Paired Comparisons. Biometrika, 39, 324–245.
BRUNK, H.D. (1960): Mathematical Models for Ranking from Paired Comparison. Journal of the American Statistical Association, 55,291, 503–520.
BUTTREY, S.E. (2005): Calling the lp_solve Linear Program Software from R, S-PLUS and Excel. Journal of Statistical Software, 14, 4.
CONDORCET, M.J.A. (1785): Essai sur l’Application de l’Analyse à la Probabilité des dÉcisions Rendues à la Pluralité des Voix. Paris.
DAY, W.H.E. and MCMORRIS, F.R. (2003): Axiomatic Choice Theory in Group Choice and Bioconsensus. SIAM, Philadelphia.
DECANI, J.S. (1969): Maximum Likelihood Paired Comparison Ranking by Linear Programming. Biometrika, 56,3, 537–545.
GRÖTSCHEL, M. and WAKABAYASHI, Y. (1989): A Cutting Plane Algorithm for a Clustering Problem. Mathematical Programming, 45, 59–96.
HOTHORN, T., LEISCH, F., ZEILEIS, A. and HORNIK, K. (2005): The Design and Analysis of Benchmark Experiments. Journal of Computational and Graphical Statistics, 14,3, 675–699.
KEMENY, J.G. and SNELL, J.L. (1962): Mathematical Models in the Social Sciences, Chapter Preference Rankings: An Axiomatic Approach. MIT Press, Cambridge.
MAKHORIN, A. (2006): GNU Linear Programming Kit (GLPK). Version 4.9.
MARCOTORCHINO, F. and MICHAUD, P. (1982): Agregation de Similarites en Classification Automatique. Revue de Statistique Appliquée, XXX, 21–44.
MEYER, D., LEISCH, F. and HORNIK, K. (2003): The Support Vector Machine under Test. Neurocomputing, 55, 169–186.
MONJARDET, B. (1981): Metrics on Partially Ordered Set: A Survey. Discrete Mathematics, 35, 173–184.
NEWMAN, D.J., HETTICH, S., BLAKE, C.L. and MERZ, C.J. (1998): UCI Repository of Machine Learning Databases.
R DEVELOPMENT CORE TEAM (2005): R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0.
RÉGNIER, S. (1965): Sur Quelques Aspects Mathématiques des Problèmes de Classification Automatique. ICC Bulletin, 175–191.
WAKABAYASHI, Y. (1998): The Complexity of Computing Medians of Relations. Resenhas, 3,3, 323–349.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Hornik, K., Meyer, D. (2007). Deriving Consensus Rankings from Benchmarking Experiments. In: Decker, R., Lenz, H.J. (eds) Advances in Data Analysis. Studies in Classification, Data Analysis, and Knowledge Organization. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-70981-7_19
Download citation
DOI: https://doi.org/10.1007/978-3-540-70981-7_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-70980-0
Online ISBN: 978-3-540-70981-7
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)