Abstract
In this paper we investigate the performance of pairwise (or round robin) classification, originally a technique for turning multi-class problems into two-class problems, as a general ensemble technique. In particular, we show that the use of round robin ensembles will also increase the classification performance of decision tree learners, even though they can directly handle multi-class problems. The performance gain is not as large as for bagging and boosting, but on the other hand round robin ensembles have a clearly defined semantics. Furthermore, we show that the advantage of pairwise classification over direct multi-class classification and one-against-all binarization increases with the number of classes, and that round robin ensembles form an interesting alternative for problems with ordered class values.
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Download to read the full chapter text
Chapter PDF
References
E. L. Allwein, R. E. Schapire, and Y. Singer. Reducing multiclass to binary: A unifyinga pproach for margin classifiers. Journal of Machine Learning Research, 1:113–141, 2000.
C. L. Blake and C. J. Merz. UCI repository of machine learning databases. http://www.ics.uci.edu/~mlearn/MLRepository.html, 1998. Department of Information and Computer Science, University of California at Irvine, Irvine CA.
L. Breiman. Bagging predictors. Machine Learning, 24(2):123–140, 1996.
P. Clark and R. Boswell. Rule induction with CN2: Some recent improvements. In Proceedings of the 5th European Working Session on Learning (EWSL-91), pages 151–163, Porto, Portugal, 1991. Springer-Verlag.
W. W. Cohen. Fast effective rule induction. In A. Prieditis and S. Russell, editors, Proceedings of the 12th International Conference on Machine Learning (ML-95), pages 115–123, Lake Tahoe, CA, 1995. Morgan Kaufmann.
W. W. Cohen and Y. Singer. A simple, fast, and effective rule learner. In Proceedings of the 16th National Conference on Artificial Intelligence (AAAI-99), pages 335–342, Menlo Park, CA, 1999. AAAI/MIT Press.
T. G. Dietterich. An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization. Machine Learning, 40(2):139–158, 2000.
T. G. Dietterich and G. Bakiri. Solvingm ulticlass learning problems via error-correcting output codes. Journal of Artificial Intelligence Research, 2:263–286, 1995.
E. Frank and M. Hall. A simple approach to ordinal classification. In L. D. Raedt and P. Flach, editors, Proceedings of the 12th European Conference on Machine Learning (ECML-01), pages 145–156, Freiburg, Germany, 2001. Springer-Verlag.
Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learninga nd an application to boosting. Journal of Computer and System Sciences, 55(1):119–139, 1997.
J. Fürnkranz. Separate-and-conquer rule learning. Artificial Intelligence Review, 13(1):3–54, February 1999.
J. Fürnkranz. Round robin rule learning. In C. E. Brodley and A. P. Danyluk, editors, Proceedings of the 18th International Conference on Machine Learning (ICML-01), pages 146–153, Williamstown, MA, 2001. Morgan Kaufmann Publishers.
J. Fürnkranz. Round robin classification. Journal of Machine Learning Research 2:721–747, 2002.
J. Fürnkranz. Hyperlink ensembles: A case study in hypertext classification. Information Fusion, to appear. Special Issue on Fusion of Multiple Classifiers.
T. Hastie and R. Tibshirani. Classification by pairwise coupling. In M. Jordan, M. Kearns, and S. Solla, editors, Advances in Neural Information Processing Systems 10 (NIPS-97), pages 507–513. MIT Press, 1998.
C.-W. Hsu and C.-J. Lin. A comparison of methods for multi-class support vector machines. IEEE Transactions on Neural Networks, 13(2):415–425, March 2002.
S. Knerr, L. Personnaz, and G. Dreyfus. Handwritten digit recognition by neural networks with single-layer training. IEEE Transactions on Neural Networks, 3(6):962–968, 1992.
A. Krieger, A. J. Wyner, and C. Long. Boosting noisy data. In C. E. Brodley and A. P. Danyluk, editors, Proceedings of the 18th International Conference on Machine Learning (ICML-2001), pages 274–281,Williamstown, MA, 2001. Morgan Kaufmann Publishers.
E. Mayoraz and M. Moreira. On the decomposition of polychotomies into dichotomies. In Proceedings of the 14th International Conference on Machine Learning (ICML-97), pages 219–226, Nashville, TN, 1997. Morgan Kaufmann.
D. Opitz and R. Maclin. Popular ensemble methods: An empirical study. Journal of Artificial Intelligence Research, 11:169–198, 1999.
B. Pfahringer. Winning the KDD99 classification cup: Bagged boosting. SIGKDD explorations, 1(2):65–66, 2000.
J. C. Platt, N. Cristianini, and J. Shawe-Taylor. Large margin DAGs for multiclass classification. In S. A. Solla, T. K. Leen, and K.-R. Müller, editors, Advances in Neural Information Processing Systems 12 (NIPS-99), pages 547–553. MIT Press, 2000.
D. Pyle. Data Preparation for Data Mining. Morgan Kaufmann, San Francisco, CA, 1999.
J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, CA, 1993.
J. R. Quinlan. Bagging, boosting, and C4.5. In Proceedings of the 13th National Conference on Artificial Intelligence (AAAI-96), pages 725–730. AAAI/MIT Press, 1996.
R. E. Schapire. Using output codes to boost multiclass learning problems. In D. H. Fisher, editor, Proceedings fo the 14th International Conference on Machine Learning (ICML-97), pages 313–321, Nachville, TN, 1997. Morgan Kaufmann.
R. E. Schapire and Y. Singer. Improved boosting algorithms using confidence-rated predictions. Machine Learning, 37(3):297–336, 1999.
I. H. Witten and E. Frank. Data Mining—Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann Publishers, 2000.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Fürnkranz, J. (2002). Pairwise Classification as an Ensemble Technique. In: Elomaa, T., Mannila, H., Toivonen, H. (eds) Machine Learning: ECML 2002. ECML 2002. Lecture Notes in Computer Science(), vol 2430. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-36755-1_9
Download citation
DOI: https://doi.org/10.1007/3-540-36755-1_9
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44036-9
Online ISBN: 978-3-540-36755-0
eBook Packages: Springer Book Archive