A Robust Ranking Methodology Based on Diverse Calibration of AdaBoost

  • Róbert Busa-Fekete
  • Balázs Kégl
  • Tamás Éltető
  • György Szarvas
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6911)


In subset ranking, the goal is to learn a ranking function that approximates a gold standard partial ordering of a set of objects (in our case, relevance labels of a set of documents retrieved for the same query). In this paper we introduce a learning to rank approach to subset ranking based on multi-class classification. Our technique can be summarized in three major steps. First, a multi-class classification model (AdaBoost.MH) is trained to predict the relevance label of each object. Second, the trained model is calibrated using various calibration techniques to obtain diverse class probability estimates. Finally, the Bayes-scoring function (which optimizes the popular Information Retrieval performance measure NDCG), is approximated through mixing these estimates into an ultimate scoring function. An important novelty of our approach is that many different methods are applied to estimate the same probability distribution, and all these hypotheses are combined into an improved model. It is well known that mixing different conditional distributions according to a prior is usually more efficient than selecting one “optimal” distribution. Accordingly, using all the calibration techniques, our approach does not require the estimation of the best suited calibration method and is therefore less prone to overfitting. In an experimental study, our method outperformed many standard ranking algorithms on the LETOR benchmark datasets, most of which are based on significantly more complex learning to rank algorithms than ours.


Learning-to-rank AdaBoost Class Probability Calibration 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Busa-Fekete, R., Kégl, B., Éltető, T., Szarvas, G.: Ranking by calibrated AdaBoost. In: JMLR W&CP, vol. 14, pp. 37–48 (2011)Google Scholar
  2. 2.
    Cao, Z., Qin, T., Liu, T., Tsai, M., Li, H.: Learning to rank: from pairwise approach to listwise approach. In: Proceedings of the 24rd International Conference on Machine Learning, pp. 129–136 (2007)Google Scholar
  3. 3.
    Cesa-Bianchi, N., Lugosi, G.: Prediction, Learning, and Games. Cambridge University Press, New York (2006)CrossRefzbMATHGoogle Scholar
  4. 4.
    Chapelle, O., Chang, Y.: Yahoo! Learning to Rank Challenge Overview. In: Yahoo Learning to Rank Challenge (JMLR W&CP), Haifa, Israel, vol. 14, pp. 1–24 (2010)Google Scholar
  5. 5.
    Chapelle, O., Metlzer, D., Zhang, Y., Grinspan, P.: Expected reciprocal rank for graded relevance. In: Proceeding of the 18th ACM Conference on Information and Knowledge Management, pp. 621–630. ACM, New York (2009)Google Scholar
  6. 6.
    Chapelle, O., Wu, M.: Gradient descent optimization of smoothed information retrieval metrics. Information Retrievel 13(3), 216–235 (2010)CrossRefGoogle Scholar
  7. 7.
    Cossock, D., Zhang, T.: Statistical analysis of Bayes optimal subset ranking. IEEE Transactions on Information Theory 54(11), 5140–5154 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Freund, Y., Iyer, R., Schapire, R.E., Singer, Y.: An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research 4, 933–969 (2003)MathSciNetzbMATHGoogle Scholar
  9. 9.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55, 119–139 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Herbrich, R., Graepel, T., Obermayer, K.: Large margin rank boundaries for ordinal regression. In: Smola, B., Schoelkopf, S. (eds.) Advances in Large Margin Classifiers, pp. 115–132. MIT Press, Cambridge (2000)Google Scholar
  11. 11.
    Kégl, B., Busa-Fekete, R.: Boosting products of base classifiers. In: International Conference on Machine Learning, Montreal, Canada, vol. 26, pp. 497–504 (2009)Google Scholar
  12. 12.
    Li, P., Burges, C., Wu, Q.: McRank: Learning to rank using multiple classification and gradient boosting. In: Advances in Neural Information Processing Systems, vol. 19, pp. 897–904. The MIT Press, Cambridge (2007)Google Scholar
  13. 13.
    Mease, D., Wyner, A.: Evidence contrary to the statistical view of boosting. Journal of Machine Learning Research 9, 131–156 (2007)Google Scholar
  14. 14.
    Niculescu-Mizil, A., Caruana, R.: Obtaining calibrated probabilities from boosting. In: Proceedings of the 21st International Conference on Uncertainty in Artificial Intelligence, pp. 413–420 (2005)Google Scholar
  15. 15.
    Rissanen, J.: A universal prior for integers and estimation by minimum description length. Annals of Statistics 11, 416–431 (1983)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Robertson, S., Zaragoza, H.: The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. 3, 333–389 (2009)CrossRefGoogle Scholar
  17. 17.
    Schapire, R.E., Singer, Y.: Improved boosting algorithms using confidence-rated predictions. Machine Learning 37(3), 297–336 (1999)CrossRefzbMATHGoogle Scholar
  18. 18.
    Valizadegan, H., Jin, R., Zhang, R., Mao, J.: Learning to rank by optimizing NDCG measure. In: Advances in Neural Information Processing Systems, vol. 22, pp. 1883–1891 (2009)Google Scholar
  19. 19.
    Wu, Q., Burges, C.J.C., Svore, K.M., Gao, J.: Adapting boosting for information retrieval measures. Inf. Retr. 13(3), 254–270 (2010)CrossRefGoogle Scholar
  20. 20.
    Xu, J., Li, H.: AdaRank: a boosting algorithm for information retrieval. In: SIGIR 2007: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 391–398. ACM, New York (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Róbert Busa-Fekete
    • 1
    • 2
  • Balázs Kégl
    • 1
    • 3
  • Tamás Éltető
    • 3
  • György Szarvas
    • 2
    • 4
  1. 1.Linear Accelerator Laboratory (LAL)University of Paris-Sud, CNRSOrsayFrance
  2. 2.Research Group on Artificial Intelligence of the Hungarian Academy of Sciences and University of SzegedSzegedHungary
  3. 3.Computer Science Laboratory (LRI)University of Paris-Sud, CNRS and INRIA-SaclayOrsayFrance
  4. 4.Ubiquitous Knowledge Processing (UKP) Lab, Computer Science DepartmentTechnische Universität DarmstadtDarmstadtGermany

Personalised recommendations