Displacement Based Unsupervised Metric for Evaluating Rank Aggregation

  • Maunendra Sankar Desarkar
  • Rahul Joshi
  • Sudeshna Sarkar
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6744)


Rank Aggregation is the problem of aggregating ranks given by various experts to a set of entities. In context of web, it has applications like building metasearch engines, combining user preferences etc. For many of these applications, it is difficult to get labeled data and the aggregation algorithms need to be evaluated against unsupervised evaluation metrics. We consider the Kendall-Tau unsupervised metric which is widely used for evaluating rank aggregation task. Kendall Tau distance between two permutations is defined as the number of pairwise inversions among the permutations. The original Kendall Tau distance treats each inversion equally, irrespective of the differences in rank positions of the inverted items. In this work, we propose a variant of Kendall-Tau distance that takes into consideration this difference in rank positions. We study, examine and compare various available supervised as well as unsupervised metrics with the proposed metric. We experimentally demonstrate that our modification in Kendall Tau Distance makes it potentially better than other available unsupervised metrics for evaluating aggregated ranking.


Information Retrieval Rank Aggregation Distance Metrics Kendall Tau Distance 


  1. 1.
    Belkin, N.J., Kantor, P., Fox, E.A., Shaw, J.A.: Combining the evidence of multiple query representations for information retrieval. In: TREC 1995, Elmsford, NY, USA, pp. 431–448. Pergamon Press, Inc., Oxford (1995)Google Scholar
  2. 2.
    Carterette, B.: On rank correlation and the distance between rankings. In: SIGIR 2009, pp. 436–443. ACM Press, New York (2009)Google Scholar
  3. 3.
    Dwork, C., Kumar, R., Naor, M., Sivakumar, D.: Rank aggregation methods for the web. In: WWW 2001, pp. 613–622 (2001)Google Scholar
  4. 4.
    Jurman, G., Riccadonna, S., Visintainer, R., Furlanello, C.: Canberra distance on ranked lists. In: Proceedings, Advances in Ranking NIPS 2009 Workshop, pp. 22–27 (2009)Google Scholar
  5. 5.
    Kumar, R., Vassilvitskii, S.: Generalized distances between rankings. In: WWW 2010, pp. 571–580 (2010)Google Scholar
  6. 6.
    T.-y. Liu, J., Xu, T., Qin, W., Xiong, H.: Li: Letor: Benchmark dataset for research on learning to rank for information retrieval. In: SIGIR 2007 Workshop on Learning to Rank for Information Retrieval (2007)Google Scholar
  7. 7.
    Liu, Y., Liu, T.-Y., Qin, T., Ma, Z., Li, H.: Supervised rank aggregation. In: WWW, pp. 481–490 (2007)Google Scholar
  8. 8.
    Sanderson, M., Soboroff, I.: Problems with kendall’s tau. In: SIGIR 2007, pp. 839–840 (2007)Google Scholar
  9. 9.
    Schalekamp, F., van Zuylen, A.: Rank aggregation: Together we’re strong. In: ALENEX, pp. 38–51 (2009)Google Scholar
  10. 10.
    Yilmaz, E., Aslam, J.A., Robertson, S.: A new rank correlation coefficient for information retrieval. In: SIGIR 2008, pp. 587–594 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Maunendra Sankar Desarkar
    • 1
  • Rahul Joshi
    • 1
  • Sudeshna Sarkar
    • 1
  1. 1.Department of Computer Science and EngineeringIndian Institute of Technology KharagpurKharagpurIndia

Personalised recommendations