Rank aggregation using latent-scale distance-based models

Abstract

Rank aggregation aims at combining rankings of a set of items assigned by a sample of rankers to generate a consensus ranking. A typical solution is to adopt a distance-based approach to minimize the sum of the distances to the observed rankings. However, this simple sum may not be appropriate when the quality of rankers varies. This happens when rankers with different backgrounds may have different cognitive levels of examining the items. In this paper, we develop a new distance-based model by allowing different weights for different rankers. Under this model, the weight associated with a ranker is used to measure his/her cognitive level of ranking of the items, and these weights are unobserved and exponentially distributed. Maximum likelihood method is used for model estimation. Extensions to the cases of incomplete rankings and mixture modeling are also discussed. Empirical applications demonstrate that the proposed model produces better rank aggregation than those generated by Borda and the unweighted distance-based models.

Appendix: Sampling from a general distance-based Model

It is straightforward to simulate ranking data from the Kendall distance-based model, because of the nice decomposition of Kendall distance into a set of independent variables $V_{i}\left( \pi _{s}\circ \pi _{0}^{-1}\right)$ (see Sect. 4.1) which can be sampled easily. The detailed algorithm can be found in Ceberio et al. (2014). Recently Irurozki et al. (2018, 2014) developed two different methods to sample from models using Cayley distance and Ulam distance. Their simulation methods require the knowledge of special properties of the distance measures and they may not be able to be generalized to other distances. Here we introduce a general method to sample from any distance-based model.

We need to sample from f(x) here. The method begins at $s=0$ with the selection of $X^{(0)}=x^{(0)}$ drawn at random from some stating distribution g, with the requirement that $f\left( x^{(0)}\right) >0$. Given $X^{(s)}=x^{(s)}$ the algorithm generates $X^{(s+1)}$ as follows:

• 1. Sample a candidate value $X^{*}$ from a proposal distribution $g\left( \cdot \mid x^{(s)}\right)$.

• 2. Compute the Metropolis–Hastings ratio $R_{MH}\left( x^{(s)},X^{*}\right)$, where

  $$\begin{aligned} R_{MH}\left( x^{(s)},X^{*}\right) =\frac{f\left( X^{*}\right) g\left( x^{(s)}\mid X^{*}\right) }{f\left( x^{(t)}\right) g\left( X^{*}\mid x^{(t)}\right) }. \end{aligned}$$

• 3. Sample a random value for $X^{(s+1)}$ as follow:

  $$\begin{aligned} X^{(s+1)}={\left\{ \begin{array}{ll} X^{*} &{} \text{ with } \text{ probability }\ \min \left\{ 1,R_{MH}\left( x^{(s)},X^{*}\right) \right\} \\ x^{(s)} &{} \text{ otherwise } \end{array}\right. } \end{aligned}$$

• 4. Assign $s:=s+1$ and go to step 1

Back to our problem, given $\pi _{0}$, $\lambda$, we need to sample from the distance-based model: $f(\pi )=\frac{\exp \left[ -\lambda d(\pi ,\pi _{0})\right] }{C(\lambda )}$ is the distance function we choose. Note that we don’t need to calculate the normalizing constant $C(\lambda )$ in the $R_{MH}\left( x^{(s)},X^{*}\right)$ function because the $C(\lambda )$ terms cancel. For the proposal distribution $g(\cdot |\centerdot )$ here, we will introduce two proposal distribution and compare their difference.

The proposal distribution for the algorithm can be chosen as a symmetric distribution so that $g(x^{*}\mid x^{(s)})=g(x^{(s)}\mid x^{*})$. In our case, our choice of $g(x^{*}\mid x^{(s)})$ imposes small perturbation of the elements of $x^{(s)}$. Given $x^{(s)}$, we uniformly picked two elements of $x^{(s)}$ and swap their positions as our proposal $\pi ^{*}$. Since this swap distribution is symmetric, so the Hasting corrections in the Metropolis–Hastings ratio cancel, and the ratio is given as:

$$\begin{aligned} R_{MH}\left( x^{(s)},X^{*}\right) =\frac{\exp \left( -\lambda d\left( \pi ^{*},\pi _{0}\right) \right) }{\exp \left( -\lambda d\left( \pi ^{(s)},\pi _{0}\right) \right) } \end{aligned}$$

When we need to sample from the distance-based Latent-scale Models, we first sample $\lambda _{k}$ from $exp(\lambda )$. Then we sample $\pi _{k}$ from simple distance-based model with $\lambda _{k}$ using the method above.