# Pairwise Comparisons in the Form of Difference of Ranks - Testing of Estimates of the Preference Relation

Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1081)

## Abstract

The paper presents tests for verification of estimates of the preference relation, obtained of the basis of multiple independent pairwise comparisons, in the form of difference of ranks, with random errors. The relation can have strict or weak form, while an estimate is obtained with the use of the idea of the nearest adjoining order (NAO). The approach to verification is the original concept of the author – it develops the ideas applied to binary comparisons. Some of the proposed tests are non-parametric, i.e. do not require any parameters of distribution of comparisons errors; remaining tests are based on exact or limiting distributions. Estimates verified with the use of the proposed tests are highly reliable and require acceptable computational costs.

## Keywords

Tests for estimates of preference relation Pairwise comparisons – difference of ranks Nearest adjoining order method

## 1 Introduction

The estimators of the preference relation based on multiple pairwise comparisons in the form of difference of ranks with random errors, proposed in Klukowski (2000, 2011 Chapt. 8, 9), have good statistical properties under weak assumptions about comparisons errors. The idea of these estimators have been introduced by Slater (1961) and developed by other authors (see David 1988, Bradley 1976) – for binary comparisons. The properties of the estimators based on differences of ranks are valid under the principal assumption that the preference relation is true model of data. Therefore, existence of the relation has to be verified with the use of statistical tests. The paper presents such tests for the relation in strict (linear) and weak form (alternative of strict relation and equivalence relation). The form of the tests is not the same as in binary case – in general test statistics have more complex distributions, e.g. multinomial with unknown parameters. Some of these tests are nonparametric and do not require any parameters, a part of them are based on exact or limiting distributions (Gaussian and t-Student).

Estimates based on NAO idea confirmed by proposed tests are highly reliable. The NAO method requires optimal solutions of an appropriate discrete optimization problems, which are more complex than in binary case. They can be solved with the use of complete enumeration - for number of element not higher that 20 - and using heuristic algorithms for higher number.

The paper consists of five sections and Appendix. The second section presents the theoretical basis of the estimation problem: assumptions about distributions of errors of pairwise comparisons, form of the estimators and their statistical properties. In next section are formulated original tests for strict preference relation – they verifies some basic feature of true relation. The fourth section present tests for weak form of the relation; some of the tests are based on original results (distributions of test statistics), while remaining use known concepts (parametric and nonparametric). Last section summarizes the results. Appendix presents the test for mode of the multinomial distribution – useful for verification of property of some distribution.

## 2 Estimation Problem, Assumptions About Comparisons Errors, Form of Estimators and Their Properties

### 2.1 Estimation Problem for Multivalent Comparisons

We are given a finite set of elements  $${\mathbf{X}} = \{ x_{1} , \ldots , x_{m} \}$$ (3≤ m < ∞). It is assumed that there exists in the set $${\mathbf{X}}$$ the complete preference relation – in strict or weak form; the weak form is alternative of: equivalence relation (reflexive, transitive, symmetric) and strict relation, while strict form is transitive and asymmetric. The relation generates some ordered family of subsets $$\chi_{1}^{*} , \ldots , \chi_{n}^{*}$$ $$\left( {2 \le n \le m} \right)$$; in the case of strict relation $$n = m$$, in the case of weak relation $$n < m$$ and each subset $$\chi_{q}^{*}$$ $$(1 \le q \le n)$$ includes equivalent elements only. The family $$\chi_{1}^{*} , \ldots , \chi_{n}^{*}$$ have the following properties:
$$\bigcup\limits_{q = 1}^{n} {\mathop \chi \nolimits_{q}^{*} } = {\mathbf{X}},\quad \chi_{r}^{*} \cap \chi_{s}^{*} = \left\{ {\mathbf{0}} \right\},$$
(1)
where:
{0} – the empty set,
$$x_{i} ,x_{j} \in \chi_{r}^{*} \; \equiv \,x_{i} ,x_{j} - \,{\text{equivalent}}\,{\text{elements}},$$
(2)
$$\left( {x_{i} \in \chi_{r}^{*} } \right) \wedge \left( {x_{i} \in \chi_{s}^{*} } \right) \left( {j \ne i, r < s} \right) \equiv x_{i} \;{\text{precedes}}\,x_{j} .$$
(3)

In the case of strict relation equivalent elements do not exist.

The relation defined by (1)–(3) can be expressed by values $$T\left( {x_{i} ,x_{j} } \right)$$ $$\left( {\left( {x_{i} ,x_{j} } \right) \in {\mathbf{X}} \times {\mathbf{X}}} \right)$$ defined as follows:
$$T\left( {x_{i} ,x_{j} } \right) = r - s \equiv x_{i} \in \chi_{r}^{*} , x_{j} \in \chi_{s }^{*} \left( {s \ne r} \right).$$
(4)

In the case of strict form of the relation the set of values of $$T\left( {x_{i} ,x_{j} } \right)$$ has a form $$\rho_{l} = \left\{ { - \left( {m - 1} \right), \ldots , - 1, 1, \ldots , \left( {m - 1} \right)} \right\}$$. In the case of weak relation the set $$\rho_{l}$$ is replaced by form $$\rho_{w} = \left\{ { - \upsilon , \ldots , - 1, 0, 1, \ldots , \upsilon } \right\}$$, where: $$\upsilon$$ – integer satisfying inequality $$\upsilon < m - 1$$.

### 2.2 Assumptions About Distributions of Errors of Pairwise Comparisons

The relation $$\chi_{1}^{*} , \ldots , \chi_{n}^{*}$$ is to be estimated on the basis of N (N ≥ 1) comparisons of each pair  $$\left( {x_{i} ,x_{j} } \right) \in {\mathbf{X}} \times {\mathbf{X}}$$; any comparison  $$g_{k} \left( {x_{i} ,x_{j} } \right)$$  (k = 1, …, N) evaluates the value of $$T\left( {x_{i} ,x_{j} } \right)$$ (assumes values from the set $$\rho_{l}$$ or $$\rho_{w}$$) and can be disturbed by a random error.

1. A1.

The number of subsets n is unknown - in the case of the weak form of the relation (in the strict case $$n = m$$).

2. A2.

The probabilities of errors $$g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right)$$ (k = 1, …, N) have to satisfy the following assumptions:

$$\begin{array}{*{20}l} {\sum\nolimits_{r \le 0} {(P(g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) = r\,|\, T\left( {x_{i} ,x_{j} } \right) = \kappa_{ij} ) > \frac{1}{2},} } \hfill \\ {(\kappa_{ij} \in \left\{ {0, \ldots , \pm \left( {m - 1} \right)} \right\})} \hfill \\ \end{array}$$
(5)
$$\begin{array}{*{20}l} {\sum\nolimits_{r \ge 0} {(P(g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) = r\,|\, T\left( {x_{i} ,x_{j} } \right) = \kappa_{ij} ) > \frac{1}{2},} } \hfill \\ {(\kappa_{ij} \in \left\{ {0, \ldots , \pm \left( {m - 1} \right)} \right\}),} \hfill \\ \end{array}$$
(6)
$$\begin{array}{*{20}l} {P\left( {g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) = r} \right) \ge P(g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) = r + 1\;| } \hfill \\ {0 \le r \le 2m - 3),} \hfill \\ \end{array}$$
(7)
$$\begin{array}{*{20}l} {P\left( {g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) = r} \right) \ge P(g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) = r - 1\;| } \hfill \\ { - 2m + 3 \le r \le 0)} \hfill \\ \end{array}$$
(8)
$$\sum\nolimits_{{r = - 2\left( {m - 1} \right)}}^{{2\left( {m - 1} \right)}} {P(g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right)) = r) = 1}$$
(9)
1. A3.

The comparisons  $$g_{k} \left( {x_{i} ,x_{j} } \right)$$   $$\left( {\left( {x_{i} ,x_{j} } \right) \in {\mathbf{X}} \times {\mathbf{X}}} \right)$$; (k = 1, …, N) are independent random variables.

The assumptions A1–A3 reflect the following properties of distributions of comparisons errors: number of subsets in the case of weak form of the relation is unknown, zero is the median and the mode of each distribution of comparison errors (inequalities (7)–(11)), the comparisons are realizations of independent random variables, the expected value of any error can differ from zero.

### 2.3 The Form of Estimators

The estimator presented in Klukowski (2011 Chap. 3, 2012), is based on the total sum of absolute differences between relation form (values $$T(x_{i} ,x_{j} )$$) and comparisons $$g_{k} \left( {x_{i} ,x_{j} } \right)$$   $$\left( {\left( {x_{i} ,x_{j} } \right) \in {\mathbf{X}} \times {\mathbf{X}}} \right)$$. The estimates will be denoted $$\hat{\chi }_{1} , \ldots ,\hat{\chi }_{{\hat{n}}}$$ or $$\hat{T}(x_{i} ,x_{j} )$$ (in the case of strict relation $$\hat{n} = m$$). They are obtained as optimal solution of the discrete minimization problem:
$$\mathop {\hbox{min} }\limits_{{\chi_{1}^{{}} , \ldots , \chi_{r}^{{}} \in F_{{\mathbf{X}}}^{{}} }} \{ \sum\nolimits_{{\left\langle {i,j} \right\rangle \in R_{m} }} {\sum\nolimits_{k = 1}^{N} {\left| {g_{k} \left( {x_{i} ,x_{j} } \right) - t\left( {x_{i} ,x_{j} } \right)} \right|,\} } }$$
(10)
where:
• $$F_{{\mathbf{X}}}^{{}}$$ - the feasible set: the family of all relations $$\chi_{1}^{{}} , \ldots , \chi_{r}^{{}}$$ in the set X; in the case of strict relation $$r = m$$, in the case of weak relation $$2 \le r \le m - 2$$,

• $$t(x_{i} ,x_{j} )$$  - the values describing any relation $$\chi_{1}^{{}} , \ldots , \chi_{r}^{{}}$$ from $$F_{{\mathbf{X}}}^{{}}$$,

• $$R_{m}$$ - the set of the form $$R_{m} = \{ \left\langle {i,j} \right\rangle \;| 1 \le i,j \le m; j > i\}$$.

The number of estimates, resulting from the criterion function (10) can exceed one, the minimal value of the function equals zero.

### 2.4 The Properties of Estimates

The analytical properties of the estimates, resulting from (10) are based on the random variables: $$\sum\nolimits_{{\left\langle {i,j} \right\rangle \in R_{m} }} {\sum\nolimits_{k = 1}^{N} {\left| {g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right)} \right|} }$$. The following results have been obtained by the author (Klukowski 2011, Chapt. 8):

### Theorem

The following relationships are true:

$$\begin{array}{*{20}l} {\left( {\text{i}} \right)} \hfill & {E(\sum\nolimits_{{\left\langle {i,j} \right\rangle \in R_{m} }} {\sum\nolimits_{k = 1}^{N} {\left| {g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right)} \right|)} } } \hfill \\ {} \hfill & { < E(\sum\nolimits_{{\left\langle {i,j} \right\rangle \in R_{m} }} {\sum\nolimits_{k = 1}^{N} {\left| {g_{k} \left( {x_{i} ,x_{j} } \right) - \tilde{T}\left( {x_{i} ,x_{j} } \right)} \right|),} } } \hfill \\ \end{array}$$
(11)
where:
$$\tilde{T}\left( {x_{i} ,x_{j} } \right)$$- values corresponding to any relation $$\tilde{\chi }_{1} , \ldots , \tilde{\chi }_{{\tilde{n}}}$$ different than $$\chi_{1}^{*} , \ldots , \chi_{n}^{*}$$,
$$\begin{array}{*{20}l} {\left( {\text{ii}} \right) } \hfill & {\mathop {\text{limVar}}\nolimits_{N \to \infty } (\frac{1}{N}\sum\nolimits_{{\left\langle {i,j} \right\rangle \in R_{m} }} {\sum\nolimits_{k = 1}^{N} {\left| {g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right)} \right|) = 0,} } } \hfill \\ {} \hfill & {\mathop {\text{limVar}}\limits_{N \to \infty } (\frac{1}{N}\sum\nolimits_{{\left\langle {i,j} \right\rangle \in R_{m} }} {\sum\nolimits_{k = 1}^{N} {\left| {g_{k} \left( {x_{i} ,x_{j} } \right) - \tilde{T}\left( {x_{i} ,x_{j} } \right)} \right|) = 0,} } } \hfill \\ \end{array}$$
(12)
$$\begin{array}{*{20}l} {\left( {\text{iii}} \right)} \hfill & {\mathop {\lim }\limits_{N \to \infty } P(\sum\nolimits_{{\left\langle {i,j} \right\rangle \in R_{m} }} {\sum\nolimits_{k = 1}^{N} {\left| {g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right)} \right|} } } \hfill \\ {} \hfill & { < \sum\nolimits_{{\left\langle {i,j} \right\rangle \in R_{m} }} {\sum\nolimits_{k = 1}^{N} {\left| {g_{k} \left( {x_{i} ,x_{j} } \right) - \tilde{T}\left( {x_{i} ,x_{j} } \right)} \right|) = 0,} } } \hfill \\ \end{array}$$
(13)
moreover:
$$\begin{array}{*{20}l} {P\left( {\sum\nolimits_{{\left\langle {i,j} \right\rangle \in R_{m} }} {\sum\nolimits_{k = 1}^{N} {\left| {g_{k} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right)} \right|} } } \right.} \hfill \\ {\left. {\left. { < \;\;\;\sum\nolimits_{{\left\langle {i,j} \right\rangle \in R_{m} }} {\sum\nolimits_{k = 1}^{N} {g_{k} \left( {x_{i} ,x_{j} } \right) - } } \tilde{T}\left( {x_{i} ,x_{j} } \right)} \right|} \right) \ge 1 - exp\left\{ { - 2N\vartheta_{ij}^{2} } \right\},} \hfill \\ \end{array}$$
(14)
where: $$\vartheta_{ij}^{2}$$ is some constant depending on distributions of comparisons $$g_{k} \left( {x_{i} ,x_{j} } \right)$$ $$\left( {i = 1, \ldots ,m;j \ne i;k = 1, \ldots ,N} \right)$$ and the form of the relation $$\tilde{\chi }_{1} , \ldots , \tilde{\chi }_{{\tilde{n}}}$$ (relationship (14) is based on the Hoeffding inequality 1963, see also Serfling 1980, point 2.3.2).

The relationships (i)–(iii) guarantee consistency of the estimator, while inequality (14) indicates fast (exponential type) convergence to the actual relation.

## 3 Tests for Verification of an Estimate of Strict Form of the Relation

The tests presented in this section verify estimates of the strict relation on the basis of properties of comparisons $$g_{k} \left( {x_{i} ,x_{j} } \right)$$ $$\left( {\left( {x_{i} ,x_{j} } \right) \in {\mathbf{X}} \times {\mathbf{X}}} \right)$$. The following random variables:
$$\begin{array}{*{20}l} {\varphi_{ijk} = g_{k} \left( {x_{i} ,x_{j} } \right) + g_{k} \left( {x_{m - i + 1} ,x_{m - j + 1} } \right)} \hfill \\ {\left( {i = 1, \ldots ,\frac{m}{2}, j \ne i, k = 1, \ldots , N} \right),\;m{\text{ - odd}}} \hfill \\ \end{array}$$
(15)
and
$$\psi_{irjk} = \left\{ {\begin{array}{*{20}l} {0\,if\,sgn\left( {g_{k} \left( {x_{i} ,x_{j} } \right)} \right) = sgn\left( {g_{k} \left( {x_{r} ,x_{j} } \right)} \right); j < i\,or\,r < j;} \hfill \\ {1\,if\,sgn\left( {g_{k} \left( {x_{i} ,x_{j} } \right)} \right) \ne sgn\left( {g_{k} \left( {x_{r} ,x_{j} } \right)} \right); j < i\,or\,r < j;} \hfill \\ {1\,if\,sgn\left( {g_{k} \left( {x_{i} ,x_{j} } \right)} \right) \ne sgn\left( {g_{k} \left( {x_{r} ,x_{j} } \right)} \right); i < j < r; } \hfill \\ {0\,if\,sgn\left( {g_{k} \left( {x_{i} ,x_{j} } \right)} \right) = sgn\left( {g_{k} \left( {x_{r} ,x_{j} } \right)} \right); i < j < r, } \hfill \\ \end{array} } \right.$$
(16)
where:

equality of $$sgn( \cdot )$$ means: both values $$g_{k} \left( {x_{i} ,x_{j} } \right)$$ and $$g_{k} \left( {x_{r} ,x_{j} } \right)$$ are negative or positive, are applied for this purpose.

For even value of m in (15), it is assumed: $$i = 1, \ldots , \frac{m - 1}{2}$$.

If an estimate obtained is the same as true relation then variables (15) and (16) have some known properties.

The variables:  $$\varphi_{ijk} \left( {1 \le i \le \frac{m}{2}, j \ne i, k = 1, \ldots , N} \right)$$ have - for fixed i - expected values, modes and medians equal zero and distributions symmetric around zero.

The variables (16) can be used for verification of the fact that a difference of ranks of elements from the subsets $$\hat{\chi }_{i}^{{}}$$ and $$\hat{\chi }_{r}^{{}}$$ is equal to $$r - i$$. It is useful to assume $$r - i = \frac{m}{2}$$ for odd value of m and $$r - i = \frac{m - 1}{2}$$ for even value, because the test can be used for verification consecutive values of i and r (m – odd), i.e.: 1 and $$\frac{m}{2} + 1, \ldots ,\frac{m}{2}$$ and m.

The null hypothesis H0 of the test based on variables (15) states that the estimate $$\hat{\chi }_{1}^{{}} , \ldots , \hat{\chi }_{n}^{{}}$$ is the same as $$\chi_{1}^{*} , \ldots , \chi_{n}^{*}$$, the alternative H1 - that is not the same. For simplicity it is assumed that $$\chi_{i}^{*} = \left\{ {x_{i} } \right\} \left( {i = 1, \ldots , m} \right)$$.

The random variables (15) have the following distributions under H0. Any variable $$\varphi_{ijk}$$ is sum of two random variables $$g_{k} \left( {x_{i} ,x_{j} } \right)$$ and $$g_{k} \left( {x_{m - i + 1} ,x_{m - j + 1} } \right)$$ – each assuming values from the set $$\left\{ { - \left( { m - 1} \right), \ldots , - 1,1, \ldots , \left( {m - 1} \right)} \right\}$$; therefore their sum assumes values from the set $$\left\{ { - 2\left( { m - 1} \right), \ldots , - 1, 0, 1, \ldots , 2\left( {m - 1} \right)} \right\}$$. It is clear that $$T\left( {x_{i} ,x_{j} } \right) = - T\left( {x_{m - i + 1} ,x_{m - j + 1} } \right)$$; therefore it is postulated that the distributions of variables $$g_{k} \left( {x_{i} ,x_{j} } \right)$$ and $$- g_{k} \left( {x_{m - i + 1} ,x_{m - j + 1} } \right)$$ are the same. Usually (in applications) the distributions of comparison $$g_{k} \left( {x_{i} ,x_{j} } \right)$$ are not known; moreover, for fixed index i they are not the same for individual values of an index j. The proposed tests take into account these features.

The properties of the variables $$\varphi_{ijk}$$ (symmetry and expected value, mode and median equal zero) can be proved in the following way. Let start from the case i = 1. The variables $$g_{k} \left( {x_{1} ,x_{j} } \right)\, \left( {j = 2, \ldots ,m} \right)$$ have distributions with mode and median equal to - respectively: $$- 1, - 2, \ldots , - \left( {m - 1} \right)$$, while the variables $$g_{k} \left( {x_{m} ,x_{m - j + 1} } \right)$$ - mode and median equal: $$1, 2, \ldots ,m - 1$$. Moreover the distributions of the variables $$g_{k} \left( {x_{1} ,x_{j} } \right)$$ and $$- g_{k} \left( {x_{m} ,x_{m - j + 1} } \right)$$ are the same. The form of distributions of the variables $$\varphi_{1,jk}$$ is presented in the Table 1.
Table 1.

The distribution of the random variables $$\varphi_{1,2,k} \left( {1 \le k \le N} \right)$$.

$$\varphi_{1,2,k} = g_{k} \left( {x_{1} ,x_{j} } \right) + g_{k} \left( {x_{m} ,x_{m - 1} } \right)$$

Value $$g_{k} \left( {x_{m} ,x_{m - 1} } \right)$$

(probability)

m−1

$$(\alpha_{m - 2} )$$

m−2

$$(\alpha_{m - 3} )$$

−(m−2)

$$(\alpha_{ - m + 1} )$$

−(m−1)

$$(\alpha_{ - m} )$$

Value $$g_{k} \left( {x_{1} ,x_{j} } \right)$$ (probability)

m + 1

$$(\alpha_{ - m + 2} )$$

0

$$\left( {\alpha_{m - 2}^{2} } \right)$$

−1

$$(\alpha_{m - 2} \alpha_{m - 3} )$$

−2 m + 3

$$(\alpha_{m - 2} \alpha_{ - m + 1} )$$

−2(m−1)

$$\left( {\alpha_{m - 2}^{{}} \alpha_{ - m} } \right)$$

m + 2

$$(\alpha_{ - m + 3} )$$

1

$$(\alpha_{m - 3} \alpha_{m - 2} )$$

0

$$\left( {\alpha_{m - 3}^{2} } \right)$$

−2(m−2)

$$\left( {\alpha_{m - 3}^{2} \alpha_{ - m + 1} } \right)$$

−2 m + 3

$$(\alpha_{m - 3} \alpha_{ - m} )$$

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

m−2

$$(\alpha_{m - 1} )$$

m−3

$$(\alpha_{m - 1} \alpha_{m - 2} )$$

2(m−2)

$$\left( {\alpha_{m - 1}^{{}} \alpha_{m - 3}^{{}} } \right)$$

0

$$\left( {\alpha_{m - 1}^{2} } \right)$$

−1

$$(\alpha_{m - 1} \alpha_{ - m} )$$

m−1

$$(\alpha_{m} )$$

2(m−1)

$$\left( {\alpha_{m}^{{}} \alpha_{m - 2} } \right)$$

m−3

$$(\alpha_{m} \alpha_{m - 3} )$$

1

$$(\alpha_{m} \alpha_{ - m + 1} )$$

0

$$\left( {\alpha_{m}^{2} } \right)$$

Upper expression in each cell means a value of the variable $$\varphi_{1,2,k} = g_{k} \left( {x_{1} ,x_{j} } \right) + g_{k} \left( {x_{m} ,x_{m - 1} } \right)$$, symbols $$\left( {\alpha_{l} } \right)$$ denote probabilities $$P(g_{k} (x_{1} ,x_{2} ) - T(x_{1} ,x_{2} ) = l) = P(g_{k} (x_{m} ,x_{m - 1} ) - T(x_{m} ,x_{m - 1} ) = - l)$$.

The data presented in the Table 1 shows that: $$P(\varphi_{1,2,k} = 0) > P(\varphi_{1,2,k} = l) \left( {k = 1, \ldots ,N;l \ne 0} \right)$$. It results from the following facts: • the probability $$P(\varphi_{1,2,k} = 0)$$ is the sum of diagonal probabilities from the Table 1, i.e. $$2\left( {m - 1} \right)$$ components, • each probability $$P(\varphi_{1,2,k} = l) \left( {l \ne 0} \right)$$ is the sum of appropriate non-diagonal probabilities from the Table 1, • the number of non-diagonal components is lower than the number of diagonal components, • each non-diagonal probability is lower or equal than some component of the probability $$P(\varphi_{1,2,k} = 0)$$. The inequality $$P(\varphi_{1,2,k} = 0) > P(\varphi_{1,2,k} = l)$$ indicates that zero is the mode of the distribution of any variable $$\varphi_{1,2,k}$$. The content of the Table 1 indicates also symmetry of the distribution of the variable $$\varphi_{1,2,k}$$ around zero and, therefore, expected value equal zero. In the same way can be shown properties of remaining variables $$\varphi_{ijk} \left( {i = 2, \ldots ,\frac{m}{2}; j \ne i, 1 \le k \le N} \right)$$. It is clear that if null hypothesis is not true then the properties of variables $$\varphi_{ijk}$$ are different.

The expected values of the variables $$\varphi_{ijk}$$ (i fixed, j ≠ i, k = 1, …, N) are equal zero, under null hypothesis, but their variances are, in general, non-equal. However, the range of possible values is limited (low) and not influences significantly results of proposed tests. The low range arises from assumptions about distributions of errors of pairwise comparisons, i.e.: median equal zero, unimodal distribution, finite set of values (see assumption A2, point 2.2). Especially, these facts are true for quasi-uniform distribution, defined in Sect. 4 below. Therefore, the following well-known tests can be used for testing the above properties of the variables $$\varphi_{ijk} \left( {i = 1, \ldots ,\frac{m}{2}; j \ne i, k = 1, \ldots ,N} \right)$$ (number of these variables is equal $$0,5m\left( {m - 2} \right)N$$):
• null expected value of the variables $$\varphi_{ijk}$$ - t-Student test (for appropriate value of $$0,5m\left( {m - 2} \right)N$$), Wilcoxon’s test for discrete random variables;

• median equal zero – test for the median, based on theory of series,

• symmetry of the distribution – the test of signs for discrete random variables with the median equal zero (for large number of variables the test statistics has limiting Gaussian distribution);

• mode of the distribution equal to zero – test for the mode in multinomial distribution (see Appendix).

As it was mentioned, the tests are based on random variables with moderately different variances and their results are approximated. It is also obvious that the best properties have the tests mentioned for low values of index $$i$$, namely $$i \le m/3$$. However, if all the tests indicate the same hypothesis (null or alternative) it increases its reliability. Rejecting of the null hypothesis indicates questionable elements of the estimate.

The idea of the test based on the random variables $$\psi_{irjk}^{{}}$$ is similar to the case of binary comparisons (see Klukowski – item 11 in Bibliography); however, now comparisons $$g_{k} \left( {x_{i} ,x_{j} } \right)$$ are multivalent with multinomial distributions. It can be shown that the variables $$\psi_{irjk}^{{}}$$ have distributions with known properties, i.e. value of mode and median for appropriate indexes $$i,j,r$$.

The distributions of the variables $$\psi_{irjk}^{{}}$$ assume, under null hypothesis H0 that difference of ranks between elements $$x_{i}$$ and $$x_{r}$$ is equal $$i - r$$, the following forms:
$$P\left( {\psi_{irjk}^{{}} = 0 \,|\,\;\;\;j\left\langle {i\,or\,j} \right\rangle i} \right) = \omega_{irjk} > 1/2,$$
(17)
$$P\left( {\psi_{irjk}^{{}} = 1 \,|\,\,\;\;\;j\left\langle {i\,or\,j} \right\rangle i} \right) = 1 - \omega_{irjk} ,$$
(18)
$$P\left( {\psi_{irjk}^{{}} = 1\,|\,\;\;\;i < j < r} \right) = \varpi_{irjk} > 1/2,$$
(19)
$$P\left( {\psi_{irjk}^{\left( \nu \right)} = 0\,|\,\;\;\;i < j < r} \right) = 1 - \varpi_{irjk} .$$
(20)

It is a result of assumptions about mode and median of distributions of comparisons errors equal zero (see inequality (5), (6)).

The relationships (17)–(20) indicate the following inequality for fixed $$i, r$$ under null hypothesis:
$$\begin{array}{*{20}l} {\frac{1}{{\left( {m - 2} \right)N}}E\left( {\sum\nolimits_{k = 1}^{N} {\sum\nolimits_{{j \notin \left( {i,r} \right)}} {\omega_{irjk}^{{}} } } + \sum\nolimits_{k = 1}^{N} {\sum\nolimits_{{j \in \left( {i,r} \right)}} {(1 - \varpi_{irjk}^{{}} )} } } \right)} \hfill \\ { = \frac{1}{{\left( {m - 2} \right)}}\left( {\sum\nolimits_{{j \notin \left( {i,r} \right)}} {\omega_{irj}^{{}} } + \sum\nolimits_{{j \in \left( {i,r} \right)}} {(1 - \varpi_{irj}^{{}} )} } \right) < \frac{1}{2}} \hfill \\ \end{array}$$
(21)

(notation $$j \in \left( {i,r} \right)$$ means values of integers: $$i + 1, \ldots ,r - 1$$, notation $$j \notin \left( {i,r} \right)$$ means values of integers: $$1, \ldots , i - 1$$ and $$r + 1, \ldots , m$$).

The value of the left hand side of the inequality (21) can be determined if the probabilities $$\mathop \omega \nolimits_{irj}^{{}}$$ and $$\mathop \varpi \nolimits_{irj}^{{}}$$ are known. In this case the random variable
$$\tfrac{1}{(m - 2)N}(\sum\limits_{k = 1}^{N} {} \sum\limits_{j \notin (i,r)}^{{}} {} \mathop \psi \nolimits_{irjk}^{{}} + (1 - \sum\limits_{k = 1}^{N} {} \sum\limits_{j \in (i,r)}^{{}} {} \mathop \psi \nolimits_{irjk}^{{}} ))$$
has limiting Gaussian distribution with known parameters – resulting from distributions of zero-one variables. If the null hypothesis is not true then the expected value assumes higher value, i.e. the rejection region of the test is right-hand side. In general, efficiency of the test increases for higher difference $$i - r$$. The probabilities (17), (19) are often in practice not known and it is suggested to estimate them (it can be done for appropriate value of N, at least several) or evaluate on the basis of the quasi-uniform multinomial distribution. Such the distribution can be regarded as an unfavorable one (for statistician), i.e. with maximal variance.
The quasi-uniform distribution is determined in the following way. Let us consider firstly the case of negative value of $$T\left( {x_{i} ,x_{j} } \right)$$, i.e. $$T\left( {x_{i} ,x_{j} } \right) = \mu \;\,(\mu < 0)$$. In this case errors of each comparison belong to the set $$\Theta _{\mu }^{{}} = \left\{ {l - \mu \,| \,\;\;\;l = - \left( {m - 1} \right), \ldots , - 1,1, \ldots ,m - 1} \right\}$$; number of elements of the set is equal 2(m–1). The probabilities of errors determined by quasi-uniform distribution have to satisfy the assumptions A1 – A3 (see point 2.2) and are obtained from the relationships:
$$P\left( {g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) < 0} \right) = P\left( {g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) > 0} \right),$$
(22)
$$\begin{array}{*{20}l} {P\left( {g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) = - l} \right) = P\left( {g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) = - l - 1} \right)} \hfill \\ {\left( {l = - 1, \ldots , - \left( {m - 1} \right) - \mu } \right),} \hfill \\ \end{array}$$
(23)
$$\begin{array}{*{20}l} {P\left( {g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) = l} \right) = P\left( {g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) = l + 1} \right)} \hfill \\ {\left( {l = 1, \ldots ,m - 1 - \mu } \right)} \hfill \\ \end{array}$$
(24)
$$\sum\nolimits_{{l\epsilon\Theta _{\mu } }} {P\left( {g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) = l} \right) = 1}$$
(25)

(in the case $$P\left( {g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) < 0} \right) = 0$$ or $$P\left( {g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) > 0} \right) = 0$$ it is assumed: $$P\left( {g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) = 0} \right) = \frac{1}{2} + \varepsilon \left( {\varepsilon \in \left( {0,\frac{1}{2}} \right)} \right)$$ and remaining probabilities equal to $$\left( {\frac{1}{2} - \varepsilon } \right)/\left( {2\left( {m - 2} \right)} \right)$$.

The solution of the system of Eqs. (22)–(25) can be obtained on the basis of the system of two linear equations with two variables, which has unique solution. Let us notice, that in the case of any distribution of error $$g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right)$$ corresponding to the case $$T\left( {x_{i} ,x_{j} } \right) \ne \pm \left( {m - 1} \right)$$ the number of values in left (negative) and right (positive) tail of distribution is not the same; in such the case the probability of the error equal zero is equal to the probabilities from the part of tail (left or right) with lower number of elements. Let us consider simple example $$T\left( {x_{i} ,x_{j} } \right) = - 1, m = 5$$. Then: $$g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) \in \left\{ { - 4, \ldots , - 1,1, \ldots ,4} \right\}$$, $$g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) - T\left( {x_{i} ,x_{j} } \right) \in \left\{ { - 3, \ldots , 0, 2, \ldots , 5} \right\}$$. The system of equations assumes the form: $$3z = 4y; 4z + 4y = 1$$, with the solution $$z = \frac{1}{7}$$, $$y = \frac{3}{28}$$, which implies $$P\left( {T\left( {x_{i} ,x_{j} } \right) - g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) = - 3} \right) = \ldots = P\left( {T\left( {x_{i} ,x_{j} } \right) - g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) = 0} \right) = \frac{1}{7};$$ $$P\left( {T\left( {x_{i} ,x_{j} } \right) - g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) = 2) = \ldots = \left( {T\left( {x_{i} ,x_{j} } \right) - g_{k}^{{}} \left( {x_{i} ,x_{j} } \right) = 5} \right) = \frac{3}{28}} \right)$$ .

The system (22)–(25) does not require any a priori parameters and any other knowledge about distributions of comparisons errors. The probabilities resulting from the system (22)–(25) determines multinomial distribution on appropriate subset of the set $$\Theta _{m}^{{}} = \left\{ { - 2\left( {m - 1} \right), \ldots , - 1, 0, 1, \ldots ,2\left( {m - 1} \right)} \right\}$$; it is possible to determine its expected value, variance and other parameters.

It is clear that results of the test based on the quasi-uniform distribution are approximated; however rejecting of null hypothesis indicates possibility of errors in an estimate under consideration. The fact that the quasi-uniform distribution has maximal variance indicates using of low significance levels, i.e. close to 0,1 rather than to 0,01.

Testing of difference of ranks with the use of quasi-uniform distribution seems a little cumbersome at first glance, but can be realized with the use of simple computer program.

## 4 Test for Weak Preference Relation

Weak form of the preference relation is the alternative of strict preference relation and equivalence relation, i.e. at least one of the subsets $$\chi_{q}^{*} \left( {1 \le q \le n} \right)$$ includes more than one element (m > n). Typical situation is $$\# \chi_{q}^{*} > 1$$ (symbol # denotes number of elements of a subset $$\chi_{q}^{*}$$) for each or majority of subsets. Verification of the estimate $$\hat{\chi }_{1}^{{}} , \ldots , \hat{\chi }_{{\hat{n}}}$$ is based on testing of identity of distributions of comparisons $$g_{k} \left( {x_{i} ,x_{j} } \right), g_{k} \left( {x_{r} ,x_{j} } \right)$$ $$\left( {j \ne i,r;k = 1, \ldots ,N} \right)$$ of elements belonging to the same subsets, i.e. $$g_{k} \left( {x_{i} ,x_{r} } \right) \in \chi_{q}^{*}$$. It is easy to show that in such the case the distribution of any difference:
$$g_{k} \left( {x_{i} ,x_{j} } \right) - g_{k} \left( {x_{r} ,x_{j} } \right)$$
(26)
have some known properties, especially:
• multinomial symmetric distribution determined on the set $$\left\{ { - \left( {2m - 1} \right), \ldots ,0, \ldots ,2\left( {m - 1} \right)} \right\}$$,

• expected value, mode and median equal zero,

• symmetry of distribution equal zero.

Moreover, the variance of the difference (26) is equal to sum of variances of both variables $$g_{k} \left( {x_{i} ,x_{j} } \right), g_{k} \left( {x_{r} ,x_{j} } \right)$$. In general, the variance is not the same for different variables: $$g_{k} \left( {x_{i} ,x_{j'} } \right) - g_{k} \left( {x_{r} ,x_{j'} } \right)$$ and $$g_{k} \left( {x_{i} ,x_{j''} } \right) - g_{k} \left( {x_{r} ,x_{j''} } \right)$$ $$\left( {j^{\prime\prime} \ne j^{\prime}} \right)$$. The variances of such variables can be estimated in the case of appropriate number of comparisons N, i.e. at least several.

The above properties of the difference (26) can be verified with a use of known tests for each pair of elements from any subset $$\hat{\chi }_{q}^{{}}$$ including more than one element. Firstly, the differences have expected values equal zero in multinomial symmetric distribution with non-equal variances for different values of index j. If the variances or their estimates are known then chi-square limiting distribution can be used as a base of the test. The test is based on the statistics:
$$\sum\nolimits_{k = 1}^{N} {\sum\nolimits_{j = 1}^{m} {\left( {g_{k} \left( {x_{i} ,x_{j} } \right) - g_{k} \left( {x_{r} ,x_{j} } \right)} \right)/s_{irj} )^{2} } } ,$$
(27)
where:

$$s_{irj}$$ - standard deviation of the variable $$g_{k} \left( {x_{i} ,x_{j} } \right) - g_{k} \left( {x_{r} ,x_{j} } \right)$$,

$$\mathop g\nolimits_{k}^{{}} (\mathop x\nolimits_{i}^{{}} ,\mathop x\nolimits_{i}^{{}} ) = 0$$ with variance equal zero.

The statistics (27) has limiting chi-square distribution with $$Nm$$ degrees of freedom, the value of the product $$Nm$$ must be not lower than 30. The rejection region is right-hand side. In the case of lower value $$Nm$$ it is necessary to use non-parametric tests, e.g. Wilcoxon’s test (the version for discrete distributions) for values: $$g_{k} \left( {x_{i} ,x_{1} } \right) - g_{k} \left( {x_{r} ,x_{1} } \right), \ldots ,g_{k} \left( {x_{i} ,x_{m} } \right) - g_{k} \left( {x_{r} ,x_{m} } \right)$$ $$\left( {k = 1, \ldots ,N} \right)$$. The test requires symmetric distributions of differences and equal variances; the first requirement can be verified by the test of signs, the second is typically not satisfied. Therefore, results of the test should be confirmed by other tests, especially by test for mode equal zero. Rejecting the null hypotheses for elements $$x_{i} ,x_{r} \in \hat{\chi }_{1}^{{}} \left( {r \ne i} \right)$$ indicates non-equivalency of these elements.

The next three properties can be verified with the use of known tests in a simple way. The mode of the variables (26) should be verified by the test for mode in multinomial distribution (see Appendix); however the result is approximated because of non-identical distributions of the differences (26) for individual values of index j. The median equal zero and symmetry of distribution can be tested by the test of signs for discrete random variables.

Positive results of the all tests confirm the obtained estimate, while negative indicate possibility of an incorrect element of the estimate. Of course, some number of negative results of tests - close to probabilities of errors do not disqualify the estimate.

The above test can be used also to verification of non-equivalency of elements from different subsets $$\hat{\chi }_{q} ,\hat{\chi }_{s}$$ $$\left( {q \ne s} \right)$$; non-equivalency corresponds with rejecting of the null hypothesis. Its efficiency increases with the value of difference $$s - q$$.

## 5 Summary and Conclusions

The paper presents proposition of the tests for verification of estimates $$\hat{\chi }_{q} \left( {q = 1, \ldots ,\hat{n}} \right)$$ of the preference relation (in strict and weak form) obtained with the use of NAO method on the basis of multiple pairwise comparisons in the form of difference of ranks. The test statistics have known distributions under null hypothesis – exact, limiting or evaluated. Some features of the estimates under H0 can be verified with the use of well-known tests. Positive results of the tests (typically acceptation of null hypotheses) indicate acceptable estimates, while negative results (rejection of null hypothesis) shows questionable features. Of course, some “slight” fraction of negative results, corresponding to significance levels and the second type errors, can also occur. Some of the tests do not require any values of parameters of random variables, but are rather rough statistical tools. The tests are especially useful in two cases: application of heuristic algorithms for the optimization problem (10) and possibility of incorrect model of data - e.g. relation with incomparable elements.

## References

1. Bradley, R.A.: Science, statistics and paired comparisons. Biometrics 32, 213–232 (1976)
2. David, H.A.: The Method of Paired Comparisons, 2nd edn. Ch. Griffin, London (1988)
3. Domański, C.: Statistical Tests. PWE, Warsaw (1990). (in Polish)Google Scholar
4. Gordon, A.D.: Classification, 2nd edn. Chapman&Hall/CRC, Boca Raton (1999)
5. Hansen, P., Jaumard, B.: Cluster analysis and mathematical programming. Math. Program. 79, 191–215 (1997)
6. Hoeffding, W.: Probability inequalities for sums of bounded random variables. JASA 58, 13–30 (1963)
7. Serfling, R.J.: Approximation Theorems of Mathematical Statistics. Wiley, Hoboken (1980)
8. Klukowski, L.: Some probabilistic properties of the nearest adjoining order method and its extensions. Ann. Oper. Res. 51, 241–261 (1994)
9. Klukowski, L.: Methods of estimation of relations of: equivalence, tolerance, and preference in a finite set. IBS PAN, Series: Systems Research, Warsaw, vol. 69 (2011)Google Scholar
10. Klukowski, L.: Determining an estimate of an equivalence relation for moderate and large sized sets. Oper. Res. Decis. Q. 27(2), 45–58 (2017)
11. Klukowski, L.: Statistical tests to verification of estimate of the preference relation resulting from pairwise comparisons. In: The paper presented on the Seventeenth International Workshop on Intuitionistic and Fuzzy Sets and Generalized Nets, Warsaw, 27–28 September 2018, the text in the same volume (2018)Google Scholar
12. Slater, P.: Inconsistencies in a schedule of paired comparisons. Biometrika 48, 303–312 (1961)