Selections from ordered sets

Abstract

We study the problem of evaluating whether the selection from a set is close to the ordering of the set determined by an exogenously given measure. Our main result is that three axioms, two naturally capturing “dominance”, and a stronger one imposing a form of symmetry in the comparison of selections, are sufficient to evaluate how close any selection from any set is to the given ordering of the set. This closeness is given by a very simple index, which is a linear function of the sum of the ranks of the selected elements. The paper ends by relating this index to the existing literature on distance between orderings, and also offers a practical application of the index.

1 Introduction

Selections are typically made according to a varying blend of objective measures and subjective judgments. A sport coach might base her team choice on the recent performance of individuals in her squad (batting and bowling/pitching averages, tennis rankings, trial results for track and field, and so on), and on her sense of who is the best person for each given role, given the expected conditions. Many universities do not strictly follow SATs results and school grades when choosing whom to admit, but take into account a student’s social background and his potential contribution to desirable characteristics of the student body, like diversity. Applicants for academic jobs might be ranked according to bibliometric measures, but the appointment panel’s judgment often leads to decision that do not map precisely into the ranking.¹ Large and complex procurement contracts often demand the subtle evaluation of complex qualitative elements, and lowest price is seldom the only criterion used to award these contracts.² And so forth and so on.

Observers and decision makers might be interested in some means of comparing the choices of different selectors. If the property of following the measurable dimension is labelled “orderliness”, someone might want to compare the orderliness of different selections, for example to determine which is more “orderly”.

We can think of at least three conceptually distinct situations where this comparison is meaningful.³ First, we may want to compare different selections from the same set. For example, the public may want to know how close the proposed shortlists of the different judges of a book or film prize are to a market-determined (sales, box office earnings) ordering. Second, the comparison might be of selections from altogether different sets. A cricket fan may want to know whether Australia’s selection for the Ashes team is closer to the players’ current rankings than England’s. A university whose admission policies are under scrutiny in court may want to argue that its admission policy is as based on SATs as those of comparable institutions. Or a government minister concerned about corruption in procurement contracts or personnel hiring may want to compare the orderliness of various commissioning boards or hiring panels, to identify and perhaps investigate atypical behaviour. On a broader scale, orderliness is a helpful yardstick when assessing inequality of opportunity: it can serve as a measure of nepotism, telling how close selection into society’s elite is to the pecking order determined by family history, or of plutocracy, measured for example by the closeness of membership of Parliament to a person’s position in the income distribution. The third kind of potential comparisons is for situations where both the selection and the selectors are the same, but there are two or more ways of ordering the elements of a set. In these cases, which are close to the topic of some related literature considered below (Kemeny 1959; Klamler 2008), the focus is on their relative importance in the selection. For example, a rugby analyst may want to know whether weight or speed is more important to be selected as a three-quarter for the Springboks. Or an external funder may want to know which of teaching and research is more important for promotion in a given university.

Comparing selections is straightforward only in the starkest cases. Sure, the selection of the best ranked is unquestionably more orderly than selections that do not. But is a university which, from its 102 applicants, admits as students those ranked according to their SAT scores second to 5th, 8th, 11th, 12th, 24th, 25th, 38th, and 47th more orderly than one that chooses the first 20, the 57th, and those ranked between 170 and 180 out of 500 applicants? Or, in an even simpler example, is picking the second ranked out of ten candidates for a job more orderly than selecting the third ranked out of 19?⁴

This paper takes an axiomatic approach to comparing selections. The axioms are requirements of orderliness: a decision maker who is called to opine which is more orderly of two selections from different sets, that is which selection is in her opinion closer to the order—determined by a given measure—of the set from which it is taken, should only make judgements that obey these axioms. This is conceptually similar to the approach taken by Sen (1976) and Cowell and Kuga (1981) in regard to judgements on which of two groups of individuals poverty or inequality, respectively, is greater.

We begin with three natural axioms, which extend the dominance principle in the comparison of sets (Barberà et al. 2004, BBP in what follows). The first requires that when an element is replaced in the selection with a better one, then the selection is deemed to be more orderly. In the next two axioms we depart from the set-up of the literature reviewed in BBP, and in some of its extensions, such as Rubinstein and Salant (2006). The departure is that we consider elements that are not in the original set, thus allowing the comparison of selections from different sets, as in many of the examples given above. These axioms require that if a new element appears which is better than all the selected (non-selected) ones, then the selection is considered more (less) orderly if the new element is selected. Analogously for the appearance of a worse element. Extending the axioms to different sets implies that we cannnot, like the literature cited, take the rank of the elements as the starting point of the analysis, since a new elements change the ranks of all the elements below it. Instead, we need to make formally explicit the mapping from the measured performance to the rank of the elements ordered by this measure.

These three axioms, which we show to be independent, are not characterising: there are different orderings of selections from sets which satisfies them. In our main result we show that characterisation is achieved by replacing the first axiom with a stronger one, which we label “mirror invariance”. This requires that if a change makes a selection more orderly, then the mirror image of the selection—that is the selection of only the non-selected elements—is made less orderly by the mirror image of the change. This is equivalent to requiring that the importance attributed to changes in the ordering be independent of where they occur: departures from the ranking are as important if they occur among the best or the worst ranked elements. This result is established in Theorem 1, which shows that the orderliness relation characterised by Axioms 2–4 is represented, in close analogy to the representation of a consumer’s preference by a utility function, by an “index of orderliness”, given by a simple function of the number of elements in the set, the number of selected elements, and the sum of the ranks of the selected elements.

The paper continues by showing that our index of orderliness has a simple relation with the concept of distance between rankings originally proposed by Kendall (1938), and given an axiomatic foundation by Kemeny (1959). This distance compares different selections from the same set, and, in this case, Proposition 4 shows that if a selection is more orderly than another selection from the same set, then the ranking of the set naturally induced by the first selection is nearer, in the sense of the Kendall-Tau distance, to the ordering of the set.

The paper is organised as follows: the dominance axioms and some preliminary results are in Sect. 2. The core of the paper is Sect. 3, which presents the index of orderliness, and strengthens one of the axioms to obtain a complete ordering of all selected sets. The relation with existing literature is explored in Sect. 4, and the paper ends with Sect.  5, which shows how the index can be used to assess the evaluation mechanism for promotion to professorship in Italian universities. A brief conclusion follows.

2 Axioms of “orderliness”

Let $\mathcal {N}\varsubsetneq \mathbb {R}$ contain $N\in \mathbb {N}\backslash \left\{ 1\right\}$ distinct elements: $x,y\in \mathcal {N}$ implies $x\ne y$ .⁵ The inequality relation in $\mathbb {R}$ induces a transitive, complete and antisymmetric relation between the members of $\mathcal {N}$. This relation is naturally defined as the rank in $\mathcal {N}$, $\rho _{\mathcal {N}}$, a bijective mapping of the elements of $\mathcal {N}$ into the set of the first N natural numbers $\rho _{\mathcal {N}} : \mathcal {N}\longrightarrow \{ 1,2,\ldots ,N\}$, which satisfies $\rho _{\mathcal {N}}( x) <\rho _{\mathcal {N}} ( y)$ if and only if $x<y$. We choose this convention, that x has a better⁶ rank in $\mathcal {N}$ than y if it is smaller, for definiteness. It would refer for example, to a situation where the real numbers are marathon runners’ personal best times; the changes needed to apply the framework to situations where a larger real number is better, such as cricketers’ career runs or scientists’ citation counts, are obvious.

We next define a selection $\mathcal {K}$ as a proper and non-empty subset of $\mathcal {N}$, that is $\mathcal {K}\varsubsetneq \mathcal {N}$, and $\mathcal {K} \ne \varnothing$. Let $K\in \left\{ 1,\ldots ,N-1\right\}$ be the number of elements of $\mathcal {K}$. We define the pair $\left( \mathcal {N},\mathcal {K} \right)$ a “selected set”.⁷ Let $\mathscr {S}$ be the family of selected sets.

Given a selected set $( \mathcal {N},\mathcal {K}) \in \mathscr {S}$ , there is a natural mapping induced by $\rho _{\mathcal {N}}$ from $\mathscr {S}$ into the set $\mathfrak {S}$ defined as

$$\begin{aligned} \mathfrak {S}=\left( \bigcup \limits _{N\in \mathbb {N}\backslash \{ 1\} }(N,2^{\{ 1,\ldots ,N\} }\backslash \{ 1,\ldots ,N\} )\right) \backslash \varnothing \text {,} \end{aligned}$$

that is as the union, over the natural numbers greater than 1, of the cartesian product between N and the power set of $\{ 1,\ldots ,N\}$ , with the set $\{ 1,\ldots ,N\}$ itself removed. The generic element of the set $\mathfrak {S}$ is therefore $( N,\{ i_{1},\ldots ,i_{K}\} )$, where $i_{k}\in \{ 1,\ldots ,N\}$. Define this map as:

$$\begin{aligned} m:\mathscr {S}\longrightarrow \mathfrak {S}, \end{aligned}$$

such that for $( \mathcal {N},\mathcal {K}) \in \mathscr {S}$, where $\mathcal {N}$ is a set with N elements, from which K are selected, $m ( \mathcal {N},\mathcal {K})$ is the pair given by the cardinality of $\mathcal {N}$ and the subset of $\{ 1,\ldots ,N \}$ which are the ranks in $\mathcal {N}$ of the elements of $\mathcal {K}$:

$$\begin{aligned} m:( \mathcal {N},\mathcal {K}) \longmapsto ( N,\{ i\in \{ 1,\ldots ,N\} |\rho _{\mathcal {N}}^{-1}( i) \in \mathcal {K}\}). \end{aligned}$$

Recall that, because the elements of $\mathcal {N}$ are different, $\rho _{\mathcal {N}}$ is invertible. Given a selected set $( \mathcal {N},\mathcal {K})$ such that $m( \mathcal {N},\mathcal {K}) =( 15,\{ 2,3,8,12\} )$, implying that the second, the third, the 8th and the 12th ranked out of the set’s 15 elements are selected, we may represent its image under m as follows:

$$\begin{aligned} ( \mathcal {N},\mathcal {K}) \longrightarrow 011000010001000. \end{aligned}$$

(1)

Elements are ranked from best to the left, to worst to the right, and a “1” in the j-th position indicates that the j-th element is selected.

We single out the situations where the selection follows exactly the ranking.

Definition 1

Given a selected set $( \mathcal {N},\mathcal {K}) \in \mathscr {S}$ , the selection $\mathcal {K}$ is “perfect ” if $x\in \mathcal {K}$ and $y\in \mathcal {N}$ $\backslash \mathcal {K}$ implies $x<y$. The selection $\mathcal {K}$ is “antiperfect” if $x\in \mathcal {K}$ and $y\in \mathcal {N}$ $\backslash \mathcal {K}$ implies $y<x$.

In words, the selection $\mathcal {K}$ is perfect if no selected element has a rank worse than a non-selected element, and it is antiperfect if every selected element has a worse rank than every non-selected element.

Let $\mathscr {S}^{P}\subseteq \mathscr {S}$ be the set of all perfect selections, and conversely, let $\mathscr {S}^{A}\subseteq \mathscr {S}$ be the set of all antiperfect selections. Note that since $\mathcal {K}$ is a proper non-empty subset of $\mathcal {N}$, $\mathscr {S}^{P}\cap \mathscr {S} ^{A}= \varnothing$: no selection is simultaneously both perfect and antiperfect.

We want to compare selections, that is order the set $\mathscr {S}$, in the following sense: consider two ranked sets $\mathcal {N}_{A}$ with $N_{A}$ elements, and $\mathcal {N}_{B}$ with $N_{B}$ elements, and selections $\mathcal {K}_{A}$ and $\mathcal {K}_{B}$ from $\mathcal {N}_{A}$ and $\mathcal {N }_{B}$, respectively. We want to answer the question: is the selection $\mathcal {K}_{A}$ from $\mathcal {N}_{A}$ closer to or farther from the ranking of set $\mathcal {N}_{A}$ than the selection $\mathcal {K}_{B}$ from $\mathcal {N}_{B}$ is to the ranking of set $\mathcal {N}_{B}$? In terms of the examples given above, we want to know whether, say, a journalist’s choice of the “all time England cricket test team” is more based on the players’ record than an Italian chemistry academy’s choice of the members of its scientific committee is based on the academics’ citation count.

To formalise this question we define a binary relation $\mathfrak {M} \subseteq \mathfrak {S}\mathcal {\times \mathfrak {S}}$, which we interpret as “orderliness”: orderliness is the property of being close to the ordering of the set.

Definition 2

Given two selected sets $( \mathcal {N}_{A},\mathcal {K} _{A}) , ( \mathcal {N}_{B},\mathcal {K}_{B}) \in \mathscr {S}$, $( \mathcal {N}_{A},\mathcal {K}_{A})$ is at least as orderly as $( \mathcal {N}_{B},\mathcal {K}_{B})$ if and only if $(m( \mathcal {N}_{A},\mathcal {K}_{A}) ,m( \mathcal {N}_{B},\mathcal {K} _{B}) )\in \mathfrak {M}$.

We are thus defining equivalence classes in $\mathscr {S}$: two selected sets $( \mathcal {N}_{A},\mathcal {K}_{A}) ,( \mathcal {N }_{B},\mathcal {K}_{B}) \in \mathscr {S}$ are in the same equivalence class if and only if $m ( \mathcal {N}_{A},\mathcal {K}_{A}) =m ( \mathcal {N}_{B},\mathcal {K}_{B})$, that is if the sets $\mathcal {N}_{A}$ and $\mathcal {N}_{B}$ have the same number of elements, and the ranks of the selected elements are the same in the selected sets $( \mathcal {N}_{A},\mathcal {K}_{A})$ and $( \mathcal {N}_{B}, \mathcal {K}_{B})$. In particular, the image of $(\{ x_{1},\ldots ,x_{N}\} ,\{ x_{i_{1}},\ldots ,x_{i_{K}}\} )$ is the same as the image of any set where $x_{j}$ is replaced by any x in an interval of the real line containing $x_{j}$ but no other element of $\left\{ x_{1},\ldots ,x_{N}\right\}$. Thus Definition 2, accordingly, defines orderliness as a relation on the classes of equivalence in $\mathscr {S}$.

We follow the standard convention used to describe preferences, and write $( \mathcal {N}_{A},\mathcal {K}_{A}) \succsim _{\mathfrak {M}} ( \mathcal {N}_{B},\mathcal {K}_{B})$ when $( \mathcal {N}_{A}, \mathcal {K}_{A})$ is at least as orderly as $( \mathcal {N}_{B}, \mathcal {K}_{B})$. “Strict orderliness”, $\succ _{\mathfrak {M}}$, and “equal orderliness”, $\sim _{\mathfrak {M}}$, are naturally defined: $( \mathcal {N}_{A},\mathcal {K}_{A})$ is strictly more orderly than $( \mathcal {N}_{B},\mathcal {K}_{B})$ if $( \mathcal {N}_{A},\mathcal {K}_{A}) \succsim _{\mathfrak {M} } ( \mathcal {N}_{B},\mathcal {K}_{B})$ and not $( \mathcal {N} _{B},\mathcal {K}_{B}) \succsim _{\mathfrak {M}} ( \mathcal {N}_{A}, \mathcal {K}_{A})$. And $( \mathcal {N}_{A},\mathcal {K}_{A})$ and $( \mathcal {N}_{B},\mathcal {K}_{B})$ are equally orderly if $( \mathcal {N}_{A},\mathcal {K}_{A}) \succsim _{\mathfrak {M} } ( \mathcal {N}_{B},\mathcal {K}_{B})$ and $( \mathcal {N}_{B}, \mathcal {K}_{B}) \succsim _{\mathfrak {M}} ( \mathcal {N}_{A}, \mathcal {K}_{A})$.

We require the orderliness relation $\mathfrak {M}$ to be reflexive, so that all selected sets with the same image are equally orderly, complete and transitive. The assumption of completeness is a strong one, though it is necessary to ensure that orderliness can have operational value in practice, as it ensures that there are no selections which are “not comparable”. Relatively simple examples show that it is in principle arbitrary to construct such a complete ordering. Consider two selected sets $( \mathcal {N}_{A},\mathcal {K}_{A}) ,( \mathcal {N}_{B},\mathcal {K}_{B}) \in \mathscr {S}$, such that $m ( \mathcal {N}_{A},\mathcal {K}_{A}) =( 35 , \{ 2,5,6,7,10,16,24,28 \} )$ and $m ( \mathcal {N} _{B},\mathcal {K}_{B}) =( 21, \{ 3,6,8,10,11\} )$. These can be represented graphically as:

$$\begin{aligned} \left( \mathcal {N}_{A},\mathcal {K}_{A}\right)\longrightarrow & {} 01001110010000010000000100010000000\text {,} \\ \left( \mathcal {N}_{B},\mathcal {K}_{B}\right)\longrightarrow & {} 001001010110000000000\text {.} \end{aligned}$$

In $( \mathcal {N}_{A},\mathcal {K}_{A})$, there are some selected elements among the best ranked, but there are also some below the median. In the second, from a smaller set, no selected element is below the median, but many are close to it: different observers might well have different views as to which of the two above selections is more orderly. As we show in this paper, this arbitrariness is fully resolved if one accepts the simple axioms we propose.

Even with completeness, without any further restrictions, the orderliness relation can still be vacuous: for example, the relation $\mathfrak {M}= \mathfrak {S}\mathcal {\times \mathfrak {S}}$, where all selected sets are equally orderly, is transitive, reflexive and complete, and so it can well represent a person’s opinion of the orderliness of selections. In the rest of the paper, therefore, we impose some further requirements. As in BBP, they are natural restrictions on $\mathfrak {M}$, the relation between the images in $\mathfrak {S}$ of the sets $( \mathcal {N}, \mathcal {K})$. BBP (p 904) begin by imposing the “extension rule”: given $x,y\in \mathcal {N}$, then $m ( \mathcal {N}, \{ x \} ) \succsim _{\mathfrak {M}}m ( \mathcal {N}, \{ y \} )$ if and only if $x<y$. We strengthen the extension rule, requiring that the orderliness comparison be strict, and that it applies to any set, not just singletons.

Axiom 1

(Swap-Dominance) For all $\left( \mathcal {N},\mathcal {K }\right) \in \mathscr {S}$ and $x\in \mathcal {N}\backslash \mathcal {K}$ and $y\in \mathcal {K}$, $\left( \mathcal {N},\mathcal {K\cup }\left\{ x\right\} \backslash \left\{ y\right\} \right) \succ _{\mathfrak {M}}\left( \mathcal {N}, \mathcal {K}\right)$ if and only if $x<y$.

In words, Axiom 1 requires that the swap between an element in the selection and an element not in the selection makes the selected set strictly more (less) orderly if the rank of the newly selected element is better (worse) than the rank of the removed element.⁸ Note that Axiom 1 is incompatible with the independence axiom (BBP p 905), which, in our framework, would require that, given selections $( \mathcal {N},\mathcal {K_{A}}) , ( \mathcal {N},\mathcal {K_{B}} ) \in \mathscr {S}$ and $x\in \mathcal {N}\backslash ( \mathcal { K_{A}}\cup \mathcal {K_{B}})$, $( \mathcal {N},\mathcal {K_{A}} ) \succ _{\mathfrak {M}} ( \mathcal {N},\mathcal {K_{B}})$ implies $( \mathcal {N},\mathcal {K_{A}}\cup \{ x \} ) \succsim _{\mathfrak {M}} ( \mathcal {N},\mathcal {K_{B}}\cup \{ x \} )$. As BPP note, this axiom rules out “certain types of complementarities” (p 906), and runs therefore contrary to the motivation of the paper, which views selections in their entirety. In fact, BBP show that independence is a strong requirement in this context, as it prevents comparison between selections, except in very special cases (BBP, pp 910–922).

The next two Axioms we impose are the natural extension of the idea of dominance, which “requires that adding an element which is better (worse) than all elements in a given set A [...] leads to a set that is better (worse) than the original set” (BBP, p 905). In our more complex set-up, we want to compare subsets selected from different sets, and therefore we state the axioms as binary comparisons between sets with different number of elements.

Axiom 2

(Better-Dominance) For all $( \mathcal {N},\mathcal {K }) \in \mathscr {S}$ and $x\notin \mathcal {N}$ such that $x<y$ for all $y\in \mathcal {K}$:

• i. $( \mathcal {N\cup }\{ x\} , \mathcal {K\cup }\{ x\} ) \succsim _{\mathfrak {M}} ( \mathcal {N},\mathcal {K})$; strictly unless $( \mathcal {N}, \mathcal {K}) \in \mathscr {S}^{P}$.

• ii. $( \mathcal {N},\mathcal {K}) \succsim _{\mathfrak {M}} ( \mathcal {N\cup } \{ x\} ,\mathcal {K} )$; strictly unless $( \mathcal {N},\mathcal {K}) \in \mathscr {S}^{A}$.

Suppose a new element is added⁹ to the set $\mathcal {N}$, and this new element has better rank than every selected element. Then Axiom 2 requires that, if this new element is selected, the selection becomes more orderly (Axiom 2.i); if it is not selected, the selection becomes less orderly (Axiom 2.ii).

Axiom 3

(Worse-Dominance) For all $( \mathcal {N},\mathcal {K} ) \in \mathscr {S}$ and $x\notin \mathcal {N}$ such that $y<x$ for all $y\in \mathcal {K}$:

• i. $( \mathcal {N\cup }\{ x\} , \mathcal {K}) \succsim _{\mathfrak {M}} ( \mathcal {N},\mathcal {K} )$; strictly unless $( \mathcal {N},\mathcal {K}) \in \mathscr {S}^{P}$.

• ii. $( \mathcal {N},\mathcal {K}) \succsim _{\mathfrak {M}} ( \mathcal {N\cup } \{ x \} ,\mathcal { K\cup } \{ x \} )$; strictly unless $( \mathcal {N}, \mathcal {K}) \in \mathscr {S}^{A}$.

Axiom 3 is the converse of Axiom 2 at the other end of the ranking of the set $\mathcal {N}$: if a new element is added to the set $\mathcal {N}$ which is worse ranked than every selected element, then the selection becomes more (respectively, less) orderly if this new element is not selected, Axiom 3.i (respectively, is selected, Axiom 3.ii).

To see these axioms “in action”, consider the selected set represented graphically in (1). Axioms 2 and 3 are illustrated by the following changes in the selected set, where the new element is boxed.

Axiom	New element in set (1)	Selected Y/N?	New selected set	More/less orderly?
2.i	Better than every selected	Yes		More
2.ii	Better than every selected	No		Less
3.ii	Worse than every selected	Yes		Less
3.i	Worse than every selected	No		More

In the first and fourth rows the selection becomes more orderly, and in the second and third less so. In the first two rows the new element can indifferently be in the first or second position of the ranking; and in the third and fourth row, in any of the bottom four positions.

The selection (1) is neither perfect nor antiperfect, and so all the comparisons between it and the selected sets in the above table are strict.

The three axioms are independent: it is possible to define different orderliness relations such that one Axiom is violated but the other two are satisfied. To see this, consider an orderliness relation, $\widehat{\mathfrak {M}}$, which, given a selected set $( \mathcal {N},\mathcal {K})$ and an element $x\notin \mathcal {N}$, requires $( \mathcal {N},\mathcal {K}) \succ _{\widehat{\mathfrak {M}}}( \mathcal {N}\cup \{ x \} ,\mathcal {K}\cup \{ x \} )$ if there exists $z\in \mathcal {N}\backslash \mathcal {K}$ such that $x<z<y$ for all $y\in \mathcal {K}$. In word, in order for a new element to make the selection more orderly, it must be better than at least one non-selected element in $\mathcal {N}$ which is better than all the selected elements. Clearly $\widehat{\mathfrak {M}}$ violates Axiom 2, and obeys Axioms 1 and 3. A symmetric argument holds in respect of Axiom 3. Lastly, consider an orderliness relation, $\widetilde{\mathfrak {M}}$, which, given a selected set $( \mathcal {N},\mathcal {K})$ and any two elements $x\in \mathcal {N}\backslash \mathcal {K}$ and $y\in \mathcal {K}$, requires $( \mathcal {N},\mathcal {K}) \succ _{\widetilde{\mathfrak {M}}} ( \mathcal {N},\mathcal {K}\cup \{ x \} \backslash \{ y \} )$ if there exists $z\in \mathcal {N}$, such that $x<z<y$. That is, in order to make a selection more orderly, a swap must be between elements which are separated by at least one element in $\mathcal {N}$. $\widetilde{\mathfrak {M}}$ violates Axiom 1, and obeys Axioms 2 and 3.

Axioms 2 and 3 dictate the relative orderliness of two selected sets where the ranked sets, $\mathcal {N}$ and $\mathcal {N\cup } \left\{ x\right\}$, differ in size by 1. The dominance axioms, labelled by BBP (p 905) the Gärdenfors principle (1976), impose an ordering on selections from a given ranked set, therefore they apply to sets of the same size. We derive them in our framework as immediate consequences of Axioms 2 and 3 in the following corollary.

Corollary 1

• i. For all $( \mathcal {N},\mathcal {K}) \in \mathscr {S}$ and $x\in \mathcal {N}\backslash \mathcal {K}$ such that $x<y$ for all $y\in \mathcal {K}$, $( \mathcal {N},\mathcal {K\cup } \{ x \} ) \succsim _{\mathfrak {M}} ( \mathcal {N},\mathcal {K})$; strictly unless $( \mathcal {N},\mathcal {K}) \in \mathscr {S}^{A}$.

• ii. for all $\left( \mathcal {N},\mathcal {K}\right) \in \mathscr {S}$ and $x\in \mathcal {K}$ such that $x<y$ for all $y\in \mathcal {K} \backslash \left\{ x\right\}$, $\left( \mathcal {N},\mathcal {K}\right) \succsim _{\mathfrak {M}}\left( \mathcal {N},\mathcal {K}\backslash \left\{ x\right\} \right)$; strictly unless $\left( \mathcal {N},\mathcal {K}\right) \in \mathscr {S}^{P}$.

• iii. For all $\left( \mathcal {N},\mathcal {K}\right) \in \mathscr {S}$ and $x\in \mathcal {K}$ such that $y<x$ for all $y\in \mathcal {K} \backslash \left\{ x\right\}$, $\left( \mathcal {N},\mathcal {K} \backslash \left\{ x\right\} \right) \succsim _{\mathfrak {M}}\left( \mathcal {N} ,\mathcal {K}\right)$; strictly unless $\left( \mathcal {N},\mathcal {K} \right) \in \mathscr {S}^{P}$.

• iv. for all $\left( \mathcal {N},\mathcal {K}\right) \in \mathscr {S}$ and $x\in \mathcal {N}\backslash \mathcal {K}$ such that $y<x$ for all $y\in \mathcal {K}$, $\left( \mathcal {N},\mathcal {K}\right) \succsim _{ \mathfrak {M}}\left( \mathcal {N},\mathcal {K\cup }\left\{ x\right\} \right)$; strictly unless $\left( \mathcal {N},\mathcal {K}\right) \in \mathscr {S}^{A}$.

Proof

We only establish the first claim, as the proof of the remainder is essentially identical. Consider a selected set $\left( \mathcal {N},\mathcal {K }\right)$, and an $x\in \mathcal {N}\backslash \mathcal {K}$ such that $x<y$ for all $y\in \mathcal {K}$. Note that this implies that $\left( \mathcal {N}, \mathcal {K}\right) \notin \mathscr {S}^{P}$. Take $z\notin \mathcal {N}$, such that $z<y$ for all $y\in \mathcal {K}$, and xRz (that is z is better than every selected element, but worse than x). By Axiom 2.ii, $\left( \mathcal {N},\mathcal {K}\right) \succ _{\mathfrak {M}}\left( \mathcal {N} \cup \left\{ z\right\} ,\mathcal {K}\right)$. Next note that, by Axiom 3.i, $\left( \mathcal {N}\cup \left\{ z\right\} ,\mathcal {K} \right) \succ _{\mathfrak {M}}\left( \mathcal {N}\cup \left\{ z\right\} \backslash \left\{ x\right\} ,\mathcal {K}\right)$. Finally, notice that $m\left( \mathcal {N}\cup \left\{ z\right\} \backslash \left\{ x\right\} , \mathcal {K}\right) =m\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {K} \right)$, since an excluded element better than all selected elements is replaced by another, and so $\left( \mathcal {N},\mathcal {K}\right) \succ _{ \mathfrak {M}}\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {K}\right)$. $\square$

Intuitively, if a non-selected element that is better ranked than every selected element were instead selected, the selection would become more orderly (Corollary 1.i), and if the best element in the selection were removed, the selection would become less orderly (Corollary 1.ii). Conversely, consider an element in $\mathcal {N}$ which is worse ranked than every selected element: if it is removed from the selection, then the selection becomes more orderly (Corollary 1.iii); if it is added to the selection, then the selection becomes less orderly (Corollary 1.iv).

While Corollary 1 is an obvious consequence of Axioms 2 and 3, the next result is less immediate, and shows that, although Axioms 1–3 may seem weak, they do have some bite, as can be inferred from one of their implications on the comparison of “extreme” selections from different sized sets.

Proposition 1

• i. Let $( \mathcal {N}_{p_{1}},\mathcal {K} _{p_{1}}) , ( \mathcal {N}_{p_{2}},\mathcal {K}_{p_{2}}) \in \mathscr {S}^{P}$. Then $( \mathcal {N}_{p_{1}},\mathcal {K} _{p_{1}}) \sim _{\mathfrak {M}} ( \mathcal {N}_{p_{2}},\mathcal {K} _{p_{2}})$.

• ii. Similarly, let $( \mathcal {N}_{a_{1}},\mathcal {K}_{a_{1}}) ,( \mathcal {N}_{a_{2}},\mathcal {K}_{a_{2}}) \in \mathscr {S}^{A}$. Then $( \mathcal {N}_{a_{1}},\mathcal {K}_{a_{1}}) \sim _{\mathfrak { M}} ( \mathcal {N}_{a_{2}},\mathcal {K}_{a_{2}})$.

• iii. Let $( \mathcal {N},\mathcal {K}) \in \mathscr {S}\backslash \mathscr {S}^{P}\backslash \mathscr {S}^{A}$; let $( \mathcal {N}_{p}, \mathcal {K}_{p}) \in \mathscr {S}^{P}$; let $( \mathcal {N}_{a}, \mathcal {K}_{a}) \in \mathscr {S}^{A}$. Then $( \mathcal {N}_{p}, \mathcal {K}_{p}) \succ _{\mathfrak {M}}( \mathcal {N},\mathcal {K} )$ and $( \mathcal {N},\mathcal {K}) \succ _{\mathfrak {M} }( \mathcal {N}_{a},\mathcal {K}_{a})$.

Proof

Let us begin with (i). Consider a perfect selection of $K_{p}$ elements from a set $( \mathcal {N}_{p},\mathcal {K}_{p})$, with $( \mathcal {N}_{p},\mathcal {K}_{p}) \in \mathscr {S}^{P}$. Clearly the best ranked $K_{p}$ elements are in $\mathcal {K}_{p}$, the rest in $\mathcal {N}_{p}\backslash \mathcal {K}_{p}$. Now add to the set and to the selection an element $z_{1}$ such that $z_{1}<x$ for all $x\in \mathcal {K} _{p}$. By Axiom 2.i, $( \mathcal {N}_{p},\mathcal {\cup }\{ z_{1}\} ,\mathcal {K}_{p}\mathcal {\cup } \{ z_{1}\} ) \succsim _{\mathfrak {M}} ( \mathcal {N}_{p},\mathcal {K}_{p})$. Next return to the selection $( \mathcal {N}_{p},\mathcal {K} _{p})$, and add, again to the set and to the selection, an element $z_{2}$ such that $x<z_{2}$ for all $x\in \mathcal {K}_{p}$, and $z_{2}<y$ for all $y\in \mathcal {N}_{p}\backslash \mathcal {K}_{p}$ (that is $z_{2}$ is worse ranked than every selected element, but better ranked than every non-selected element). By Axiom 3.ii, $( \mathcal {N} _{p},\mathcal {K}_{p}) \succsim _{\mathfrak {M}} ( \mathcal {N}_{p}, \mathcal {\cup } \{ z_{2} \} ,\mathcal {K}_{p}\mathcal {\cup } \{ z_{2} \} )$. But now notice that both new selections select only the best ranked $K_{p}+1$ elements: hence, $m ( \mathcal {N}_{p},\mathcal { \cup }\{ z_{1}\} ,\mathcal {K}_{p}\mathcal {\cup }\{ z_{1}\} ) =m( \mathcal {N}_{p},\mathcal {\cup }\{ z_{2}\} ,\mathcal {K}_{p}\mathcal {\cup }\{ z_{2}\} )$ . By transitivity, these both equal $m ( \mathcal {N}_{p},\mathcal {K} _{p})$. The process can be repeated to show that all perfect selections are equally orderly. The proof of (ii) is identical. Consider (iii) next. Take a selection $( \mathcal {N},\mathcal {K})$ with $( \mathcal {N},\mathcal {K}) \in \mathscr {S}\backslash \mathscr {S} ^{P}\backslash \mathscr {S}^{A}$. Let N and K be the number of elements in $\mathcal {N}$ and $\mathcal {K}$. Now let $x_{1}\in \mathcal {K}$ be the best ranked selected element such that there is a non-selected element $y\in \mathcal {N}\backslash \mathcal {K}$ such that $y<x_{1}$. Because the selection is not perfect, it is possible to find such a $x_{1}$. Next let $y_{1}\in \mathcal {N}\backslash \mathcal {K}$ be the best ranked non-selected element such that there a selected element $x\in \mathcal {K}$ such that $y_{1}<x$. Again, because the selection is not antiperfect, it is possible to find such a $y_{1}$. Now by Axiom 1, $( \mathcal {N },\mathcal {K\cup }\{ x_{1}\} \backslash \{ y_{1}\} ) \succ _{\mathfrak {M}}( \mathcal {N},\mathcal {K})$. If the selection $( \mathcal {N},\mathcal {K\cup }\{ x_{1}\} \backslash \{ y_{1}\} )$ is perfect, then we are done. If not, we can repeat, until a perfect selection in reached. This happens in at most $\min \{ K,N-K\}$ steps, and establishes that $( \mathcal {N}_{p},\mathcal {K}_{p}) \succ _{\mathfrak {M}}( \mathcal {N},\mathcal {K})$. The demonstration that $( \mathcal {N}, \mathcal {K}) \succ _{\mathfrak {M}}( \mathcal {N}_{a},\mathcal {K} _{a})$ is identical. $\square$

In words, Proposition 1 says that all perfect selections are equally orderly, and similarly, that all antiperfect selections are equally orderly. The third statement asserts that every perfect selection is strictly more orderly than every non-perfect selection, and every antiperfect selection is strictly less orderly than every non-antiperfect selection. This is entirely reasonable when the selection is from the same set. Thus, selecting the best ten from a set of one hundred elements is clearly more orderly than selecting the best nine and the eleventh. However, as we want to extend the range of selected sets to be compared, it is possible to think of extreme situations when the size of the sets and of the selections is different, and when the argument that a perfect selection is more orderly is less clear-cut. Consider, for example, the following case: is selecting the better of the two elements of a set unquestionably more orderly than selecting the best 25 and the 27-th ranked out of a set with 10,000 elements? Proposition 1 answers unambiguously yes, but someone might argue in favour of the opposite, on the grounds that the former is more likely than the latter to have been determined by other criteria, which by chance happen to coincide with the rank, whereas the selection of the first 25 and the 27-th elements out of a very large set would almost surely be seen as the consequence of a determinate intention to use the rank as the criterion for selection, with the 26-th element being excluded due to some other criterion.

3 An index of “orderliness”

Definition 3

An index of orderliness is a function $M:\mathfrak {S}\longrightarrow \left[ -1 ,1\right]$ such that, given any two selected sets $\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) ,\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \in \mathscr {S}$, $\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \succsim _{\mathfrak {M}}\left( \mathcal {N}_{B},\mathcal {K}_{B}\right)$ if and only if $M\left( m\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \right) \geqslant M\left( m\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \right)$.

Note that we require the index to takes values ranging from $-1$ to 1, in the cases of an antiperfect and a perfect selection, respectively. Of course, a different normalisation would allow us to choose any two end points. However, in analogy with the measures of correlation, having 1 indicating perfect orderliness, and $-1$ its antithesis, where the selection perversely chooses the worst elements, does seem the natural choice. In analogy with the theory of consumers’ preferences, existence of the index of orderliness is ensured by the additional requirement that the orderliness relation is continuous. The formal proof is identical with the proof of the existence of a utility function; as with consumer theory, a lexicographic relation illustrates well the need of continuity to represent orderliness through an index. Consider the following way of comparing the orderliness of any two selected sets: take first the best ranked element in each: if only one selection includes it, then that selection is strictly more orderly than the other. Otherwise, look next at the second best ranked element, and again, if it is included only in one selection, this is the more orderly one. Again, if they are both or neither selected, go to the next ranked element and so on. If one selection “runs out of elements” before the other, then it is less orderly. In analogy with consumer preferences, this relation is reflexive, complete and transitive, but it is not continuous, and so it cannot be represented by an index of orderliness.¹⁰

To ensure existence of a orderliness index representing a relation $\mathfrak {M}$, therefore we require that the orderliness relation be continuous.¹¹

Uniqueness, on the other hand, is not ensured by the three axioms proposed: it is possible to find pairs of relations both satisfying the three axioms, which rank differently the orderliness of given selected sets, and therefore are represented by indices of orderliness which attach differently ordered values to given selected sets. To see this, consider the following two indices.

$$\begin{aligned} M\left( m \left( \mathcal {N},\mathcal {K}\right) \right) =\frac{ N+1 -\frac{2}{K}\sum _{x\in \mathcal {K}}\rho _{\mathcal {N}}\left( x\right) }{N-K} \text {,} \end{aligned}$$

(2)

and

$$\begin{aligned} M_{1}{\left( m \left( \mathcal {N},\mathcal {K} \right) \right) =}\frac{\frac{2K^{2}+1}{3}+N\left( N-K+1\right) -\frac{2}{K}\sum _{x\in \mathcal {K}}\rho _{\mathcal {N}}\left( x\right) ^{2}}{\left( N+1\right) \left( N-K\right) }\text {.} \end{aligned}$$

(3)

For future reference, it is convenient to denote the sum of the ranks of the selected elements as:

$$\begin{aligned} r=\sum _{x\in \mathcal {K}}\rho _{\mathcal {N}}\left( x\right) \text {.} \end{aligned}$$

(4)

Using (4), the expression in (2) can then be written as

$$\begin{aligned} M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }=\frac{r^{N,K}+r_{K}-2r}{r^{N,K}-r_{K}}\text {, } \end{aligned}$$

where $r_{K}$ and $r^{N,K}$ are the sum of the ranks when the best and the worst K elements are selected, respectively, and therefore they are given by: $r_{K} =\sum _{j=1}^{K}j$, and $r^{N,K} =\sum _{j=N-K+1}^{N}j$. Similarly for $M_{1}{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }$: it can be written as

$$\begin{aligned} M_{1}{\left( m \left( \mathcal {N},\mathcal {K} \right) \right) }=\frac{r_{1}^{N,K}+r_{1,K}-2\sum _{x\in \mathcal {K}} \rho _{\mathcal {N}}\left( x\right) ^{2}}{r_{1}^{N,K}-r_{1,K}}\text {, } \end{aligned}$$

where $r_{1,K}$ and $r_{1}^{N,K}$ are the sum of the squares of the ranks when the best and the worst K elements are selected, $r_{1,K} =\sum _{j=1}^{K}j^{2}$, and $r_{1}^{N,K} =\sum _{j=N-K+1}^{N}j^{2}$.

Proposition 2

A relation represented by the index $M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }$ satisfies Axioms 1–3.

Proof

Suppose a relation $\mathfrak {M}$ on $\mathfrak {S}\times \mathfrak {S}$ is given, which can be represented by the index (2), $M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }$. We begin by showing that $\mathfrak {M}$ satisfies Axioms 2 and 3. In each of these axioms, a new element z is added to the set $\mathcal {N}$. Therefore N increases by 1. What happens to $K\$and to the total rank r depends on which part of which Axiom is considered. In Axiom 2.i the new element z is in the selection, and so K also increases by 1, and as z is better than every selected element, the total rank increases by $\rho _{\mathcal {N}}\left( z \right) +K$: each of the K previously selected elements increases by 1, and the new element’s rank $\rho \left( z \right)$ is added to the total. The value of M therefore changes from

$$\begin{aligned} \frac{N+1 - \frac{2}{K}r}{N-K } \end{aligned}$$

(5)

to

$$\begin{aligned} \frac{N+2 -\frac{2}{K+1}\left( r+\rho _{\mathcal {N}}\left( z \right) + K\right) }{N-K}. \end{aligned}$$

The difference is

$$\begin{aligned} \frac{2\frac{r}{K}-2\rho _{\mathcal {N}}\left( z \right) - K+1}{\left( N-K\right) \left( K+1\right) }\text {,} \end{aligned}$$

(6)

which is increasing in r, and equals $\frac{2}{\left( N-K\right) \left( K+1\right) }>0$ when r takes its lowest possible value given $\rho _{\mathcal {N}}\left( z \right)$, $\sum _{j=\rho _{\mathcal {N}}\left( z \right) +1}^{\rho _{\mathcal {N}}\left( z \right) +K}j$. Thus it is positive for every feasible value of r.

If the new element is not selected, Axiom 2.ii, then K does not change, but the total rank of the selected elements increases by K, since the rank of every selected element increases by 1, and so the value of the index changes from (5) to

$$\begin{aligned} \frac{N - \frac{2}{K}r}{N+1-K }. \end{aligned}$$

The difference with (5) is $-\frac{2N-K+1-\frac{2}{K} r}{\left( N+1-K\right) \left( N-K\right) }$ which is increasing in r, and since it is 0 at the maximum value of r, $r^{N,K}=\frac{K\left( 2N-K+1\right) }{2}$, it is strictly negative for any other value of r, establishing the result.

Next consider Axiom 3. There is a new element in $\mathcal {N}$ which is worse than all the elements selected. In Axiom 3.i, the new element is not selected, and so neither K nor r change. In this case, the index (2) becomes

$$\begin{aligned} \frac{N+2 - \frac{2}{K}r}{ N+1-K }. \end{aligned}$$

The difference with its previous value, (5), is $-\frac{K+1-\frac{2}{K}r}{\left( N+1-K\right) \left( N-K\right) }$, which is increasing in r. Since it is 0 at the minimum value of r, which is $r_{K}=\frac{K\left( K+1\right) }{2}$, it is strictly positive for any other value of r, and so the index increases in this case. Finally Axiom 3.ii: the new element, worse than all those already selected, is itself selected. Thus K increases by 1 and r by $\rho _{\mathcal {N}}\left( z\right)$, and so index (2) becomes

$$\begin{aligned} \frac{N+2 - \frac{2}{K+1}\left( r+\rho _{\mathcal {N}}\left( z\right) \right) }{N-K }, \end{aligned}$$

and the difference with (5) is $\frac{K+1 -\rho _{\mathcal {N}}\left( z\right) +\frac{2}{K}r}{\left( N-K\right) \left( K+1\right) }$, increasing in r and taking, for given $\rho _{\mathcal {N}}\left( z\right)$, its maximum value, 0, at $r=\sum _{j=\rho _{\mathcal {N}}\left( z\right) -K}^{\rho _{\mathcal {N}}\left( z\right) -1}j=K\big ( \rho _{\mathcal {N}}\left( z\right) -\frac{1+K}{2}\big )$.

Consider finally Axiom 1, which is straightforward: the swap between an element in the selection and an element not in the selection changes neither K nor N. It only changes r, and so clearly the index M increases if r decreases, that is if a better ranked element takes the place of a worse ranked one in the selection. This establishes that the index (2) satisfies all the Axioms and completes the proof. $\square$

Table 1 summarises the proof of Lemma 2, by presenting a schematic description of the effects of adding a new element to a selected set, and of the effects which Axioms 2 and 3 require on the value of the index: in all cases, N increases by 1, and K, r, and $M\left( \mathcal {N},\mathcal {K}\right)$ are the original values of the size of the selection, of the sum of the ranks, and of the index of orderliness.

Table 1 Summary of the proof of Proposition 2

The analogous of Proposition 2 for the index (3) holds, but its proof is essentially identical to the above proof and is omitted.

Proposition 3

A relation represented by the index $M_{1}{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }$ satisfies Axioms 1–3.

Now consider the following two selected sets, $\left( \mathcal {N_{A}},\mathcal {K}_{A}\right)$ and $\left( \mathcal {N_{B}},\mathcal {K}_{B}\right)$ such that $m\left( \mathcal {N_{A}},\mathcal {K}_{A}\right) = \left( 11 , \left\{ 2,3,5,8,9 \right\} \right)$ and $m\left( \mathcal {N_{B}},\mathcal {K}_{B}\right) = \left( 11 , \left\{ 1,2,3,8,11 \right\} \right)$. Graphically, these differ only in the selection of the elements in the boxes:

We have $M{\left( m \left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \right) } < M{\left( m \left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \right) }$ and $M_{1}{\left( m \left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \right) }> M_{1}{\left( m \left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \right) }$: if the relation of orderliness is represented by index (2), then the second is more orderly, and vice versa, if orderliness is described by a relation represented by index (3) then the first is more orderly. Thus both indices M and $M_{1}$ satisfy Axioms 1–3, and yet they give a different answer to the question of the relative orderliness of the two sets: Axioms 1–3 are not characterising.

The restriction required to ensure characterisation is remarkably simple, as the rest of the paper shows. We replace Axiom 1 with the following.

Axiom 4

(Mirror invariance) For all $\left( \mathcal {N} ,\mathcal {K}\right) \in \mathscr {S}$ and $x\notin \mathcal {N}$, $\big ( \mathcal {N}\cup \left\{ x\right\} , \mathcal {K}\cup \left\{ x\right\} \big ) \succsim _{\mathfrak {M}}\left( \mathcal {N},\mathcal {K} \right)$ if and only if $\left( \mathcal {N},\mathcal {N}\backslash \mathcal {K}\right) \succsim _{\mathfrak {M}}\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {N}\backslash \mathcal {K}\right)$.

In words, suppose that a new element is added to the set $\mathcal {N}$, and that this makes the new selected set more orderly. Then it must be the case that the “mirror image” of the new selected set, that is the selection from the same set which includes all the elements which are not selected in the original selection, and excludes all those which were included, is less orderly than the mirror image of the original selected set. Axiom 4 is stated for the case when the new element is selected, but it of course implies the opposite case: $\big ( \mathcal {N}\cup \left\{ x\right\} , \mathcal {K} \big ) \succsim _{\mathfrak {M}}\left( \mathcal {N},\mathcal {K}\right)$ if and only if $\left( \mathcal {N} ,\mathcal {N}\backslash \mathcal {K}\right) \succsim _{\mathfrak {M}}\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {N}\backslash \mathcal {K} \cup \left\{ x\right\} \right)$. Axiom 4 is illustrated below, using the representation of a selected set given above:

If the first selected set is more orderly than the third, which differ from the first as it has a new element, ranked 5, which is selected, then it must be the case that the second is less orderly than the fourth, which is the mirror image of the third, and vice versa.

We can now establish the main result of the paper.

Theorem 1

A reflexive transitive, complete and continuous orderliness relation ${\mathfrak {M}}$ satisfies Axioms 2–4 if and only if it can be represented by a monotonic transformation of the index of orderliness $M{( m ( \mathcal {N},\mathcal {K}) ) }$, given in (2).

The “if” part of this result follows immediately from Proposition 2: to see this simply note that Axiom 4 implies Axiom 1. To establish the “only if” part, suppose that there is an orderliness relation which satisfies Axiom 4, which is represented by an index M which is not a function of the sum of the ranks of the selected elements. In particular take a set $\left( \mathcal {N},\mathcal {K}\right)$ and a new element $x \notin \mathcal {N}$, and let the index attached to the set $\left( \mathcal {N},\mathcal {K}\right)$ have value $M_{0}$. This can be mapped into the index of orderliness (2), so that

$$\begin{aligned} M\left( m\left( \mathcal {N},\mathcal {K}\right) \right) =\frac{N+1-\frac{2}{K} r}{N-K}\text {,} \end{aligned}$$

(7)

The “only if” part is based on the following Lemma, which also has independent interest.

Lemma 1

An index of orderliness represents a relation $\mathfrak {M}$ satisfying Axiom 4 if and only if it is a decreasing function of the sum of the ranks of the selected elements.

Proof

Consider a set $\left( \mathcal {N},\mathcal {K}\right)$. We show, first, that the index (7), which is decreasing in the sum of the ranks of the selected elements, does indeed satisfy Axiom 4, and, second, that an index which satisfies Axiom 4 must be decreasing in the sum of the ranks of the selected elements.

Starting from the set $\left( \mathcal {N},\mathcal {K}\right) ,$ suppose that we add a new element x, which takes rank $\rho _{\mathcal {N}}\left( x\right)$, and with $k_{x}$ selected elements in the “initial” set $\left( \mathcal {N},\mathcal {K}\right)$ with better rank than x. The sum of the ranks of the selected elements in the new set $\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {K}\cup \left\{ x\right\} \right)$ is $\left( r+\rho _{\mathcal {N}}\left( x\right) +\left( K-k_{x}\right) \right)$, and hence the index of orderliness of this new set is given by:

$$\begin{aligned} M\left( m\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {K}\cup \left\{ x\right\} \right) \right) =\frac{N+2-\frac{2}{K+1}\left( r+\rho _{\mathcal {N}}\left( x\right) +K-k_{x}\right) }{N-K}\text {.} \end{aligned}$$

(8)

Suppose that this new element x makes the selection more orderly, and so the expression on the RHS of (8) must exceed the corresponding expression in (7):

$$\begin{aligned} \frac{N+2-\frac{2}{K+1}\left( r+\rho \left( x\right) +K-k_{x}\right) }{N-K}>\frac{N+1-\frac{2}{K}r}{N-K}\text {.} \end{aligned}$$

Re-arranging, we can see that the above inequality holds, and therefore that the index of orderliness increases when the new element x is added to the selected set and selected, if and only if

$$\begin{aligned} \frac{r}{K}-\rho _{\mathcal {N}}\left( x\right) +k_{x}>\frac{K-1}{2}. \end{aligned}$$

(9)

Now, for Axiom 4 to hold, we require that the mirror image of the new set is less orderly than the mirror image $\left( \mathcal {N} ,\mathcal {N}\backslash \mathcal {K}\right)$ of the “initial” set, since the set $\left( \mathcal {N\cup }\left\{ x\right\} ,\mathcal {K\cup }\left\{ x\right\} \right)$ is more orderly than the “initial” set $\left( \mathcal {N} ,\mathcal {K}\right)$.

We therefore proceed to calculate the index of orderliness for the mirror image of the “initial” set $\left( \mathcal {N},\mathcal {K}\right)$, that is for the selected set $\left( \mathcal {N},\mathcal {N}\backslash \mathcal {K}\right)$. In this set, there are N elements of which $\left( N-K\right)$ are selected. The sum of the selected elements of $\left( \mathcal {N},\mathcal {N}\backslash \mathcal {K} \right)$ is given by $\frac{N\left( N+1\right) }{2}-r$, and so we have:

$$\begin{aligned} M\left( m\left( \mathcal {N},\mathcal {N}\backslash \mathcal {K}\right) \right) =\frac{N+1-\frac{2}{N-K}\left( \frac{N\left( N+1\right) }{2}-r\right) }{K}\text {.} \end{aligned}$$

(10)

The mirror image of the “new” set $\left( \mathcal {N\cup }\left\{ x\right\} ,\mathcal {K\cup }\left\{ x\right\} \right)$, is the set $\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {N}\backslash \mathcal {K}\right)$. In this set there are $N-\rho _{\mathcal {N}}\left( x\right) +1$ elements with rank worse than x, of which $\left( K-k\right)$ are not selected. Therefore $N-\rho _{\mathcal {N}}\left( x\right) +1-\left( K-k_{x}\right)$ are selected, increasing the sum of the ranks of the selected elements by this amount. Thus:

$$\begin{aligned}&M\left( m\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {N} \backslash \mathcal {K}\right) \right) \nonumber \\&\quad = \frac{N+2-\frac{2}{N-K}\left( \left( \frac{N\left( N+1\right) }{2}-r\right) +N-\rho _{\mathcal {N}}\left( x\right) +1-\left( K-k_{x}\right) \right) }{K+1}. \end{aligned}$$

(11)

Axiom 4 requires that the RHS of (10) exceeds the RHS of (11). Simple algebraic re-arrangement shows that this is the case if and only if condition (9) holds. We have therefore shown that an orderliness relationship that is represented by the index (7) satisfies Axiom 4.

Next we show that if an orderliness relationship satisfies Axiom 4, then it must be decreasing in the sum of the ranks of the selected elements. And thus there is a monotonic transformation such that it equals index (7).

We proceed from a selected set $\left( \mathcal {N},\mathcal {K}\right)$, to which we add a new element $x\notin \mathcal {N}$, which is selected, and then remove a selected element of $\mathcal {N}$, $y\in \mathcal {K}$. Thus we start from set $\left( \mathcal {N},\mathcal {K}\right)$ and arrive at a “final” set $\left( \mathcal {N} \cup \left\{ x\right\} \backslash \left\{ y\right\} ,\mathcal {K}\cup \left\{ x\right\} \backslash \left\{ y\right\} \right)$, which has the same N and the same K. Suppose that this makes the new set more orderly. We show that the change in the sum of the ranks of the selected elements from the “initial” set and the “final” set has the opposite sign of the corresponding change in their mirror images. Recall that, since this orderliness relation satisfies Axiom 4, then the mirror image of the “new” set is more orderly than the mirror image of the original set $\left( \mathcal {N},\mathcal {N}\backslash \mathcal {K}\right)$, and so the index is monotonic in the sum of the ranks of the selected elements. Moreover, because it must take values $-1$ and 1 at antiperfect and perfect selections respectively, then it must be monotonically decreasing, and this proves the Lemma.

So to establish the above argument, we compute the sum of the ranks of the selected elements in these four sets, namely the “initial” set $\left( \mathcal {N},\mathcal {K}\right)$, the “final” set $\left( \mathcal {N} \cup \left\{ x\right\} \backslash \left\{ y\right\} ,\mathcal {K}\cup \left\{ x\right\} \backslash \left\{ y\right\} \right)$, and their respective mirror images, $\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {N} \backslash \mathcal {K}\right)$ and $\left( \mathcal {N}\cup \left\{ x\right\} \backslash \left\{ y\right\} ,\mathcal {N}\backslash \mathcal {K} \right)$. Begin with a selected set $\left( \mathcal {N},\mathcal {K}\right) ,$ and let r be the sum of the ranks of the selected elements. Add x, and the sum of the selected ranks of the selected elements of this set, $\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {K}\cup \left\{ x\right\} \right)$, is $r+\rho _{\mathcal {N}}\left( x\right) +\left( K-k_{x}\right)$, where, as above, $k_{x}$ is the number of selected elements in the “initial” set $\left( \mathcal {N} ,\mathcal {K}\right)$ with better rank than x. Now remove from this set the element y, which has rank $\rho _{\mathcal {N}\cup \left\{ x\right\} }\left( y\right)$ in the set $\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {K}\cup \left\{ x\right\} \right)$. The sum of the rank in the set thus obtained, $\left( \mathcal {N}\cup \left\{ x\right\} \backslash \left\{ y\right\} ,\mathcal {K}\cup \left\{ x\right\} \backslash \left\{ y\right\} \right)$, is

$$\begin{aligned} (r+\rho _{\mathcal {N}}( x) +( K-k_{x}) )-\rho _{\mathcal {N}\cup \{ x \} } ( y ) - ( K+1-k_{y} ) , \end{aligned}$$

(12)

where $k_{y}$ is the number of selected elements in the “intermediate” set $\big ( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {K}\cup \left\{ x\right\} \big )$ with better rank than y.

Now consider the mirror images. For the initial set $\left( \mathcal {N} ,\mathcal {N}\backslash \mathcal {K}\right)$, this is $\frac{\left( N+1\right) N}{2}-r$. The sum of the ranks of the selected elements in the mirror image of the intermediate set $\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {N}\backslash \mathcal {K}\right)$ is, as shown above, $\frac{\left( N+1\right) N}{2}-r+\left( N-K-\left( \rho _{\mathcal {N} }\left( x\right) -1-k_{x}\right) \right)$ (the initial number of selected elements, minus the number of those which are better than x: there are $\left( \rho _{\mathcal {N}}\left( x\right) -1\right)$ elements of rank lower than x and $k_{x}$ are selected). Finally, remove y from the mirror image of the intermediate set, to obtain the mirror image of the final set. The sum of the ranks of the selected elements of this set is

$$\begin{aligned}&\left( \frac{\left( N+1\right) N}{2}-r+\left( N-K-\left( \rho _{\mathcal {N}}\left( x\right) -1-k_{x}\right) \right) \right) \nonumber \\&\quad -( N-K-\rho _{\mathcal {N}\cup \left\{ x\right\} }\left( y\right) +k_{y}) \text {.} \end{aligned}$$

(13)

The first brackets contain the sum of the ranks of the selected elements in the intermediate set. This is changed by the removal of y: we must subtract 1 for each of those elements that were in the “intermediate” set $\left( \mathcal {N}\cup \{ x\} ,\mathcal {K}\cup \left\{ x\right\} \right)$, had worse rank than y, and were not selected: $N-K$ were not selected, and $k_{y}$ were better than y. So $\rho _{\mathcal {N}\cup \left\{ x\right\} }\left( y\right) -k_{y}$ were not selected and are better than y.

To establish the Lemma we need to show that if the sum of ranks (12) is lower than r (the sum of the ranks in the “initial” set $\left( \mathcal {N},\mathcal {K}\right)$), then the sum of the ranks of (13) is higher than $\frac{\left( N+1\right) N}{2}-r$, the sum of the ranks in the mirror image of the “initial” set $\left( \mathcal {N} ,\mathcal {N}\backslash \mathcal {K}\right)$. That is if

$$\begin{aligned} \big (r+\rho _{\mathcal {N}}\left( x\right) +\left( K-k_{x}\right) \big )-\rho _{\mathcal {N}\cup \left\{ x\right\} }\left( y\right) -\left( K+1-k_{y}\right) <r\text {,} \end{aligned}$$

(14)

then

$$\begin{aligned}&\left( \frac{\left( N+1\right) N}{2}-r+\left( N-K-\left( \rho _{\mathcal {N}}\left( x\right) -1-k_{x}\right) \right) \right) \nonumber \\&\quad - ( N-K-\rho _{\mathcal {N}\cup \left\{ x\right\} }\left( y\right) +k_{y}) > \frac{\left( N+1\right) N}{2}-r\text {.} \end{aligned}$$

(15)

As required, (14) and (15) are equivalent, and the proof is complete. $\square$

This Lemma implies that a change that decreases the sum of the ranks of the selected elements makes the set more orderly would make the mirror image of the selected set less orderly, as required by Axiom 4.

We can now prove the main theorem, by showing that every index of orderliness representing a relation $\mathfrak {M}$ satisfying Axioms 2 and 3 which is a decreasing function of the sum of the ranks of the selected elements is a strictly monotonic transformation of $M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }$.

Proof of Theorem 1

Given the ordinal property of the ranking determined by the index, any strictly decreasing function of the sum of ranks can be mapped through a monotonic transformation into a decreasing linear function of the sum of ranks. That is, any index of orderliness which is a function of the sum of the ranks, r, can be transformed into one that is written as

$$\begin{aligned} M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }=a_{K,N} -b_{K,N} r . \end{aligned}$$

Moreover Proposition 1 constrains all perfect selections have the same value, and all antiperfect selections also to have the same value: these values can be normalised to 1 and $-1$ respectively . This implies that for every $N>1$ and every $K<N$, the following must hold

$$\begin{aligned} a_{K,N} -b_{K,N} \frac{K\left( K+1\right) }{2}&=1\text {,}\end{aligned}$$

(16)

$$\begin{aligned} a_{K,N} -b_{K,N} \frac{K\left( 2N-K+1\right) }{2}&=-1\text {.} \end{aligned}$$

(17)

The first condition requires all perfect selections to give value 1 to the index and the second all antiperfect selections to give value $-1$. Solving the above in $a_{K,N}$ and $b_{K,N}$, we get:

$$\begin{aligned} a_{K,N}&=\frac{N+1 }{N-K}\text {,}\end{aligned}$$

(18)

$$\begin{aligned} b_{K,N}&=\frac{2}{K\left( N-K\right) }\text {.} \end{aligned}$$

(19)

Next, we proceed by induction on N. When $N=2$, its lowest possible value, there are two possible selections from the set $\left\{ 1,2\right\}$, $K=\left\{ 1\right\}$ which has sum of ranks $r=1$ and $K=\left\{ 2\right\}$ which has sum of ranks $r=2$. The former is perfect, and so we must have $a_{1,2} -b_{1,2} =1$, and the latter is antiperfect, and so $a_{1,2} -2b_{1,2} =-1$, (18) and (19) satisfy these constraints. In addition, (18) and (19) must also satisfy Axioms 2 and 3.

• When the selection is $\mathcal {K}=\left\{ 1\right\}$, adding a new element to $\mathcal {N}$ can result in a selected set with any of the following images: $\left\{ 1,2\right\}$ (Axiom 2.i or Axiom 3.i), $\left\{ 2\right\}$ (Axiom 2 .ii), $\left\{ 1\right\}$ (Axiom 3.i) and $\left\{ 1,3\right\}$ (Axiom 3.ii). Of these $\left\{ 1,2\right\}$ and $\left\{ 1\right\}$ are perfect, and so it must be $a_{2,3} -3b_{2,3} =1$ and $a_{1,3} -b_{1,3} =1$: both of these hold. Conversely, $\left\{ 2\right\}$ and $\left\{ 1,3\right\}$ are not perfect, and so $a_{1,3} -2b_{1,3} \in \left( -1,1\right)$ and $a_{2,3} -4b_{2,3} \in \left( -1,1\right)$ which hold.

• Similarly, when the selection is $\mathcal {K}=\left\{ 2\right\}$, the add operation can result in any of the following new selected sets: $\left\{ 1,3\right\}$ (Axiom 2.i) $\left\{ 2,3\right\}$ (again Axiom 2.i and also Axiom 3 .ii), $\left\{ 3\right\}$ (Axiom 2.ii), $\left\{ 2\right\}$ (Axiom 3.i). Selection $\left\{ 2,3\right\}$ is antiperfect, and so $a_{2,3} -5b_{2,3}$ must equal $-1$, and selection is also antiperfect $\left\{ 3\right\}$, which requires $a_{1,3} -3b_{1,3} =-1$. The remaining selections are not antiperfect, and this requires: $a_{2,3} -4b_{2,3} \in \left( -1,1\right)$, and $a_{1,3} -2b_{1,3} \in \left( -1,1\right)$. All these hold when $a_{K,N}$ and $b_{K,N}$ are given by (18) and (19).

This establishes that the coefficients a and b satisfy (16) and (17) and so are given by (18) and (19) when $N=2$, and establishes the first step of the induction process. For the second step, assume to have shown that the statement holds for $N-1$, that is:

$$\begin{aligned} a_{K,N-1}&=\frac{N}{N-1-K}\text {,}\end{aligned}$$

(20)

$$\begin{aligned} b_{K,N-1}&=\frac{2}{K\left( N-1-K\right) }\text {,} \end{aligned}$$

(21)

for all $K=1,\ldots ,N-2$. Recall that $r_{K}$ and $r^{N,K}$ are the lowest and highest possible values for the sum of ranks r when the set has size N and the selection has size K. We must have:

$$\begin{aligned} a_{K-1,N-1}-b_{K-1,N-1}r_{K-1}&=a_{K,N}-b_{K,N}r_{K}\nonumber \\&=a_{K,N}-b_{K,N}\left( r_{K-1}+K\right) \text {,} \end{aligned}$$

(22)

because if the selection is perfect then including a new element both in $\mathcal {N}$ and in $\mathcal {K}$ increases the lowest possible sum of the ranks by K. And similarly for the highest possible sum of ranks:

$$\begin{aligned} a_{K-1,N-1}-b_{K-1,N-1}r^{N-1,K-1}&=a_{K,N}-b_{K,N}r^{N,K}\nonumber \\&=a_{K,N}-b_{K,N} ( r^{N-1,K-1}+N). \end{aligned}$$

(23)

By the induction hypothesis, (20)–(21), we have that

$$\begin{aligned} a_{K-1,N-1}&=\frac{N}{ N-K },\\ b_{K-1,N-1}&=\frac{2}{( N-K) ( K-1) }\text {.} \end{aligned}$$

Substitute these into the two above equations, (22) and (23):

$$\begin{aligned} \frac{N}{ N-K }-\frac{2r_{K-1}}{\left( N-K\right) \left( K-1\right) }&=a_{K,N}-b_{K,N}\left( r_{K-1}+K\right) \text {,}\\ \frac{N}{ N-K }-\frac{2r^{N-1,K-1}}{\left( N-K\right) \left( K-1\right) }&=a_{K,N}-b_{K,N}( r^{N-1,K-1}+N); \end{aligned}$$

now substitute

$$\begin{aligned} r_{K-1}&=\frac{\left( K-1\right) \left( \left( K-1\right) +1\right) }{2}=\frac{K\left( K-1\right) }{2}\text {,}\\ r^{N-1,K-1}&=\frac{\left( K-1\right) \left( 2\left( N-1\right) -\left( K-1\right) +1\right) }{2}=\frac{\left( 2N-K\right) \left( K-1\right) }{2}\text {,} \end{aligned}$$

to get:

$$\begin{aligned} \frac{N}{ N-K }-\frac{K\left( K-1\right) }{\left( N-K\right) \left( K-1\right) }&=a_{K,N}-b_{K,N}\left( \frac{K\left( K-1\right) }{2}+K\right) \text {,}\\ \frac{N}{ N-K }-\frac{\left( 2N-K\right) \left( K-1\right) }{\left( N-K\right) \left( K-1\right) }&=a_{K,N}-b_{K,N}\left( \frac{\left( 2N-K\right) \left( K-1\right) }{2}+N\right) \text {.} \end{aligned}$$

Finally, solve the above in $a_{K,N}$ and $b_{K,N}$, to obtain (18) and (19). This establishes the Theorem. $\square$

Obviously any strictly monotonic transformation of the index (2) would represent a relation satisfying Axioms 2–4. The functional form in (2), which is linear in the sum of ranks, has the natural property of being 0, in expectation, if the selection is completely random: to see this, note that the expected rank of K random draws is $\frac{K(N+1)}{2}$. Substitute this in (2), to obtain 0.

Theorem 1 is the main result of the paper: in words, it shows that the index of orderliness (2) satisfies the dominance Axioms 2 and 3, and the mirror invariance Axiom 4, and, conversely, that any index which satisfies the dominance Axioms 2 and 3, and the mirror invariance Axiom 4, is a monotonic transformation of the index of orderliness (2). Only index (2) satisfies all three axioms, and it is the only one (up to monotonic transformations) to do so: Axioms 2, 3 and 4 together characterise the index of orderliness (2).

The structure of the proof of Theorem 1 allows to replace the mirror invariance Axiom with an equivalent one, which gives a different interpretation to the restriction imposed on the orderliness relation to obtain characterisation.

Axiom 5

(Position Irrelevance) For every $\left( \mathcal {N} ,\mathcal {K}\right) \in \mathscr {S}$ and for every $y_{1}\in \mathcal {N} \backslash \mathcal {K}$ and $x_{1}\in \mathcal {K}$, satisfying $\rho \left( x_{1}\right) =\rho \left( y_{1}\right) +a$, and any $y_{2}\in \mathcal {N} \backslash \mathcal {K\cup }\left\{ x_{1}\right\}$ and $x_{2}\in \mathcal {K\cup }\left\{ y_{1}\right\}$, satisfying $\rho \left( x_{2}\right) =\rho \left( y_{2}\right) -a$, where a is any integer such that $\rho \left( y_{1}\right) +a, \rho \left( y_{2}\right) -a\in \left\{ 1,\ldots ,N\right\}$, then $\left( \mathcal {N},\mathcal {K}\right) \sim _{\mathfrak {M}}\left( \mathcal {N},\mathcal {K\cup }\left\{ y_{1} ,y_{2}\right\} \backslash \left\{ x_{1},x_{2}\right\} \right)$.

In words, Axiom 5 considers two subsequent “swaps” between a selected and a non-selected element. The element newly included in the selection and the one removed differ in rank by a. The second swap turns the intermediate selection obtained with the first swap into the final one, by selecting an element not selected and removing another element from the (intermediate) selection which are $-a$ ranks apart. Thus, for example, if the first swap selects the (originally non-selected) 12-th ranked element and de-selects the 19-th ranked; their difference in rank is $-7$. Suppose the second swap selects the (originally non-selected) 56-th ranked element and de-selects the 49-th ranked; their difference in rank is 7.¹² Then, by Axiom 5, the initial and the final selection are equally orderly. In shorter, looser words, the effect of a change in the selection depends only on the extent of the change, not on whether it affects the upper or the lower tail of the distribution.¹³ We note that the relative importance of the position in the ranking is at the core of the analysis of distance between preferences in Can (2014). It is straightforward to establish the following.

Corollary 2

A reflexive transitive, complete and continuous orderliness relation ${\mathfrak {M}}$ satisfies Axioms 2, 3, and 5 if and only if it can be represented by a monotonic transformation of the index of orderliness $M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }$, given in (2).

Proof

This is simply a consequence of Lemma 1 together with the obvious observation that the sum of the ranks is left unchanged by a swap that satisfies Axiom 5. $\square$

Note that Theorem 1 and Corollary 2 together imply that Axioms 4, mirror invariance, and Axiom 5, position irrelevance, are equivalent: each holds for a relation if and only if the other does.

4 The index M and the Kendall-Tau distance

The study of metrics on orders, initiated by Kendall (1938) has developed a measure of distance between two rankings of the elements of a given set $\mathcal {N}$, the Kendall-Tau distance (Kemeny 1959), recently extended to choice functions (Klamler 2008). This is obtained by counting the number of times the two rankings “switch” two elements $x,y \in \mathcal {N}$. That is, if $\rho _{1}$ and $\rho _{2}$ are the two rankings, the distance between them is the number of pairs $\left( x,y\right) \in \mathcal {N}\times \mathcal {N}$ such that $\rho _{1}\left( x\right) >\rho _{2}\left( x\right)$ and $\rho _{1}\left( y\right) <\rho _{2}\left( y\right)$. This count can then be normalised by the maximum possible number of switches.

This idea has been applied to voting mechanism, whereby a ranking is interpreted as a vote, and the outcome of an election as the aggregation of the rankings of different voters.¹⁴ We view the Kendall-Tau distance from a different angle: note in the first place that the index of orderliness (2) applies to a broader range of situations, such as those where the selections being compared are from different sets. Secondly, the orderliness index (2) compares selections, rather than rankings, as the Kendall-Tau distance does. Nevertheless, a selection does rank the elements of a set, albeit in a very coarse manner: the K selected elements are joint first, and the non-selected ones are joint $\left( K+1\right)$-th. Recall that the Kendall-Tau distance is defined when there are ties, and so we can measure the Kendall-Tau distance between the ranking determined by a selection and a given ranking $\rho$ of $\mathcal {N}$. In this section we study the relation between this Kendall-Tau distance and the index of orderliness (2), $M{\left( \mathcal {N},\mathcal {K}\right) }$.

To develop the formal analysis, given a selected set $\left( \mathcal {N} ,\mathcal {K}\right)$, define a mapping $\kappa ^{\mathcal {N}\mathcal {K} }:\mathcal {N}\longrightarrow \left\{ 1,\ldots ,N\right\}$ as follows:

$$\begin{aligned} \kappa ^{\mathcal {N}\mathcal {K}}&:x\longmapsto \Bigg \{ \begin{array}{lll} 1 &{}\quad \hbox {if} &{}\quad \, x\in \mathcal {K}\text {,}\\ K+1 &{}\quad \hbox {if} &{}\quad \, x\in \mathcal {N}\backslash \mathcal {K}. \end{array} \end{aligned}$$

(24)

That is the ordering $\kappa ^{\mathcal {N}\mathcal {K}}$ assigns a better rank to $x \in \mathcal {N}$ than to $y \in \mathcal {N}$ if and only x is selected and y is not. We can label $\kappa ^{\mathcal {N}\mathcal {K}}$ the ordering induced by the selection $\mathcal {K}$. Since it is an ordering of the set $\mathcal {N}$, we can define the Kendall-Tau distance between $\rho$ and $\kappa ^{\mathcal {N} \mathcal {K}}$. To count the number of switches between rankings $\rho$ and $\kappa ^{\mathcal {N}\mathcal {K}}$, note that a switch occurs only when the better element is not selected and the worse one is. The total number of switches is thus measured by:

$$\begin{aligned} \frac{\left| \kappa ^{\mathcal {N}\mathcal {K}}\left( x\right) -\kappa ^{\mathcal {N}\mathcal {K}}\left( y\right) \right| }{2K}\left( 1-\frac{\rho \left( x\right) -\rho \left( y\right) }{\left| \rho \left( x\right) -\rho \left( y\right) \right| }\frac{\kappa ^{\mathcal {N}\mathcal {K}}\left( x\right) -\kappa ^{\mathcal {N}\mathcal {K}}\left( y\right) -\varepsilon }{\left| \kappa ^{\mathcal {N}\mathcal {K}}\left( x\right) -\kappa ^{\mathcal {N}\mathcal {K}}\left( y\right) -\varepsilon \right| }\right) \text {,} \end{aligned}$$

where $\varepsilon \in \left( 0,1\right)$. To see this, note simply that the first term is $\frac{1}{2}$ if only one of x and y is selected, and is 0 otherwise. Consider the second factor: given that $\kappa ^{\mathcal {N}\mathcal {K}}\left( x\right) \ne \kappa ^{\mathcal {N}\mathcal {K}}\left( y\right)$, its second term is 1 if $\rho$ and $\kappa$ agree, and $-1$ if they do not, and so the whole term is 2 if a switch occurs. The role of $\varepsilon$ is to ensure that the denominator is never 0. To sum up, the Kendall distance, between the ranking of a set $\mathcal {N}$ and the ranking induced by the selection $\mathcal {K}$ is given by

$$\begin{aligned} \tau {\left( \mathcal {N},\mathcal {K}\right) }= & {} \sum _{x\in \mathcal {N}} \sum _{y\in \mathcal {N}\backslash \left\{ x\right\} }\frac{\left| \kappa ^{\mathcal {N}\mathcal {K}}\left( x\right) -\kappa ^{\mathcal {N} \mathcal {K}}\left( y\right) \right| }{2K}\\&\quad \left( 1-\tfrac{\rho \left( x\right) -\rho \left( y\right) }{\left| \rho \left( x\right) -\rho \left( y\right) \right| }\tfrac{\kappa ^{\mathcal {N}\mathcal {K}}\left( x\right) -\kappa ^{\mathcal {N}\mathcal {K}}\left( y\right) -\frac{1}{2}}{\left| \kappa ^{\mathcal {N}\mathcal {K}}\left( x\right) -\kappa ^{\mathcal {N} \mathcal {K}}\left( y\right) -\frac{1}{2}\right| }\right) . \end{aligned}$$

Next note that the maximum number of switches is $\left( N-K\right) K$, which happens in an antiperfect selection, where the first $N-K$ elements have K switches each, and therefore the normalised Kendall-Tau distance is

$$\begin{aligned} \hat{\tau }{\left( \mathcal {N},\mathcal {K}\right) }=\frac{\tau {\left( \mathcal {N},\mathcal {K}\right) }}{K\left( N-K\right) } \text {.} \end{aligned}$$

(25)

One would want that the shorter the distance between the given ranking in the set $\mathcal {N}$ and the ordering induced by the selection $\mathcal {K}$, the more orderly the selected set $\left( \mathcal {N},\mathcal {K}\right)$. The next results shows that this is indeed the case: it shows that there is a very simple link between the normalised Kendall-Tau distance (25) and the index of orderliness (2).

Proposition 4

$M{\left( m \left( \mathcal {N} ,\mathcal {K}\right) \right) }=1-2\hat{\tau }{\left( \mathcal {N} ,\mathcal {K}\right) }$.

Proof

We proceed by induction on N, the number of elements of N. It is trivially true for $N=2$. In this case, $K=1$: when the better (worse) element is selected $M{\left( m \left( \mathcal {N} ,\mathcal {K}\right) \right) }=1$ and $\hat{\tau }{\left( \mathcal {N} ,\mathcal {K}\right) }=\tau {\left( \mathcal {N},\mathcal {K}\right) }=0$ ($M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }=-1$ and $\hat{\tau }{\left( \mathcal {N},\mathcal {K}\right) } =\tau {\left( \mathcal {N},\mathcal {K}\right) }=1$). Next, suppose to have demonstrated the result for $N-1$; let K be the number of selected elements, so we have

$$\begin{aligned} M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }=1-2\hat{\tau }{\left( \mathcal {N},\mathcal {K}\right) }\text {,} \end{aligned}$$

and

$$\begin{aligned} M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }=\frac{N-\frac{2 }{K}\sum _{x\in \mathcal {K}}\rho \left( x\right) }{ N-1-K }=\frac{K\left( N-1-K\right) -2\tau {\left( \mathcal {N},\mathcal {K} \right) }}{K\left( N-1-K\right) }\text {,} \end{aligned}$$

(26)

which implies:

$$\begin{aligned} \sum _{x\in \mathcal {K}}\rho \left( x\right) -\tau {\left( \mathcal {N} ,\mathcal {K}\right) }=\frac{1}{2}K\left( K+1\right) \text {.} \end{aligned}$$

(27)

Now increase the number of elements in the set from $N-1$ to N, which is achieved via the inclusion in $\mathcal {N}$ of a new element, z. Its rank in the new set $N\cup \left\{ z\right\}$ is $\rho \left( z\right) \in \left\{ 1,\ldots ,N\right\}$, and z is either selected, giving the new selected set $\left( \mathcal {N}\cup \left\{ z\right\} ,\mathcal {K}\cup \left\{ z\right\} \right)$, or not selected, giving the set $\left( \mathcal {N} \cup \left\{ z\right\} ,\mathcal {K}\right)$. Consider the first case: the new number of selected elements is $K+1$. In this case note that the total rank of the selected elements increases by $\rho \left( z\right) +s$, where s is the number of elements in the set $\left\{ y\in \mathcal {K}|\rho \left( y\right) >\rho \left( z\right) \right\}$. The new value of $\tau$, is instead equal to the previous one, $\tau {\left( \mathcal {N},\mathcal {K} \right) }$, increased by the number of new switches generated by z: all the previous switches remain such of course. This is given simply by the number of elements in the set $\left\{ y\in \mathcal {N}\backslash \mathcal {K}|\rho \left( y\right) <\rho \left( z\right) \right\}$, which can be obtained by noting that there are $\rho \left( z\right) -1$ elements ranked better than z, and that $K-s$ are selected (because s is the number of selected elements with rank worse than z). Thus the increase in $\tau$ is $\left( \rho \left( z\right) -1-\left( K-s\right) \right)$. This gives:

$$\begin{aligned}&M\left( m \left( \mathcal {N} \cup \left\{ z\right\} ,\mathcal {K}\cup \left\{ z\right\} \right) \right) \\&\quad =\frac{N+1 - \frac{2}{K+1}\sum _{x\in \mathcal {K}}\rho \left( x\right) -\left( \rho \left( z\right) +s\right) }{\left( K+1\right) \left( N-K-1\right) }\\&\qquad =\frac{\left( K+1\right) \left( N-K-1\right) -2\tau {\left( \mathcal {N},\mathcal {K}\right) }-2\left( \rho \left( z\right) -1-\left( K-s\right) \right) }{\left( K+1\right) \left( N-K-1\right) }\text {.} \end{aligned}$$

Rearrange to obtain (27) again. Suppose instead z is not selected, and the new set is $\left( \mathcal {N}\cup \left\{ z\right\} ,\mathcal {K}\right)$. The total rank of the selected elements increases only by s, where again s is the number of elements in the set $\left\{ y\in \mathcal {K}|\rho \left( y\right) >\rho \left( z\right) \right\}$. The new value of $\tau$ is instead equal to the previous one, $\tau {\left( \mathcal {N},\mathcal {K}\right) }$, increased by the number of new switches generated by z, which in this case is s, the number of selected elements with rank worse than z. Hence we can write

$$\begin{aligned} M \left( m \left( \mathcal {N} \cup \left\{ z\right\} ,\mathcal {K}\right) \right)= & {} \frac{N+1 -\frac{2}{K}\left( \sum _{x\in \mathcal {K}}\rho \left( x\right) -s\right) }{N-K} \\= & {} \frac{K\left( N-K\right) -2\tau {\left( \mathcal {N},\mathcal {K}\right) }-2s}{K\left( N-K\right) }\text {.} \end{aligned}$$

which again gives (27) and completes the proof. $\square$

Thus, in addition to satisfying natural axioms, the index (2) coincides, in the situations where both can be applied, with an established measure of distance between rankings. This paper therefore provides a micro-foundation of the Kendall-Tau distance, which currently lacks one.

The Kendall-Tau distance and the Spearman coefficient of rank correlation give in general different answers to the question of the comparison between two rankings: this implies that there at least one of the axioms is violated by the Spearman coefficient of correlation between the original ranking and the ordering $\kappa ^{\mathcal {N}\mathcal {K}}$ derived in (24). The Spearman coefficient, originally intended for situations with no ties in either ordering, has been adapted by Taylor (1964) for the case of ties: the adaptation involves replacing the ties with “fractional ties” and adjust with one correction factor for each set of tied values, again, in either of the two orderings. Given the assumption of no ties in the ordering of the set $\mathcal {N}$, and given that there are only two ranks in (24), and thus exactly two correction factors will be required, one for K, the other for (N-K) tied elements, the Spearman coefficient of correlation between the original ranking and the ordering $\kappa ^{\mathcal {N}\mathcal {K}}$ derived in (24) is given by:

$$\begin{aligned} 1-\frac{\frac{6}{N}\sum _{i=1}^{N}\left( \rho _{i}\left( x\right) -\kappa ^{\mathcal {N}\mathcal {K}} \left( x\right) \right) ^{2}+\frac{N^{2}-1-3K\left( N-K\right) }{2}}{N^{2}-1} \text {.} \end{aligned}$$

(28)

We now show one example of a violation of one of the Axioms. Consider a selected set $\left( \mathcal {N},\mathcal {K}\right)$ represented by 0101110000000. Applying the above formula, we find that the Spearman correlation coefficient is 0.3929. Now consider adding a new element which occupies the seventh place in the new ranking, hence is worse that all of the selected elements, and is itself selected, so the new selected set can be represented by (the new element is the one in the box). By Axiom 3.ii, the new set must be less orderly. However, applying the above formula to the new set gives a higher value of 0.4154, in violation of Axiom 3.ii. Similar counterexamples can be found for the other Axioms, except Axiom 1.

Another measure of correlation is the Kappa correlation. This applies to ranking with very few steps (and coincides with the selection when there are only two categories). This suggests a direction for future research in this area, namely the extension of the index of orderliness to situations where the selection is more nuanced than simply “in” and “out”. An example is a coach selection of the first team and of the second team for a sport tournament.

5 Example: orderliness in Italian universities

The orderliness index proposed here finds a natural application in the analysis of promotions in hierarchical organisations, where, at given intervals, individuals from the pool of potential candidates are assessed and some are promoted, some are not. The determinants of promotions may be stated formally or known implicitly: thus for example, academic promotions may be decided by criteria ranging from scientific productivity, to teaching performance, fund-rising ability, seniority, or age; the relative importance of performance along the various criteria may of course vary from institution to institution and from discipline to discipline.

As explained in Checchi et al. (2014), academic careers in Italy are firmly channelled along narrowly defined research fields: every academic is allocated to one and only one of 371 scientific sectors (SSDs), changing sector is relatively unusual, and the members of the promotion panels in each scientific sector are chosen exclusively among academic post holders in that sector.

The dataset assembled by Checchi et al. (2014) allows us to rank all the candidates for promotion to associate professor in each scientific sector in the period from 1995 to 2011 according to two criteria, their record of publication in international journals¹⁵ and their age.

Since with minor exceptions, each panel could select for promotion only candidates from the lower rank within the same sector, it is a simple matter to calculate the orderliness of the selections made by the promotion panel in each scientific sector, in each of the four separate sub-periods which Checchi et al. (2014) suggest to aggregate calendar years to reflect the pattern of the promotion rounds.

Fig. 1

Orderliness of scientific sectors by productivity and by age—1995–2011

Figure 1 describes the outcome of this process. Each dot corresponds to one of 371 scientific sectors, except the smaller ones, in one 4-year interval. The abscissa of a dot is the orderliness according to scientific productivity, its ordinate the orderliness according to age.¹⁶ Broad scientific areas are colour coded; we have also singled out economics and econometrics among the social science sectors. An analysis of the Italian university sector might use these indices as characteristics of the selection procedures used in promotion and appointments. A cursory visual analysis suggests that overall productivity matters more than age, and also that it matters more in STEM subjects.

6 Concluding remarks

Often an agent chooses a number of options from a larger set, the elements of which can be ranked in some objective or generally accepted manner. We propose a way to assess how close this agent’s selection is to the order of the set. Aside from its intrinsic interest in the examples given in the introduction, availability of this measure might address the need reported in the medical literature for an objective evaluation of clinical services (Iverson 1998; Bickman 2012), or help the study of aspects of social mobility, such as the importance of a person’s family position in her access to leadership positions in society.¹⁷ In this paper, we require that the comparison between selections satisfies some natural dominance requirements (Axioms 2 and 3), and the requirement that the mirror image of a change that nears a selection to the ranking must push the mirror image of that selection away from the ranking (Axiom 4). These three simple axioms prove very strong, in that they identify a unique index which provides an unambiguous comparison of the relative closeness of any selection to the order of the set from which it is taken. This index has a very simple expression, which depends only on the sum of the ranks of the selected elements, and the number of elements in the set and in the selection. The paper ends with a specific example of the potential applications of our index to the Italian university system.