On Fair Person Classification Based on Efficient Factor Score Estimates in the Multidimensional Factor Analysis Model

Abstract

Since Hooker, Finkelman and Schwartzman (Psychometrika 74(3): 419–442, 2009) it is known that person parameter estimates from multidimensional latent variable models can induce unfair classifications via paradoxical scoring effects. The open question as to whether there is a fair and at the same time multidimensional scoring scheme with adequate statistical properties is addressed in this paper. We develop a theorem on the existence of a fair, multidimensional classification scheme in the context of the classical linear factor analysis model and show how the computation of the scoring scheme can be embedded in the context of linear programming. The procedure is illustrated in the framework of scoring the Wechsler Adult Intelligence Scale (WAIS-IV).

Appendices

Appendix A: Proofs of the Theorems

Proof of Theorem 1

(Existence of a regular and fair scoring):$\square$

Proof

The reader who is not familiar with the concept of a recession cone and a recession function should first review “Appendix B,” wherein we gathered all necessary tools from convex analysis which are needed for the following proof.

Our aim is to apply a key theorem of convex analysis (stated as Theorem 5 in “Appendix B”) to the present case. With this in mind, define $C_1:=\{\varvec{u} \in C |\, \sum _i u_i \ge 1 \}$. The recession cone of $C_1$ is the nonnegative orthant: $0^+(C_1)=C$ (see “Appendix B”). The directions of recession of $f_i$ are those vectors $\varvec{u}\ne \varvec{0}$, which satisfy the inequality $f_i(\varvec{u})\le 0$ (see “Appendix B”).

If there is no common direction of recession of the functions $(f_i)_{i=1, \ldots p}$, which is also a direction of recession of $C_1$, then, according to Theorem 5 in “Appendix B,” there is a nonnegative vector $\varvec{\lambda }$ satisfying $\varvec{\lambda }^T \varvec{f(u)}\ge \epsilon$ for some $\epsilon >0$ and for all $\varvec{u} \in C_1$.

Now, let $\varvec{u}$ be any nonzero vector of the nonnegative orthant. Then for a large enough positive scalar c, the vector $\tilde{\varvec{u}}:=c\varvec{u}$ will be in $C_1$. According to the previously established inequality, $\varvec{\lambda }^T \varvec{f(\tilde{u})}\ge \epsilon$. Hence, $\varvec{\lambda }^T \varvec{f(u)}=\frac{1}{c}\varvec{\lambda }^T \varvec{f(\tilde{u})}\ge \frac{\epsilon }{c}>0$. The theorem follows. $\square$

Remark

Incidentally, the proof shows that the existence of the fair composite just relies on the linear structure of the estimate. There is no specific reference to underlying model properties.

Proof of Theorem 2

(Linear independent composites):$\square$

Proof

Let A denote the matrix representation of the linear estimate $\varvec{f}$.

The regularity of the linear combination means that $\varvec{\lambda }^T A \varvec{u}>0$ holds for all $\varvec{u} \in C\setminus \{\varvec{0}\}$.

The existence of a $\delta >0$ (from which the assertion of the theorem follows), such that $\varvec{\lambda ^{*T}}\varvec{f}(C)\subset [0,\infty )$ holds for all $\varvec{\lambda ^*} \in \mathbb {B}_\delta (\varvec{\lambda })$ (the ball of radius $\delta$ centered at $\varvec{\lambda }$) will be proven by contradiction.

Assume on the contrary that to each $\delta _n:=\frac{1}{n}$ ($n \in {\mathbb {N}}$) corresponds a $\varvec{\lambda }_n \in \mathbb {B}_{\delta _n}(\varvec{\lambda })$ and a vector $\varvec{u}_n \in C$ such that

$$\begin{aligned} \varvec{\lambda }_n^TA\varvec{u}_n<0 \end{aligned}$$

(2)

is valid. By multiplying both sides with $|\varvec{u}_n|^{-1}$ and redefining $\varvec{u}_n$ upon using the fact that any positive multiple of an element of C is still in C, we may suppose without loss of generality, that $|\varvec{u}_n|=1$ holds for all n. The sequence $(\varvec{u}_n)_{n \in {\mathbb {N}}}$ is thus bounded. We may therefore extract a subsequence $(\varvec{u}_{n_v})_{v \in {\mathbb {N}}}$  which converges to a point $\varvec{u}$.

The mapping $g(\varvec{x},\varvec{y}):=\varvec{x}^TA\varvec{y}$ is continuous, therefore passing to the limit ($v \rightarrow \infty$) in the inequality (2) leads to:

$$\begin{aligned} \varvec{\lambda }^T_{n_v} A \varvec{u_{n_v}}<0 \Rightarrow \varvec{\lambda }^T A \varvec{u}\le 0. \end{aligned}$$

As a limit point of elements of the closed set C, $\varvec{u}:=\lim _{v \rightarrow \infty }\varvec{u}_{n_v}$ belongs itself to C. Furthermore, the norm equality $|\varvec{u}_{n_v}|=1$ gives $|\varvec{u}|=1$. Hence, in particular $\varvec{u}\ne \varvec{0}$, so that we have established the existence of a nonzero vector $\varvec{u} \in C$ such that $\varvec{\lambda }^Tf(\varvec{u})\le 0$ holds—contradicting the regularity of $\psi$. $\square$

Proof of Theorem 3

(Weighted least squares and ultraparadox):$\square$

Proof

We argue by contradiction: Assume that there is a nonzero vector $\varvec{u^*} \in C$, such that $f_i(\varvec{u^*})\le 0$ holds for all i. Then the inner product of $\varvec{f(u^*)}$ with any nonnegative vector $\varvec{z}$ is nonpositive. The vector $W \varvec{u^*}$ is nonnegative with at least one positive entry, due to the assumption that W is a positive diagonal matrix (and the fact that $\varvec{u^*}$ is part of the nonnegative orthant).

As the matrix A is nonnegative and as no column of $A^T$ equals $\varvec{0}$, it follows that the vector $\varvec{z}:=A^T W \varvec{u^*}$ is nonnegative with at least one positive component.

For this $\varvec{z}$

$$\begin{aligned} 0\ge \varvec{z}^T\varvec{f(u^*)}=\varvec{z}^T(A^TWA)^{-1}A^T W \varvec{u^*}=\varvec{z}^T(A^TWA)^{-1}\varvec{z} \end{aligned}$$

holds, which contradicts the positive definiteness of the matrix $A^TWA$.

Suppose now that all entries of $A^TWA$ are positive. Then, the Perron–Frobenius theorem provides the existence of a positive eigenvector of the matrix $A^TWA$ with eigenvalue $\delta >0$. Using this vector $\varvec{\lambda }$, we have:

$$\begin{aligned} \varvec{\lambda }^T(A^TWA)^{-1}A^TW\varvec{u}=\frac{1}{\delta }\varvec{\lambda }^TA^TW\varvec{u}. \end{aligned}$$

As A and W contain nonnegative elements, we have $\frac{1}{\delta }\varvec{\lambda }^TA^TW\varvec{u}\ge 0$ for all nonnegative $\varvec{u}$. Hence, we have found a fair composite. The regularity of the composite follows from the fact that A has nonzero rows. $\square$

Proof of Theorem 4

(Bayes-EAP and ultraparadox):$\square$

Proof

Assume to the contrary, that there is a nonzero nonnegative vector $\varvec{u^*}$ such that $f(\varvec{u^*})\le \varvec{0}$ holds and denote by $\varvec{x}$ the (unique) solution to the equation $(A\Phi A^T+W)\varvec{x}=\varvec{u^*}$. As $f(\varvec{u^*})\le \varvec{0}$ and as A has nonnegative entries, we have

$$\begin{aligned} \varvec{0}\ge Af(\varvec{u^*})=A\Phi A^T(A\Phi A^T + W)^{-1}\varvec{u^*}=A\Phi A^T\varvec{x} \end{aligned}$$

It follows that $A\Phi A^T\varvec{x}=\varvec{z}$ holds for some nonpositive $\varvec{z}$ and also that $W\varvec{x}$, and therefore $\varvec{x}$, needs to be nonnegative as well as nonzero. Taking the inner product with $\varvec{x}$ it follows from the nonnegative definiteness of $A\Phi A^T$ that $\varvec{x}^T \varvec{z}=\varvec{x}^T A\Phi A^T \varvec{x}\ge 0$ holds. However, W is a positive diagonal matrix. Hence, $\varvec{x}$ needs to be strictly positive on any coordinate where $\varvec{z}$ is strictly negative—otherwise, a contradiction to $A\Phi A^T\varvec{x} + W\varvec{x}=\varvec{u^*}$ with $\varvec{u^*}$ nonnegative would arise. If $\varvec{z}$ is nonzero, this contradicts $\varvec{x}^T \varvec{z}\ge 0$. Hence, we need to rule out the case $\varvec{z}=\varvec{0}$. The latter holds if and only if $A^T \varvec{x}=\varvec{0}$ holds. A has nonnegative entries, and no row of A equals the null vector. Hence, a nonnegative, nonzero vector $\varvec{x}$ satisfying $A^T \varvec{x}=\varvec{0}$ cannot exist.

In the case wherein the matrix $A\Phi A^T$ consists of only positive entries, we can go even further and highlight directly the computation of a fair and regular composite. For this purpose, let $\varvec{v}$ denote the eigenvector corresponding to the largest eigenvalue $\delta$ of the matrix $W^{-1}(A\Phi A^T)$. Note that $\varvec{v}$ may be chosen to consist of positive entries due to the assumed positivity of the entries of $A\Phi A^T$ (see, e.g., ch. 16 of Lax, 2007). Define $\varvec{\lambda }:=A^T\varvec{v}$. Then, we find:

$$\begin{aligned} \varvec{\lambda }^T f(\varvec{u})=\varvec{v}^T A\Phi A^T(A\Phi A^T + W)^{-1}\varvec{u}=\varvec{v}^T A\Phi A^TW^{-1}(A\Phi A^TW^{-1} + I)^{-1}\varvec{u}. \end{aligned}$$

As $\varvec{v}$ is an eigenvector of $W^{-1}A\Phi A^T$ with eigenvalue $\delta >0$, $\varvec{v}$ is also an eigenvector of $W^{-1}A\Phi A^T + I$ with eigenvalue $(\delta +1)$ and therefore also an eigenvector of $(W^{-1}A\Phi A^T + I)^{-1}$ with eigenvalue $\frac{1}{\delta +1}$. Using these facts in the above derivation leads to:

$$\begin{aligned} \varvec{\lambda }^T f(\varvec{u})=\delta \varvec{v}^T (A\Phi A^TW^{-1} + I)^{-1}\varvec{u}=\delta (((A\Phi A^TW^{-1} + I)^{-1})^T \varvec{v})^T \varvec{u}=\frac{\delta }{1+\delta } \varvec{v}^T \varvec{u}. \end{aligned}$$

As $\varvec{v}$ has strictly positive entries, $\varvec{\lambda }^T f(\varvec{u})>0$ will hold for every nonzero $\varvec{u}\in C$. The latter implies that we have found a regular fair composite. $\square$

Appendix B: Tools from Convex Analysis

For the derivation of the main theorem, the concept of a recession cone of a convex set (see also ch. 8 of Rockafellar (1970) for the following definitions) and the concept of a direction of recession of a convex function are crucial. We introduce these concepts in this appendix and also highlight some consequences which are tacitly used throughout the paper.

Definition 1

Let $D \subset {\mathbb {R}}^k$ be a nonempty convex set. The recession cone of D—abbreviated by $0^+(D)$—is the set of vectors $\varvec{u} \in {\mathbb {R}}^k$ such that $D+\varvec{u} \subset D$ holds. Any nonzero vector of $0^+(D)$ is called a direction of recession of the convex set D.

Remark

An equivalent definition would be given by the following: A vector $\varvec{u}$ is a direction of recession of a convex set D if for any $\varvec{v} \in D$ the complete half line emanating from the point $\varvec{v}$ in the direction $\varvec{u}$—i.e., the set of points $(\varvec{v}+\lambda \varvec{u})_{\lambda \in {\mathbb {R}}^+}$—is contained in D. Thus, directions of recessions are identifiable with half lines which never leave the set D (see also Fig. 1).

Definition 2

Let f be a convex function on ${\mathbb {R}}^k$ (finite everywhere) and let $L_{f_{\alpha }}:=\{\varvec{u}| f(\varvec{u}) \le \alpha \}$ be nonempty for some $\alpha$. The recession cone of the latter set is denoted by $0^+(L_{f_{\alpha }})$. Any nonzero vector of this recession cone is called a direction of recession of f.

For the special case of a linear function f, one obtains the important result:

$$\begin{aligned} 0^+(L_{f_{\alpha }})=\{\varvec{u}| f(\varvec{u})\le 0\}. \end{aligned}$$

(3)

Indeed, let $\varvec{u} \in 0^+(L_{f_{\alpha }})$ and let $\varvec{x} \in L_{f_{\alpha }}$. Then, $f(\varvec{x}+\varvec{u})\le \alpha$ and by successively applying the definition of a recession cone, one notices that for any integer n, $f(\varvec{x}+n\varvec{u})\le \alpha$ has to hold. However, due to linearity it follows that:

$$\begin{aligned} \alpha \ge f(\varvec{x}+n\varvec{u})=f(\varvec{x})+nf(\varvec{u}). \end{aligned}$$

The inequality can only hold for each integer n if and only if $f(\varvec{u})\le 0$.

Fig. 1

The set $C_1$ (area on the upper right of the bold boundary line segments) and two translates of the set (red) by two vectors $\varvec{u}$. On the left picture, the translated set remains within $C_1$, whereas on the right picture the translated set contains vectors outside of $C_1$. Only in the first case, $\varvec{u}$ is called a direction of recession of $C_1$ (Color figure online).

In the derivation of the main theorem, the recession cone of a special convex set—namely the convex set

$$\begin{aligned} C_1:=\{\varvec{u} \in C |\, \sum _i u_i \ge 1 \}, \end{aligned}$$

(4)

wherein C denotes the nonnegative orthant—plays a key role. For the special case $k=2$ the set $C_1$ is depicted in Fig. 1. Also included are two translates of the set. As can be seen on the left, if the set is translated by a vector $\varvec{u}$ pointing to the upper right (i.e., a vector of the nonnegative quadrant), then the translated set $C_1+\varvec{u}$ is contained within $C_1$. By definition this means that $\varvec{u}$ is a direction of recession. On the other hand, if the set is translated by a vector not pointing toward the upper right (shown on the right of Fig. 1), then the translated set $C_1+\varvec{u}$ is not a subset of the original $C_1$. Therefore, such a vector cannot be a direction of recession.

We now argue for the general case that $0^+(C_1)=C$ holds.

To show this, let $\varvec{u} \in C$ (a vector with nonnegative entries) and let $\varvec{v}$ be any vector of $C_1$, i.e., a vector with nonnegative entries satisfying $\varvec{v}^T\varvec{1}\ge 1$. Then, the sum $\varvec{u}+\varvec{v}$ satisfies

$$\begin{aligned} (\varvec{u}+\varvec{v})^T\varvec{1}=\varvec{u}^T \varvec{1}+\varvec{v}^T\varvec{1}\ge 0 + 1=1. \end{aligned}$$

Therefore, $\varvec{u}+\varvec{v} \in C_1$ and as $\varvec{v}$ was an arbitrary vector of $C_1$ it follows that $\varvec{u}+C_1 \subset C_1$. As this holds for any $\varvec{u} \in C$, the inclusion

$$\begin{aligned} C \subset 0^+(C_1) \end{aligned}$$

is established. Now, let $\varvec{u} \notin C$, then $u_i <0$ for some component i. If we take for some $j \ne i$ $\varvec{v}:=\varvec{e}_j$ (the jth unit vector), then $\varvec{v} \in C_1$ but $(\varvec{u}+\varvec{v})$ is negative with respect to the ith component. Therefore, $(\varvec{u}+\varvec{v}) \notin C$ and a fortiori $(\varvec{u}+\varvec{v}) \notin C_1$ as $C_1 \subset C$. Therefore, the recession cone $0^+(C_1)$ does not contain any vectors $\varvec{u} \notin C$.

The derivation of the main theorem (Theorem 1) relies on a result due to Rockafellar (1970). The original theorem refers to general convex functions (covering also the case that the functions under considerations are only defined on a subset of ${\mathbb {R}}^k$ and are not necessarily continuous). We only need a special case of it, wherein all the functions are in fact linear. More specifically, during the course of the proof of the main result, we only invoke the subsequent theorem for the linear components $f_i(\varvec{u})$ of the estimate $f(\varvec{u})$ of the latent abilities (and with $C_1$ acting as the convex set D).

Theorem 5

(see also Theorem 21.3, Rockafellar, 1970) Let $\{f_i| i \in I\}$ be a collection of finite convex functions on ${\mathbb {R}}^k$, where I is an arbitrary index set. Let D be any nonempty closed convex set in ${\mathbb {R}}^k$. Assume the functions $f_i$ have no common direction of recession which is also a direction of recession of D. Then, one and only one of the following alternatives holds:

• (a) There exists a vector $\varvec{u} \in D$ such that

  $$\begin{aligned} f_i(\varvec{u})\le 0, \qquad \forall i \in I \end{aligned}$$

• (b) There exist nonnegative real numbers $\lambda _i$, only finitely many nonzero, such that, for some $\epsilon >0$, one has

  $$\begin{aligned} \sum _{i \in I} \lambda _i f_i(\varvec{u})\ge \epsilon , \qquad \forall \varvec{u} \in D. \end{aligned}$$

The proof of Theorem 1 basically rests on case (b) established for linear functions and the convex set $C_1$. With these specifications, case (b) implies the existence of nonnegative weights such that the composite $\sum _{i} \lambda _i f_i(\varvec{u})$ scores any nonnegative vector whose sum exceeds one positively. This is almost the result we aim for! The details of the proof of Theorem 1 indicate, how the above argument can be made rigorous and how this result implies the existence of a fair composite.

Appendix C: Paradoxical Results Within the Bifactor Model

In Sect. 5, the special case of a bifactor model of compensatory type was introduced to demonstrate the computation of the scoring method. For the specific data set (WAIS-IV), it has been highlighted that every manifest variable induces a paradoxical effect. The purpose of this appendix is to show that this result holds not only for this special data set but under bifactor models of quite general type. As a byproduct of the subsequent derivation, we will highlight a general purpose method which settles the question as to whether a specific manifest variable induces paradoxical effect.

The derivation relies on the following result—also known as Farkas Lemma (see, e.g., Lemma 6.45 of Rockafellar & Wets, 2009).

Lemma 1

(Farkas). Let $\varvec{b} \in {\mathbb {R}}^p$ and let B be a $(k \times p)$-Matrix. An inequality $\varvec{b}^T \varvec{u}\le 0$ is a consequence of the system

$$\begin{aligned} B \varvec{u} \le \varvec{0}, \end{aligned}$$

(5)

if and only if $\varvec{b}$ can be expressed as a nonnegative linear combination of the rows of B.

The following observation provides the motivation for the usage of Farkas Lemma for the examination of variable-specific paradoxical effects (we restrict ourselves to the ordinary least square estimator—but the method generalizes): an increase in the score of the ith manifest variable by one unit changes the estimates for the latent abilities by the expression $(A^TA)^{-1}A^T\varvec{e}_i$. Therefore, item i does not induce paradoxical scoring effects if and only if

$$\begin{aligned} (A^TA)^{-1}A^T\varvec{e}_i \ge \varvec{0} \end{aligned}$$

holds. Note that $A^T\varvec{e}_i$ equals the ith row of A, denoted as $\varvec{a}_i$, and note also that the vector $\varvec{x}:=(A^TA)^{-1}A^T\varvec{e}_i=(A^TA)^{-1}\varvec{a}_i$ is the unique solution of the linear equation $A^TA\varvec{x}=\varvec{a}_i$. Therefore, the above condition can equally be rephrased as such: Item i does not induce paradoxical scoring effects if and only if there exists a nonnegative solution $\varvec{x}$ to the equation $A^TA\varvec{x}=\varvec{a}_i$. The latter means that the vector $\varvec{a}_i$ is a nonnegative linear combination of the columns of the matrix $A^TA$. Due to the symmetry of the matrix, this is equivalent to $\varvec{a}_i$ being expressible as a nonnegative linear combination of the rows of the matrix $A^TA$ - which in turn, according to Farkas Lemma, is equivalent to the following: Whenever a vector $\varvec{u}$ satisfies $A^TA\varvec{u}\le \varvec{0}$, it also satisfies $\varvec{a}^T_i\varvec{u}\le 0$. Therefore, a paradoxical effect with respect to an increase in the ith item score will be established, if there exists a vector $\varvec{u}$ satisfying $A^TA\varvec{u}\le \varvec{0}$ and $\varvec{a}^T_i\varvec{u}> 0$.

1.1 Application of Farkas Lemma to the Bifactor Model

Mimicking the concrete data example (WAIS-IV), we will argue for a five-dimensional bifactor model. Without loss of generality, let the ith manifest variable measure the second dimension (i.e., the ith vector of factor loadings has two nonzero entries—one corresponding to the general factor and one corresponding to the second dimension). Let $\varvec{u}=(u_1,1,u_3,u_4,u_5), \varvec{v}=(v_1,-1,v_3,v_4,v_5)$ be two vectors. We will examine under what conditions on the components of these vectors the inequalities $A^TA\varvec{u}\le \varvec{0}$ and $\varvec{a}^T_i\varvec{u}> 0$ (or the same inequalities with the vector $\varvec{v}$) hold. We have

$$\begin{aligned} \varvec{u}^T\varvec{a}_i=u_1 a_{i1}+a_{i2}. \end{aligned}$$

Hence, choosing $u_1 > -\frac{a_{i2}}{a_{i1}}$ will ensure $\varvec{a}^T_i\varvec{u}> 0$. Likewise, for $\varvec{v}$ we can choose $v_1>\frac{a_{i2}}{a_{i1}}$ and thus ensure $\varvec{v}^T\varvec{a}_i>0$. We will now argue that either $\varvec{u}$ or $\varvec{v}$ satisfies for appropriately chosen entries the inequalities $A^TA\varvec{u}\le \varvec{0}$ (or $A^TA\varvec{v}\le \varvec{0}$)—establishing a paradoxical result according to the preceding reasoning via Farkas Lemma.

We first consider the second component of the vector $A^TA\varvec{u}$. Denoting $u_1=-\frac{a_{i2}}{a_{i1}}+\epsilon$, the latter is

$$\begin{aligned} \sum _{j} a_{j2}(u_1 a_{j1}+a_{j2})=\sum _{j}a_{j2}\left( -\frac{a_{i2}}{a_{i1}}a_{j1}+\epsilon a_{j1}+a_{j2}\right) \end{aligned}$$

The last expression will be nonpositive if and only if (multiply both sides by $a_{i1}$)

$$\begin{aligned} \sum _{j}a_{j2}(a_{j2}a_{i1}-a_{i2}a_{j1}+\epsilon a_{j1}a_{i1})\le 0 \end{aligned}$$

It will be possible to choose $\epsilon >0$ such that the above inequality holds if and only if for some $c<0$

$$\begin{aligned} \sum _{j}a_{j2}(a_{j2}a_{i1}-a_{i2}a_{j1})\le c \end{aligned}$$

(6)

holds. Note that the expression on the left simplifies using vector and inner product notation. Indeed, if $a_{(v)}$ denotes the vth column of the matrix of factor loadings, then the above inequality may be expressed as:

$$\begin{aligned} a_{i1}(a^T_{(2)}a_{(2)})-a_{i2}(a^T_{(1)}a_{(2)})\le c<0 \end{aligned}$$

The same derivation applied to $\varvec{v}$ with $v_1:=\frac{a_{i2}}{a_{i1}}+\epsilon$ shows that

$$\begin{aligned} a_{i1}(a^T_{(2)}a_{(2)})-a_{i2}(a^T_{(1)}a_{(2)})\ge c>0 \end{aligned}$$

is sufficient for ensuring a nonpositive second component of the vector $A^TA\varvec{v}$. Thus, unless $a_{i1}(a^T_{(2)}a_{(2)})-a_{i2}(a^T_{(1)}a_{(2)})=0$ holds, it is always possible to define a vector $\varvec{w}$ such that the second component of $A^TA\varvec{w}$ is nonpositive while $\varvec{w}^T\varvec{a}_i>0$ holds. Note that the chosen values for the remaining components ($u_3,u_4,u_5$) of the vectors $\varvec{u}, \varvec{v}$ do not alter the above derivation. Hence, if we choose large negative values for each of these components, the above conclusions remain valid. Moreover, the remaining components of $A^TA\varvec{w}$ are then guaranteed to be nonpositive as well. Hence, unless $a_{i1}(a^T_{(2)}a_{(2)})-a_{i2}(a^T_{(1)}a_{(2)})=0$ holds, we can always find a vector $\varvec{w}$ (which equals either $\varvec{u}$ or $\varvec{v}$ with appropriately chosen large negative values for the third, fourth and fifth component) such that $\varvec{w}^T \varvec{a}_i>0$ and $A^TA\varvec{w}\le \varvec{0}$ holds. The latter implies according to Farkas Lemma that there will be at least one paradoxically scored dimension with respect to changes in the response to the ith manifest variable.

Appendix D: Bias and Inconsistency of Simple Sumscores

It is common practice to derive estimators of latent abilities based on the following method: The matrix of factor loadings is inspected and based on some cutoff (e.g., $\tau =.3$) it is decided to include a manifest variable for the inference of the underlying latent ability if the corresponding factor loading exceeds the cutoff. Based on the included variables, an unweighted sumscore is computed to infer the underlying latent ability. This procedure—though commonly applied—does not lead to reasonable estimates. We shall show this by a counterexample which can be extended to include more general settings. (We only use this to keep the notation simple.)

Suppose the scale is two-dimensional and of compensatory type. Let a sequence of manifest variables with corresponding factor loadings $a_{i1},a_{i2}$ and variances $\sigma ^2$ of the unique variables be given. The corresponding unscaled sequence of estimators based on the sumscore rule equals

$$\begin{aligned} \varvec{I}^T_{n}\varvec{u}_n \sim N(\varvec{I}^T_n A_n \varvec{\theta }, \varvec{I}^T_n \varvec{I}_n \sigma ^2), \end{aligned}$$

wherein $\varvec{I_n}$ is a (nonrandom) vector of binary entries, whose ith entry equals 1 if $a_{i1}>0.3$ holds.

The bias of a corresponding sequence of scaled estimators equals

$$\begin{aligned} {\mathbb {E}}\left( \frac{\varvec{I}^T_{n}\varvec{u}_n}{b_n}\right) -\theta _1=b^{-1}_n\sum _{i: a_{i1}>0.3}(a_{i1}\theta _1+a_{i2}\theta _2)-\theta _1 \end{aligned}$$

In order for the bias to vanish as n goes to infinity, it is necessary that

$$\begin{aligned} \lim _{n \rightarrow \infty } \frac{\sum _{i: a_{i1}>0.3}^na_{i1}}{b_n}=1 \end{aligned}$$

(7)

However, it is also necessary that the coefficient of $\theta _2$ in the above equation for the bias vanishes asymptotically, i.e.,

$$\begin{aligned} \lim _{n \rightarrow \infty } \frac{\sum _{i: a_{i1}>0.3}^na_{i2}}{b_n}=0 \end{aligned}$$

Assuming condition (7) to hold, this last condition will, however, be violated whenever

$$\begin{aligned} a_{i2}\ge \frac{a_{i1}}{k} \end{aligned}$$

for some k. If, for example, the factor loadings corresponding to the second dimension are at least one tenth ($k=10$) of the factor loadings of the first dimension, then the bias does not vanish. Perhaps even more importantly: The bias depends on the value of the second dimension.

Remark

Using the fact that the sequence $y_i:=u_i-\varvec{a}^T_i\varvec{\theta }$ is a stationary sequence (in the wide sense), it can also be deduced that the sumscore estimator is not a consistent estimator for $\theta _1$ either.