A Simple Method for Testing Global and Individual Hypotheses Involving a Limited Number of Possibly Correlated Outcomes

Abstract

Tests for the presence of a global treatment effect expressed by possibly correlated “primary” outcome variables taken together frequently use Bonferroni-type adjustments. These procedures accommodate an arbitrary number of comparisons, but can be conservative if the outcome variables are highly correlated. This conservatism can be ameliorated by a simple rule requiring essentially no calculation (and therefore convenient to apply when exact calculation is impractical) that is relatively robust to the correlation structure of the responses when the number of comparisons is not large (16 or less for 5 % level tests). The recommended global testing rule is: For a type 1 error rate of α and up to K(α) “primary” response variables, reject the global null hypothesis if (a) the smallest marginal p value is slightly less than α ₁  = α/K, (b) the second smallest marginal p value is ≤ 2α ₁, or (c) the third smallest marginal p value is ≤ α. Analytic expressions that do not assume independence or any particular distribution for the responses are provided for the probability of rejecting the global null hypothesis. The type 1 error rates and power generally are preserved regardless of the correlation structure. Individual comparisons can be tested if the global null hypothesis is rejected, with reasonable preservation of comparison-wise type 1 error rates and of the false discovery rates (FDRs).

Appendices

Appendix 1 Technical Details

1.1 A1.1 Acceptance Sets

Let X_i denote the i-th of K measures of the effect of an intervention obtained from a trial, with marginal cumulative distribution function (cdf) ${{\text{F}}_{\text{i}}}\left( \text{x;}\,{{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}} \right)$, where θ_i characterizes the intervention effect. The null hypothesis of no intervention effect with respect to the i-th measure X_i is H_0i: ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}={{\text{ }\!\!\theta\!\!\text{ }}_{\text{i0}}}$ and the alternative is H_1i: ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}\ne {{\text{ }\!\!\theta\!\!\text{ }}_{\text{i0}}}$. These could be expressed as one-sided hypotheses H_1i: ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}>{{\text{ }\!\!\theta\!\!\text{ }}_{\text{i0}}}$. The global null hypothesis of no overall intervention effect,  ${{\text{H}}_{0}}\,=\,\bigcap\nolimits_{\text{i}\,=\,1}^{\text{K}}{{{\text{H}}_{0\text{i}}}}$ is false if any individual null hypothesis is false. Let p_i denote the usual p value (unadjusted for multiplicity) calculated for testing H_0i, $\text{i=1}$, …, K so that H_0i would be rejected at the 100α% level of significance if p_i < α when multiplicity is ignored. Denote the ordered values of p₁, …, p_K by ${{\text{p}}_{\left( 1 \right)}}\le {{\text{p}}_{\left( 2 \right)}}\le \ldots \le {{\text{p}}_{\left( \text{K} \right)}}$

Let ${{\alpha }_{1}}\le {{\alpha }_{2}}\le \ldots \,\,{{\alpha }_{\text{K}}}$ denote a set of adjusted Type 1 error rates for the ordered p values, and let A_i ^(h) denote the set of realizations of X_i for which H_0i: ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}=\text{ }\!\!\theta\!\!\text{ }_{\text{i}}^{\left( 0 \right)}\,$ would not be rejected at the 100α_h% level of significance, that is, for which ${{\text{p}}_{\text{i}}}>{{\alpha }_{\text{h,}}}\,\text{i=1}$, …, K. A_i ^(h) is the “acceptance set” for measure X_i when the null hypothesis H_0i is tested at level α_h. For positive integers h and h΄ between 1 and K,

$$\begin{aligned} & \text{h}\,<\,{{\rm h}'}\ \Rightarrow \ (\text{i})\ {{\alpha }_{\rm h}}\,\le \,{{\alpha }_{\rm h'}} \\ & (\text{ii)}\ {{\text{A}}_{\text{i}}}^{({{\rm h}')}}\,\subset \,{{\text{A}}_{\text{i}}}^{(\text{h})} \\ & (\text{iii})\ {{\text{A}}_{\text{i}}}^{({\rm h})}{{\text{A}}_{\text{i}}}{{^{({\rm h }')}}}={{\text{A}}_{\text{i}}}^{(\text{h})}\,\bigcap \,{{\text{A}}_{\text{i}}}{{^{({\rm h}')}}}={{\text{A}}_{\text{i}}}{{^{(\text{h }\!\!'\!\!\text{ )}}}} \\ & (\text{iv})\ {{\text{A}}_{\text{i}}}^{(\text{h})}\,\cup \,{{\text{A}}_{\text{i}}}^{(\text{h }\!\!'\!\!\text{ )}}\,=\,{{\text{A}}_{\text{i}}}^{(\text{h})} \\ \end{aligned}$$

(A1)

For notational convenience,

$${{\text{A}}_{\text{i}}}^{(\text{h})}\,\cup \,{{\text{A}}_{\text{i}}}^{(\text{h }\!\!'\!\!\text{ )}}\,=\,{{\text{A}}_{\text{i}}}^{(\text{h})}$$

$\text{A}_{-\text{j}}^{\left( \text{h} \right)}\,\equiv \,\text{ }\!\!~\!\!\text{ }\bigcup\nolimits_{\begin{matrix} i\,=\,1, \\ i\ne j \\\end{matrix}}^{\text{K}}{\text{A}_{\text{i}}^{\left( \text{h} \right)}}$

and, in general,

$$\text{A}_{-{{\text{j}}_{1}}{{\text{j}}_{2}}\cdots }^{\left(\text{h} \right)}\,\equiv \,\text{ }\!\!~\!\!\text{}\bigcap\nolimits_{\begin{aligned} \text{i}\,=\,1, \\\text{i}\,\ne \,{{\text{j}}_{1}}{{\text{j}}_{2}}\cdots \\\end{aligned}}^{\text{K}}{\text{A}_{\text{i}}^{(\text{h})}}$$

A^(h) is the set of outcomes such that all of the p values exceed α_h and ~A^(h) denotes its complement.

1.2 A1.2 Rejection Regions

Denote by

$${{\text{S}}_{\text{1}}}\,=\,\tilde{\ }{{\text{A}}^{(\text{1})}}$$

the set of outcomes for which ${{\text{p}}_{\left( 1 \right)}}\le {{\alpha }_{1}}$. S₁ is the set of outcomes among X₁, …, X_K for which at least one of the component hypotheses H₀₁, …, H_0K would be rejected at the 100α₁ % level. If ${{\alpha }_{1}}=\alpha$, the nominal type 1 error rate, then controlling the FWER at α requires P(S₁ | H₀) ≤ α, which implies that ${{\alpha }_{1}}\le 1-{{(1-\alpha )}^{1/K}}$ = α_S if the outcomes are independent. The Bonferroni approach replaces α_S with ${{\alpha }_{\text{B}}}=\alpha /\text{K}{{\alpha }_{\text{S}}}$.

Let

$${{\text{S}}_{2}}\,=\,\tilde{\ }\bigcup\nolimits_{\text{i}\,=\,1}^{\text{K}}{\text{A}_{-\text{i}}^{(2)}}$$

denote the set of outcomes X₁, …, X_K for which p₍₂₎ ≤ α₂, and let

$${{\text{S}}_{3}}\,=\,\tilde{\ }\mathop{\bigcup{\bigcup\nolimits_{{{\text{i}}_{1}}\,<\,{{\text{i}}_{2}}}{\text{A}_{-{{\text{i}}_{1}}{{\text{i}}_{2}}}^{(3)}}}}^{}$$

denote the set of outcomes for which p₍₃₎ < α₃. S₃ is the set of outcomes for which at least three of the component null hypotheses are rejected at the 100α_3 % level.

Lemma: The set of outcomes for which the global null hypothesis will be rejected if ${{\text{p}}_{\left( 1 \right)}}<{{\alpha }_{1}}$ or ${{\text{p}}_{\left( 2 \right)}}<{{\alpha }_{2}}$ or ${{\text{p}}_{\left( 3 \right)}}<{{\alpha }_{3}}$ is defined in terms of the acceptance sets by

$${{\text{S}}_{1}}\cup {{\text{S}}_{2}}\cup {{\text{S}}_{3}}\,=\,\tilde{\ }\bigcup{\bigcup\nolimits_{{{\text{i}}_{1}}\,<\,{{\text{i}}_{2}}}{\left\{ \text{A}_{{{\text{i}}_{1}}}^{\left( 1 \right)}\text{A}_{{{\text{i}}_{2}}}^{\left( 2 \right)}\mathop{\cup }^{}\text{A}_{{{\text{i}}_{1}}}^{\left( 2 \right)}\text{A}_{{{\text{i}}_{2}}}^{\left( 1 \right)} \right\}\text{A}_{-{{\text{i}}_{1}}{{\text{i}}_{2}}}^{\left( 3 \right)}\,=\,\tilde{\ }\bigcup\nolimits_{\text{i}\,=\,1}^{{{\text{C}}_{\text{K},2}}}{{{\text{E}}_{\text{i}}}}}}$$

(A2)

where ${{\text{C}}_{\text{K},\text{2}}}\,=\,\text{K}\left( \text{K}-\text{1} \right)/\text{2}$.

Proof:

Repeated application of relationship (iii) of (A1) yields:

$$~\sim ({{S}_{1}}\cup {{S}_{2}})\,=\,~\sim {{S}_{1}}\cap ~\sim {{S}_{2}}\,=\,\bigcup\nolimits_{\text{i}\,=\,1}^{\text{K}}{\left(\bigcap\nolimits_{\text{i}\,=\,1}^{\text{K}}{\text{A}_{\text{i}}^{\text{(1)}}}\right)}\cap \left( \bigcap\nolimits_{\text{j}\,\ne \,\text{i}}{A_{\text{j}}^{(2)}}\right)$$

$$=\bigcup\nolimits_{\text{i}\,=\,1}^{\text{K}}{{}}_{{}}^{{}} \left( \text{A}_{\text{i}}^{(1)}\cap \left( \bigcap\nolimits_{\text{j}\,\ne \,\text{i}}{\text{A}_{\text{j}}^{(1)}\text{A}_{\text{j}}^{(2)}}\right)\right)$$

$$=\,\bigcup\nolimits_{\text{i}\,=\,1}^{\text{K}}{(\text{A}_{\text{i}}^{\left( 1 \right)}\cap (\bigcap\nolimits_{\text{j}\ne \text{i}}{\text{A}_{\text{j}}^{\left( 2 \right)}}))\,=\,\bigcup\nolimits_{\text{i}\,\text{=1}}^{\text{K}}{\text{A}_{\text{i}}^{(1)}\text{A}_{-\text{i}}^{(2)}}}$$

Expression (A2) follows from

$$\text{ }\!\!~\!\!\text{ }\tilde{\ }({{\text{S}}_{1}}\cup {{\text{S}}_{2}}\cup {{\text{S}}_{3}})\,=\,\sim {{\text{S}}_{1}}\cap \tilde{\ }{{\text{S}}_{2}}\cap \tilde{\ }{{\text{S}}_{3}}$$

$$=\,(\bigcup\nolimits_{\text{i}\,=\,1}^{\text{K}}{\text{A}_{\text{i}}^{\left( 1 \right)}\text{A}_{-\text{i}}^{\left( 2 \right)}})\cap (\bigcup{\bigcup\nolimits_{{{\text{i}}_{1}}\,<2{{\text{i}}_{2}}}{\text{A}_{-{{\text{i}}_{1}}{{\text{i}}_{2}}}^{\left( 3 \right)}}})$$

$$=\,(\text{A}_{{{\text{i}}_{1}}}^{\left( 1 \right)}\text{A}_{-{{\text{i}}_{1}}}^{\left( 2 \right)}\cup \text{A}_{{{\text{i}}_{2}}}^{\left( 1 \right)}\text{A}_{-{{\text{i}}_{2}}}^{\left( 2 \right)}\cup (\bigcup\nolimits_{\text{i}\ne {{\text{i}}_{1}},{{\text{i}}_{2}}}{\text{A}_{\text{i}}^{\left( 1 \right)}\text{A}_{-\text{i}}^{\left( 2 \right)}}))\cap (\bigcup{\bigcup\nolimits_{{{\text{i}}_{1}}\,<\,{{\text{i}}_{2}}}{\text{A}_{-{{\text{i}}_{1}}{{\text{i}}_{2}}}^{\left( 3 \right)}}})$$

$$=\,\bigcup{\bigcup\nolimits_{{{\text{i}}_{1}}\,<{{\text{i}}_{2}}}{(\text{A}_{{{\text{i}}_{1}}}^{\left( 1 \right)}\text{A}_{-{{\text{i}}_{1}}}^{\left( 2 \right)}\mathop{\cup }^{}\text{A}_{{{\text{i}}_{2}}}^{\left( 1 \right)}\text{A}_{-{{\text{i}}_{2}}}^{\left( 2 \right)}\mathop{\cup }^{}\left( \bigcup\nolimits_{\text{i}\ne {{\text{i}}_{1}},{{\text{i}}_{2}}}{\text{A}_{\text{i}}^{\left( 1 \right)}\text{A}_{-\text{i}}^{\left( 2 \right)}} \right))}}\cap \text{A}_{-{{\text{i}}_{1}}{{\text{i}}_{2}}}^{\left( 3 \right)}$$

$$=\,\bigcup{\bigcup\nolimits_{{{\text{i}}_{1}}\,<{{\text{i}}_{2}}}{\left( \text{A}_{{{\text{i}}_{1}}}^{\left( 1 \right)}\text{A}_{{{\text{i}}_{2}}}^{\left( 2 \right)}\cup \text{A}_{{{\text{i}}_{1}}}^{\left( 2 \right)}\text{A}_{{{\text{i}}_{2}}}^{\left( 1 \right)}\cup \text{A}_{{{\text{i}}_{1}}}^{\left( 2 \right)}\text{A}_{{{\text{i}}_{2}}}^{\left( 2 \right)} \right)}}\cap \text{A}_{-{{\text{i}}_{1}}{{\text{i}}_{2}}}^{\left( 3 \right)}$$

from relationship (iii) of (A1)

$$=\,\bigcup{\bigcup\nolimits_{{{\text{i}}_{1}}\,<{{\text{i}}_{2}}} {\left( \text{A}_{{{\text{i}}_{1}}}^{\left( 1 \right)}\text{A}_{{{\text{i}}_{2}}}^{\left( 2 \right)}\cup \text{A}_{{{\text{i}}_{1}}}^{\left( 2 \right)}\text{A}_{{{\text{i}}_{2}}}^{\left( 1 \right)} \right)}}\cap \text{A}_{-{{\text{i}}_{1}}{{\text{i}}_{2}}}^{\left( 3 \right)}$$

from relationship (iv) of (A1)QED

1.3 A1.3 Probabilities Associated with Rejection Regions

In general (Feller 1957, p. 89), the probability associated with the events E_i in (A2) is given by

$$\text{P}\left( {{\text{S}}_{1}}\cup {{\text{S}}_{2}}\cup {{\text{S}}_{3}} \right)\,=\,1-\text{P}\left( \bigcup\nolimits_{\text{i}\,=\,1}^{{{\text{C}}_{\text{K},2}}}{{{\text{E}}_{\text{i}}}} \right)$$

(A3)

$$=\,1\,-\,\left\{ \underset{\text{i}\,=\,1}{\overset{{{\text{C}}_{\text{K},2}}}{\mathop \sum }}\,\text{P}\left( {{\text{E}}_{\text{i}}} \right)-\underset{\text{h}\,=\,2}{\overset{{{\text{C}}_{\text{K},2}}}{\mathop \sum }}\,{{\left( -1 \right)}^{\text{h}}}\underset{{{\text{i}}_{1}}\,<-}{\mathop \sum }\,\underset{{{\text{i}}_{2}}\,<-}{\mathop \sum }\,\cdots \underset{{{\text{i}}_{\text{h}}}}{\mathop \sum }\,\text{P}({{\text{E}}_{{{\text{i}}_{1}}}}{{\text{E}}_{{{\text{i}}_{2}}}}\cdots {{\text{E}}_{{{\text{i}}_{\text{h}}}}}) \right\}$$

Expressions for the joint probabilities in (A3) simplify appreciably because of the relationships among the acceptance sets. The general result is given in the following.

Theorem:

$$\text{P}\left( {{\text{S}}_{1}}\cup {{\text{S}}_{2}}\cup {{\text{S}}_{3}} \right)$$

(A4)

$\, = \,1 - \left\{ {\sum\nolimits_{{\rm{i}}\, = \,1}^{{{\rm{C}}_{{\rm{K}},2}}} {{\rm{P}}\left( {{{\rm{E}}_{\rm{i}}}} \right) - \left( {{\rm{K}} - 2} \right)} \mathop {\;\sum\nolimits_{{\rm{j}}\, = \,1}^{\rm{K}} {{\rm{P}}\left( {{\rm{A}}_{\rm{j}}^{\left( 1 \right)}{\rm{A}}_{ - {\rm{j}}}^{\left( 3 \right)}} \right)} }\limits_{}^{} \, + \,{{\rm{C}}_{{\rm{K}} - 1,2}}{\rm{P}}\left( {{{\rm{A}}^{\left( 3 \right)}}} \right){\rm{}}} \right\}$

where the E_i are defined by (A2).

Proof:

From the Lemma, and the fact that P(A$\mathop{\cup }^{}$B) = P(A) + P(B)—P(AB),

$$\text{P}\left( {{\text{E}}_{\text{i}}} \right)\,=\,\text{P}\left( \text{A}_{{{\text{i}}_{1}}}^{\left( 1 \right)}\text{A}_{{{\text{i}}_{2}}}^{\left( 2 \right)}\text{A}_{-{{\text{i}}_{1}}{{\text{i}}_{2}}}^{\left( 3 \right)} \right)\,+\,\text{P}\left( \text{A}_{{{\text{i}}_{1}}}^{\left( 2 \right)}\text{A}_{{{\text{i}}_{2}}}^{\left( 1 \right)}\text{A}_{-{{\text{i}}_{1}}{{\text{i}}_{2}}}^{\left( 3 \right)} \right)\,-\,\text{P}\left( \text{A}_{{{\text{i}}_{1}}}^{\left( 2 \right)}\text{A}_{{{\text{i}}_{2}}}^{\left( 2 \right)}\text{A}_{-{{\text{i}}_{1}}{{\text{i}}_{2}}}^{\left( 3 \right)} \right)$$

A typical product E_iE_j can be written as

$${{\text{E}}_{\text{i}}}{{\text{E}}_{\text{j}}}=\left( \text{A}_{{{\text{i}}_{\text{1}}}}^{\text{(1)}}\text{A}_{{{\text{i}}_{\text{2}}}}^{\text{(2)}}\cup \text{A}_{{{\text{i}}_{\text{1}}}}^{\text{(2)}}\text{A}_{{{\text{i}}_{\text{2}}}}^{\text{(1)}} \right)\ \left( \text{A}_{{{\text{i}}_{\text{3}}}}^{\text{(1)}}\text{A}_{{{\text{i}}_{\text{4}}}}^{\text{(2)}}\cup \text{A}_{{{\text{i}}_{\text{3}}}}^{\text{(2)}}\text{A}_{{{\text{i}}_{\text{4}}}}^{\text{(1)}} \right)\text{A}_{\text{-}{{\text{i}}_{\text{1}}}{{\text{i}}_{\text{2}}}}^{\text{(3)}}\text{A}_{\text{-}{{\text{i}}_{\text{3}}}{{\text{i}}_{\text{4}}}}^{\text{(3)}}$$

The pairs (i₁, i₂) and (i₃, i₄) are the index pairs of E_i and E_j, respectively. If i₁, i₂, i₃, and i₄ are four distinct integers, then E_iE_j = A⁽³⁾. Otherwise, if ${{\text{i}}_{1}}={{\text{i}}_{3}}=\text{k}$ or ${{\text{i}}_{2}}={{\text{i}}_{4}}=\text{k}$, then E_iE_j = $A_{\rm k}^{(1)}A_{-{\rm k}}^{(3)}$ Hence, P(E_iE_j) = Pr(A⁽³⁾) or P($A_{\rm k}^{(1)}A_{-{\rm k}}^{(3)}$) depending on whether E_i and E_j do not or do share a common index value. The product ${{E}_{{{\rm i}_{1}}}}\cdots {{E}_{{{\rm i}_{\rm h}}}}$  = A⁽³⁾ if the indices of the A sets for any two E factors consist of four distinct integers. Also, if k is one of the members of the index pair corresponding to each E_i of the product ${{E}_{{{\rm i}_{1}}}}\cdots {{E}_{{{\rm i}_{\rm h}}}}$, then the product is equal to $A_{\rm k}^{(1)}A_{-{\rm k}}^{(3)}$. Consequently, P(${{E}_{{{\rm i}_{1}}}}\cdots {{E}_{{{\rm i}_{\rm h}}}}$) = P(A⁽³⁾) or P($A_{\rm k}^{(1)}A_{-{\rm k}}^{(3)}$) accordingly as the factors of the product ${{E}_{{{\rm i}_{1}}}}\cdots {{E}_{{{\rm i}_{\rm h}}}}$ do not or do share a common index value. All told, there are $\left( \begin{matrix} {{C}_{{\rm K},2}} \\ {\rm h} \\\end{matrix} \right)$ distinct h-tuples ${{E}_{{{\rm i}_{1}}}}\cdots {{E}_{{{\rm i}_{\rm h}}}}$. As long as h < K, there are $\left( \begin{matrix} {\rm K}-1 \\ {\rm h} \\\end{matrix} \right)$ ways to choose h additional distinct indices to pair with any index value i to form ${{E}_{{{\rm i}_{1}}}}\cdots {{E}_{{{\rm i}_{\rm h}}}}$ products whose members’ index pairs all contain i. Consequently, the term P $(A_{\rm i}^{(1)}A_{-{\rm i}}^{(3)})$ occurs $\left( \begin{matrix} {\rm K}-1 \\ {\rm h} \\\end{matrix} \right)$ times in the sum $\sum\limits_{{{\rm i}_{1}}<}{\sum\limits_{{{\rm i}_{2}}<}{\cdots \sum\limits_{<{{\rm i}_{\rm h}}}{{\rm P}({{E}_{{{\rm i}_{1}}}}{{E}_{{{\rm i}_{2}}}}\cdots }}}{{E}_{{{\rm i}_{\rm h}}}})$ and this is true for each value of i, so there are K$K\left( {\begin{array}{*{20}{c}} {{\rm K} - 1} \\ {\rm h} \\ \end{array}}\right)$ such terms. The remaining $\left( {\begin{array}{*{20}{c}} {{C_{{\rm K},2}}} \\ {\rm h}\\ \end{array}} \right)$—K$\left( {\begin{array}{*{20}{c}} {{\rm K} - 1} \\ {\rm h} \\ \end{array}} \right)$ terms of the sum all equal P(A⁽³⁾). If h ≥ K, then all of the products ${{E}_{{{\rm i}_{1}}}}\cdots {{E}_{{{\rm i}_{\rm h}}}}$ must equal A⁽³⁾ and so P(A⁽³⁾) must occur $\left( {\begin{array}{*{20}{c}} {{C_{{\rm K},2}}} \\ {\rm h} \\ \end{array}} \right)$ times in the sum $\sum\limits_{{{\text{i}}_{\text{1}}}<}{\sum\limits_{{{\text{i}}_{2}}<}{\cdots \sum\limits_{<{{\text{i}}_{\text{h}}}}{\text{P}({{\text{E}}_{{{\text{i}}_{\text{1}}}}}{{\text{E}}_{{{\text{i}}_{2}}}}\cdots }}}{{\text{E}}_{{{\text{i}}_{\text{h}}}}})$. This completes the proof.

Expression (A4) does not require independence or continuity of the outcome variables. Computationally useful forms can be obtained by assuming independence, as in the following corollaries.

Corollary 1

If the outcome variables are independent and $\text{P}\left( \text{A}_{\text{i}}^{\left( \text{h} \right)} \right)\,=\,\text{p}_{\text{i}}^{\left( \text{h} \right)}$, then

$$\text{P}\left( {{\text{S}}_{1}}\cup {{\text{S}}_{2}}\cup {{\text{S}}_{3}} \right)$$

$$\,=\,1-\left\{ \begin{matrix} \sum\nolimits_{{{\text{i}}_{1}}\,<={{\text{i}}_{2}}}{\left( \text{p}_{{{\text{i}}_{1}}}^{\left( 1 \right)}\text{p}_{{{\text{i}}_{2}}}^{\left( 2 \right)}\,+\,\text{p}_{{{\text{i}}_{1}}}^{\left( 2 \right)}\text{p}_{{{\text{i}}_{2}}}^{\left( 1 \right)}-\text{p}_{{{\text{i}}_{1}}}^{\left( 2 \right)}\text{p}_{{{\text{i}}_{2}}}^{\left( 2 \right)} \right)}\prod\nolimits_{\text{j}\ne {{\text{i}}_{1}},{{\text{i}}_{2}}}{\text{p}_{\text{j}}^{\left( 3 \right)}} \\ -\left( K-2 \right)\sum\nolimits_{\text{i}\,=\,1}^{\text{K}}{\text{p}_{\text{i}}^{\left( 1 \right)}}\prod\nolimits_{\text{j}\ne \text{i}}{\text{p}_{\text{j}}^{\left( 3 \right)}}\,+\,{{\text{C}}_{\text{K}-1,2}}\prod\nolimits_{\text{i}\,=\,1}^{\text{K}}{\text{p}_{\text{i}}^{\left( 3 \right)}} \\\end{matrix} \right\}$$

(A5)

Corollary 2

If the outcome variables are independent and continuous, and all of the component null hypotheses are true, so that $\text{p}_{\text{i}}^{\left( \text{h} \right)}$ = 1—α_h, then the probability of rejecting the global null hypothesis is

$$\text{P}\left( {{\text{S}}_{1}}\cup {{\text{S}}_{2}}\cup {{\text{S}}_{3}} \right)$$

$$=\,\text{1} – {{{\text{C}}_{\text{K},\text{2}}}{{(\text{1}-{{\alpha }_{\text{3}}})}^{\text{K}-\text{2}}}(\text{1}-{{\alpha }_{\text{2}}})(\text{1}-\text{2}{{\alpha }_{\text{1}}}\,+\,{{\alpha }_{\text{2}}}) – \text{K}\left( \text{K}-\text{2} \right)(\text{1}-{{\alpha }_{\text{1}}}){{(\text{1}-{{\alpha }_{\text{3}}})}^{\text{K}-\text{1}}}\,+\,{{\text{C}}_{\text{K}-\text{1},\text{2}}}{{(\text{1}-{{\alpha }_{\text{3}}})}^{\text{K}}}}$$

This is the same as expression (3.3) of Sen (1999), when r = 3.

1.4 A1.4 Critical Values

Corollary 2 implies that the global null hypothesis test will have level at most α under independence and continuity if and only if

$$\text{f}({{\alpha }_{\text{1}}},{{\alpha }_{\text{2}}},{{\alpha }_{\text{3}}})\,=\,{{\text{C}}_{\text{K},\text{2}}}{{(\text{1}-{{\alpha }_{\text{3}}})}^{\text{K}-\text{2}}}(\text{1}-{{\alpha }_{\text{2}}})(\text{1}-\text{2}{{\alpha }_{\text{1}}}\,+\,{{\alpha }_{\text{2}}}) – \text{K}\left( \text{K}-\text{2} \right)(\text{1}-{{\alpha }_{\text{1}}}){{(\text{1}-{{\alpha }_{\text{3}}})}^{\text{K}-\text{1}\,}}\,+\,{{\text{C}}_{\text{K}-\text{1},\text{2}}}{{(\text{1}-{{\alpha }_{\text{3}}})}^{\text{K}}}\,\ge \,1\,-\,\alpha $$

(A6)

Given α₁, the maximum value of f in (A6) occurs when ${{\alpha }_{2}}={{\alpha }_{3}}={{\alpha }_{1}}$ (the derivative of f with respect to ${{\alpha }_{2}}$ is zero when ${{\alpha }_{2}}={{\alpha }_{1}}$; the derivative of f with respect to ${{\alpha }_{3}}$ is zero when ${{\alpha }_{3}}={{\alpha }_{1}}$ if ${{\alpha }_{2}}={{\alpha }_{1}}$). Inequality (A6) is satisfied if and only if ${{\alpha }_{1}}\le {{\alpha }_{\text{S}}}$. If ${{\alpha }_{1}}={{\alpha }_{\text{S}}}$, then f$\left( {{\alpha }_{1}},{{\alpha }_{2}},\,{{\alpha }_{3}}\, \right)=1\,\alpha$ so that neither ${{\alpha }_{2}}$ nor ${{\alpha }_{3}}$ can exceed ${{\alpha }_{\text{S}}}$ (Berger 1982). If ${{\alpha }_{1}}<{{\alpha }_{S}}$, which would be true if ${{\alpha }_{1}}<{{\alpha }_{\text{B}}}$, then ${{\alpha }_{2}}$ and ${{\alpha }_{3}}$ both can exceed ${{\alpha }_{1}}$. This is the key point. In particular, (A6) can be satisfied for ${{\alpha }_{3}}=\alpha$ and ${{\alpha }_{2}}=2{{\alpha }_{1}}$ as long as ${{\alpha }_{1}}\le {{\alpha }_{1\text{max}}}$, where

$${{\alpha }_{\text{1max}}}\,=$$

$$\frac{{{\text{C}}_{\text{K},2}}-\text{K}\left( \text{K}-2 \right)\left( 1-\text{ }\!\!\alpha\!\!\text{ } \right)\,+\,{{\text{C}}_{\text{K}-1,2}}{{\left( 1-\text{ }\!\!\alpha\!\!\text{ } \right)}^{2}}-{{\left( 1-\text{ }\!\!\alpha\!\!\text{ } \right)}^{3-\text{K}}}\text{ }\!\!~\!\!\text{ }}{\text{K}\left( \text{K}-1-\left( \text{K}-2 \right)\left( 1-\text{ }\!\!\alpha\!\!\text{ } \right) \right)}$$

It is easy to verify that ${{\alpha }_{1\text{max}}}>0$ when $\alpha =0.05$ as long as K ≤ K(0.05) = 16. Smaller values of α allow for greater values of $\text{K}\left( \alpha \right)$: K(0.025) = 25 and K(0.01) = 44. The value of ${{\alpha }_{1\text{max}}}$ is not much smaller than $\alpha /\text{K}$ when $\text{K}\le \text{10}$. Table 29.1 in Sect. 2 displays the values of ${{\alpha }_{1\text{max}}},\,\alpha /\text{K}$, and their difference for $\alpha =0.05$ and 0.25, and K = 3(1)16. The quantity $\text{ }\!\!\varepsilon\!\!\text{ }$ mentioned in the introduction is the difference between ${{\alpha }_{1\text{max}}}$ and ${{\alpha }_{\text{B}}}=\alpha /\text{K}$ .

1.5 A1.5 Power

The power and, therefore, the sample size needed, for rejecting a global null hypothesis will depend on the joint distribution of the outcomes under an alternative hypothesis. An alternative hypothesis could specify a constant shift for each component outcome such as ${{\text{H}}_{1\text{i}}}:\,{{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}={{\text{ }\!\!\theta\!\!\text{ }}_{\text{i0}}}$ for all i. Or, the alternative hypothesis could specify a shift with respect to some, but not all, of the component outcome distributions, so that the alternative hypothesis would be defined by ${{\text{H}}_{1\text{i}}}:\,{{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}={{\text{ }\!\!\theta\!\!\text{ }}_{\text{i1}}}\ne {{\text{ }\!\!\theta\!\!\text{ }}_{\text{i0}}}$ for i $\in$ {i ₁, …, i _m} ⊂ {1, …, K}, and ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}={{\text{ }\!\!\theta\!\!\text{ }}_{\text{i0}}}$ otherwise.

If the outcomes are independent, then the probability of rejecting the global null hypothesis when there is a shift in m (1 ≤ m ≤ K) of the component distributions, is, from (A5)

$$\text{P}\left( {{\text{S}}_{1}}\cup {{\text{S}}_{2}}\cup {{\text{S}}_{3}} \right)\,=\,1$$

(A7)

$$\left\{ {\begin{aligned} {\left[ {{{\rm{C}}_{{\rm{K}}- 1,2}}\gamma_3 - {\rm{m}}{{( {{\rm{K}} - 2})}\gamma_1}}\right]\gamma_3^{{\rm{m}} - 1}{{\left( {1{ -\alpha _3}}\right)}^{{\rm{K}} - {\rm{m}}}}} \\ { +{{\rm{C}}_{{\rm{m}},2}}{{\left( {{2\gamma_1}{ - \gamma_2}}\right)}\gamma_2}\gamma_3^{{\rm{m}} - 2}{{\left( {1{ - \alpha_3}}\right)}^{{\rm{K}} - {\rm{m}}}}\qquad \rm{m}> 1} \\ { + \left({{\rm{K}} - {\rm{m}}} \right)\left[ {\begin{aligned} {\left({{\rm{m}} - 1} \right)\left[ {\left( {\gamma_1{ - \gamma_2}}\right)\left( {1{ - \alpha_2}} \right){ + \gamma_2}\left( {1{ -\alpha_1}} \right)} \right]}\\ { - \left({{\rm{K}} - 2} \right){{\left( {1{ - \alpha_1}} \right)}\gamma_3}} \\\end{aligned}} \right]\gamma_3^{{\rm{m}} - 1}{{\left( {1{ - \alpha_3}}\right)}^{{\rm{K}} - {\rm{m}} - 1}}} \\ {m < K} \\ { +{{\rm{C}}_{{\rm{K}} - {\rm{m}},2}}\left( {1 - {2\alpha_1}{ +\alpha_2}} \right)\left( {1{ - \alpha_2}}\right)\gamma_3^{\rm{m}}{{\left( {1{- \alpha_3}} \right)}^{{\rm{K}} - {\rm{m}} - 2}}m < K - 1} \\\end{aligned}} \right\}$$

where ${{\text{ }\!\!\gamma\!\!\text{ }}_{\text{K}}}$ denotes the probability of a component event falling inside its level ${{\alpha }_{\text{K}}}$ acceptance set when H_1i is true. If H_0i is true, then ${{\text{ }\!\!\gamma\!\!\text{ }}_{\text{K}}}=1-\,{{\alpha }_{\text{K}}};$ if H_1i is true, then ${{\text{ }\!\!\gamma\!\!\text{ }}_{\text{K}}}$ denotes the corresponding type 2 error rate (assumed same for all components).

The functional form of F, the distribution generating the observations, is needed to calculate the type 2 error rates ${{\text{ }\!\!\gamma\!\!\text{ }}_{\text{i}}}$ in (A7) corresponding to the type 1 error rates ${{\alpha }_{\text{i}}},\,\text{i}=1,2,3$. Suppose the probabilities $\text{p}_{\text{i}}^{(\text{h})}$ in (A5) can be calculated from

$$\text{p}_{\text{i}}^{\text{(h)}}=\text{pr(A}_{\text{i}}^{\text{(h)}}\text{)}=\text{F(}{{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}+{{\zeta }_{\text{1-}{{\text{ }\!\!\alpha\!\!\text{ }}_{\text{h}}}/2}};\xi )-\text{F(}{{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}+{{\zeta }_{{{\alpha }_{h}}/2}};\xi )$$

(A8)

for two-sided tests of H_0i: ${{\theta }_{\text{i}}}=0$ vs ${{\text{H}}_{1\text{i}}}:\,|{{\theta }_{\text{i}}}|>0$, where F denotes an appropriate cumulative distribution function such as the standard normal, Student t, etc., $\xi$ denotes parameters with known values such as the degrees of freedom, ${{\theta }_{\text{i}}}\,\left( \ge 0 \right)$ denotes the expectation of the i-th component outcome under the alternative hypothesis, and the $\zeta$ are percentiles of the null distribution of the appropriate test statistic. For power calculations under independence, we want $\text{p}_{\text{i}}^{\text{(h)}}\,\le {{\text{ }\!\!\gamma\!\!\text{ }}_{\text{h}}}$ if ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}={{\text{ }\!\!\theta\!\!\text{ }}_{\text{i1}}}>0$ and $\text{p}_{\text{i}}^{\text{(h)}}$ ≥ 1 − α_h if θ_i = 0. The first term on the right-hand side of (A8) will be only slightly less than 1 when Δ_i > 0, so that the requirement $\text{p}_{\text{i}}^{\text{(h)}}\,\le {{\text{ }\!\!\gamma\!\!\text{ }}_{\text{h}}}$ if ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}={{\text{ }\!\!\theta\!\!\text{ }}_{\text{i1}}}>0$ implies that a slightly conservative estimate of ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i1}}}$ is

$${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}\,\cong \,{{\text{ }\!\!\zeta \!\!\text{ }}_{1-\text{ }\!\!\gamma\!\!\text{ }}}\,-\,{{\text{ }\!\!\zeta \!\!\text{ }}_{{{\alpha }_{\text{h/2}}}}}$$

(A9)

The value of ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}$ must be the same for all h. Consequently, if ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}$ is determined by ${{\text{ }\!\!\alpha\!\!\text{ }}_{\text{1}}}$ and ${{\text{ }\!\!\gamma\!\!\text{ }}_{\text{1}}}$ in (A9), then ${{\text{ }\!\!\gamma\!\!\text{ }}_{2}}$ and ${{\text{ }\!\!\gamma\!\!\text{ }}_{3}}$ must be determined from

$${{\text{ }\!\!\zeta \!\!\text{ }}_{1{{\text{ }\!\!\gamma\!\!\text{ }}_{\text{h}}}}}={{\text{ }\!\!\zeta \!\!\text{ }}_{1-{{\text{ }\!\!\gamma\!\!\text{ }}_{1}}}}\,-\,{{\text{ }\!\!\zeta \!\!\text{ }}_{{{\text{ }\!\!\alpha\!\!\text{ }}_{1}}/2}}\,+\,{{\text{ }\!\!\zeta \!\!\text{ }}_{{{\text{ }\!\!\alpha\!\!\text{ }}_{\text{h}}}/2}}$$

i.e., ${{\text{ }\!\!\gamma\!\!\text{ }}_{\text{h}}}\,-\,\text{1}\,\,\text{F}({{\text{ }\!\!\zeta \!\!\text{ }}_{1-{{\text{ }\!\!\gamma\!\!\text{ }}_{1}}}}\,-\,{{\text{ }\!\!\zeta \!\!\text{ }}_{{{\alpha }_{1}}/2}}\,\text{+}\,{{\text{ }\!\!\zeta \!\!\text{ }}_{{{\alpha }_{\text{h}}}/2}}\text{;}\ \text{ }\!\!\theta\!\!\text{ })$.

The quantity ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i1}}}$ is the value of the noncentrality parameter that gives power $\text{1-}{{\text{ }\!\!\gamma\!\!\text{ }}_{\text{1}}}$ for rejecting the i-th individual null hypothesis H_0i when ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}\text{=}{{\text{ }\!\!\theta\!\!\text{ }}_{\text{i1}}}$. It determines the required sample size through expressions such as ${{\text{ }\!\!\theta\!\!\text{ }}_{\text{i}}}={{\text{ }\!\!\mu\!\!\text{ }}_{\text{i}}}\surd \text{n/}\sigma$ when the values of ${{\text{ }\!\!\mu\!\!\text{ }}_{\text{i}}}$ and $\sigma$ are specified. Table 29.2 in Sect. 2 above provides noncentrality parameters values calculated assuming normality using (A9) for K = 3 (1) 16 and when m = 1 (1) min(5,K) = number of positive means.

1.6 A1.6 Confidence Sets

Let θ denote the parameters of the joint distribution of the K outcomes addressed by the null hypothesis H₀: $\text{ }\!\!\theta\!\!\text{ =}{{\text{ }\!\!\theta\!\!\text{ }}_{\text{0}}}$. H₀ is rejected at the 100α% level when Pr$\left( {{\text{S}}_{1}}\cup {{\text{S}}_{2}}\cup {{\text{S}}_{3}}\text{ }\!\!\theta\!\!\text{ =}{{\text{ }\!\!\theta\!\!\text{ }}_{0}} \right)\le \alpha$. A 100(1-α)% joint confidence region for $\text{ }\!\!\theta\!\!\text{ }$ consists of the parameter values for which H₀ would not be rejected, i.e., $\left\{ {{\text{ }\!\!\theta\!\!\text{ }}^{*}}|\text{P}\left( {{\text{S}}_{1}}\cup {{\text{S}}_{2}}\cup {{\text{S}}_{3}}|~{{\text{ }\!\!\theta\!\!\text{ }}^{\text{*}}} \right)\,\ge \,\text{ }\!\!\alpha\!\!\text{ }\!\!|\!\!\text{ } \right\}$(Lehmann 1959, Theorem 4, p. 79). The region resembles a notched hyper-rectangle when the outcomes are independent (Benjamini and Stark 1996).

Appendix 2 R Code for Simulations