Simultaneous Diagonalization Algorithms with Applications in Multivariate Statistics

Abstract

The following problem arises from multivariate statistical models in principal component and canonical correlation analysis. Let

$$S\, = \,\,\left[ \begin{gathered} {S_{11}}\,\,\,\,\, \cdots \,\,\,\,\,{S_{1k}}\, \hfill \\ \vdots \,\,\,\,\,\,\,\,\,\, \ddots \,\,\,\,\,\, \vdots \hfill \\ {S_{k1\,}}\,\,\,\, \cdots \,\,\,\,\,\,{S_{kk}} \hfill \\\end{gathered} \right]$$

denote a positive definite symmetric (pds) matrix of dimension pk × pk, partitioned into submatrices S _ij of dimension p × p each, and suppose we wish to find a nonsingular p × p matrix B such that all B′S _ij B are “almost diagonal”. More precisely, for a partitioned pk × pk matrix

$$A = \left[ \begin{gathered} {A_{11\,\,\,\,\,}} \cdots \,\,\,\,\,{A_{1k}} \hfill \\ \,\,\,\vdots \,\,\,\,\,\, \ddots \,\,\,\,\,\,\,\, \vdots \hfill \\ {A_{k1\,\,\,}}\, \cdots \,\,\,\,\,{A_{kk}} \hfill \\\end{gathered} \right]$$

we define the parallel-diagonal operator as

$$pdiag (A)= \left[ \begin{gathered} diag\left( {{A_{11}}} \right)\,\,\,\, \cdots \,\,\,\,diag\left( {{A_{1k}}} \right) \hfill \\ \,\,\,\,\,\,\,\,\,\,\,\,\vdots\,\,\,\,\,\,\,\,\,\,\, \ddots \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \hfill \\ diag\left( {{A_{k1}}} \right)\,\,\,\, \cdots \,\,\,\,diag\left( {{A_{kk}}} \right) \hfill \\ \end{gathered} \right]$$

and suggest to use det{pdiag(A)}/det(A) as a measure of deviation from “parallel-diagonality”, provided A is pds. For a nonsingular p × p matrix B, we study the function

$$\Phi \left( {B;{\kern 1pt} \,S} \right)\, = \,\frac{{\det \,\left[ {pdag\left\{ {{{\left( {{I_k} \otimes \,B} \right)}^\prime }S\,\left( {{I_k}\, \otimes \,B} \right)} \right\}} \right]}}{{\det \left[ {{{\left( {{I_k}\, \otimes \,B} \right)}^\prime }S\left( {{I_k}\, \otimes \,B} \right)} \right]}}$$

The matrix B which minimizes Ф is said to transform S to almost parallel-diagonal form. We give an algorithm for minimizing Ф over B in (i) the group of orthogonal p × p matrices, and (ii) the set of nonsingular p × p matrices such that diag(B′B) = I _p , and study its convergence. Statistical applications of the algorithm occur in maximum likelihood estimation of (i) common principal components for dependent random vectors (Neuenschwander 1994), and (ii) common canonical variates (Neuenschwander and Flury 1994). This work generalizes and extends the FG diagonalization algorithm of Flury and Gautschi (1986).

work supported by a grant from the Swiss National Science Foundation