Optimization algorithms on the Grassmann manifold with application to matrix eigenvalue problems

Abstract

This article deals with the Grassmann manifold as a submanifold of the matrix Euclidean space, that is, as the set of all orthogonal projection matrices of constant rank, and sets up several optimization algorithms in terms of such matrices. Interest will center on the steepest descent and Newton’s methods together with applications to matrix eigenvalue problems. It is shown that Newton’s equation in the proposed Newton’s method applied to the Rayleigh quotient minimization problem takes the form of a Lyapunov equation, for which an existing efficient algorithm can be applied, and thereby the present Newton’s method works efficiently. It is also shown that in case of degenerate eigenvalues the optimal solutions form a submanifold diffeomorphic to a Grassmann manifold of lower dimension. Furthermore, to generate globally converging sequences, this article provides a hybrid method composed of the steepest descent and Newton’s methods on the Grassmann manifold together with convergence analysis.

Appendix A: Proof of Proposition 2.4

In this section, we give the proof of Proposition 2.4 on the variational principle.

Proof

Since the geodesic equation of a Riemannian manifold is viewed as the equation of motion of a free particle on the Riemannian manifold [5], we consider the Lagrangian $L$ of a free particle on the Grassmann manifold (2.13), which is put in the form

$$\begin{aligned} L=\frac{1}{2}\langle {\dot{X}},{\dot{X}}\rangle _X+{\mathrm{tr}}\left( \Omega \left( X^2-X\right) \right) +\lambda ({\mathrm{tr}}(X)-p), \end{aligned}$$

(6.1)

where $\Omega$ and $\lambda$ are Lagrange multipliers, and where $\Omega$ should be a symmetric matrix on account of the fact that $X^2-X$ is symmetric. The variation of the Lagrangian $L$ is given by and calculated as

$$\begin{aligned} \delta L&={\mathrm{tr}}(\delta {\dot{X}}{\dot{X}})+{\mathrm{tr}}(\delta \Omega (X^2-X))\!+\!{\mathrm{tr}}(\Omega (2X\delta X-\delta X))\!+\!\delta \lambda ({\mathrm{tr}}(X)-p)+\lambda {\mathrm{tr}}(\delta X)\nonumber \\&=\frac{d}{dt}({\mathrm{tr}}(\delta X{\dot{X}}))+{\mathrm{tr}}(\delta X(-{\ddot{X}}+\Omega (2X-I_n)+\lambda I_n))+{\mathrm{tr}}(\delta \Omega (X^2-X))\nonumber \\&\quad +\delta \lambda ({\mathrm{tr}}(X)-p). \end{aligned}$$

(6.2)

On the variational principle, we have

$$\begin{aligned} \int _{t_1}^{t_2}\delta L\,dt=0 \end{aligned}$$

(6.3)

for any $\delta X$ subject to the condition $\delta X(t_1)=\delta X(t_2)=0$. From (6.2) and (6.3), we obtain

$$\begin{aligned} -{\ddot{X}}+\Omega (2X-I_n)+\lambda I_n=0,\end{aligned}$$

(6.4)

$$\begin{aligned} X^2-X=0,\end{aligned}$$

(6.5)

$$\begin{aligned} {\mathrm{tr}}(X)-p=0. \end{aligned}$$

(6.6)

Our next task is to determine $\Omega$ and $\lambda$. Transposing Eq. (6.4), we have

$$\begin{aligned} -\ddot{X}+(2X-I_n)\Omega +\lambda I_n=0. \end{aligned}$$

(6.7)

Equations (6.4) and (6.7) are put together to provide

$$\begin{aligned} X\Omega =\Omega X. \end{aligned}$$

(6.8)

Multiplying (6.4) by $X$ from the right and using $X^2=X$, we obtain

$$\begin{aligned} \Omega X={\ddot{X}} X-\lambda X. \end{aligned}$$

(6.9)

We see that ${\ddot{X}}X=X{\ddot{X}}$ from (6.8) and (6.9). Putting together (6.9) and (6.4), we have

$$\begin{aligned} \Omega =-{\ddot{X}}+2{\ddot{X}}X+\lambda (I_n-2X). \end{aligned}$$

(6.10)

On the other hand, differentiating $X^2=X$ with respect to $t$, we obtain

$$\begin{aligned} {\ddot{X}}=X{\ddot{X}}+2{\dot{X}}^2+{\ddot{X}}X. \end{aligned}$$

(6.11)

Since ${\ddot{X}}X=X{\ddot{X}}$, the above equation becomes

$$\begin{aligned} {\ddot{X}}=2{\ddot{X}}X+2{\dot{X}}^2. \end{aligned}$$

(6.12)

On account of (6.12), Eq. (6.10) is put in the form

$$\begin{aligned} \Omega =-2{\dot{X}}^2+\lambda (I_n-2X). \end{aligned}$$

(6.13)

Substituting (6.13) for $\Omega$ in (6.4), we obtain

$$\begin{aligned} {\ddot{X}}+4{\dot{X}}^2 X-2{\dot{X}}^2=0. \end{aligned}$$

(6.14)

Since ${\dot{X}} X+X{\dot{X}}={\dot{X}}$, one has ${\dot{X}}^2X+{\dot{X}}X{\dot{X}}={\dot{X}}^2$, and hence Eq. (6.14) is brought into

$$\begin{aligned} {\ddot{X}}+2{\dot{X}}^2-4{\dot{X}}X{\dot{X}}=0. \end{aligned}$$

(6.15)

This completes the proof. $\square$