Inexact methods for the low rank solution to large scale Lyapunov equations

Abstract

The rational Krylov subspace method (RKSM) and the low-rank alternating directions implicit (LR-ADI) iteration are established numerical tools for computing low-rank solution factors of large-scale Lyapunov equations. In order to generate the basis vectors for the RKSM, or extend the low-rank factors within the LR-ADI method, the repeated solution to a shifted linear system of equations is necessary. For very large systems this solve is usually implemented using iterative methods, leading to inexact solves within this inner iteration (and therefore to “inexact methods”). We will show that one can terminate this inner iteration before full precision has been reached and still obtain very good accuracy in the final solution to the Lyapunov equation. In particular, for both the RKSM and the LR-ADI method we derive theory for a relaxation strategy (e.g. increasing the solve tolerance of the inner iteration, as the outer iteration proceeds) within the iterative methods for solving the large linear systems. These theoretical choices involve unknown quantities, therefore practical criteria for relaxing the solution tolerance within the inner linear system are then provided. The theory is supported by several numerical examples, which show that the total amount of work for solving Lyapunov equations can be reduced significantly.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2

References

  1. 1.

    Ahmad, M.I., Szyld, D.B., van Gijzen, M.B.: Preconditioned multishift BiCG for \(H_{2}\)-optimal model reduction. SIAM J. Matrix Anal. Appl. 38(2), 401–424 (2017)

    MathSciNet  MATH  Google Scholar 

  2. 2.

    Antoulas, A.C.: Approximation of Large-Scale Dynamical Systems. Advances in Design and Control, vol. 6. SIAM, Philadelphia, PA (2005)

    Google Scholar 

  3. 3.

    Baker, J., Embree, M., Sabino, J.: Fast singular value decay for Lyapunov solutions with nonnormal coefficients. SIAM J. Matrix Anal. Appl. 36(2), 656–668 (2015)

    MathSciNet  MATH  Google Scholar 

  4. 4.

    Bartels, R.H., Stewart, G.W.: Solution of the matrix equation \({AX}+{XB}={C}\): algorithm 432. Commun. ACM 15, 820–826 (1972)

    MATH  Google Scholar 

  5. 5.

    Beckermann, B., Townsend, A.: On the singular values of matrices with displacement structure. SIAM J. Matrix Anal. Appl. 38(4), 1227–1248 (2017)

    MathSciNet  MATH  Google Scholar 

  6. 6.

    Benner, P., Bujanović, Z., Kürschner, P., Saak, J.: A numerical comparison of solvers for large-scale, continuous-time algebraic Riccati equations and LQR problems. SIAM J. Sci. Comput. 42(2), A957–A996 (2020)

    MathSciNet  MATH  Google Scholar 

  7. 7.

    Benner, P., Bujanović, Z., Kürschner, P., Saak, J.: RADI: a low-rank ADI-type algorithm for large scale algebraic Riccati equations. Numer. Math. 138(2), 301–330 (2018)

    MathSciNet  MATH  Google Scholar 

  8. 8.

    Benner, P., Heinkenschloss, M., Saak, J., Weichelt, H.K.: An inexact low-rank Newton-ADI method for large-scale algebraic Riccati equations. Appl. Numer. Math. 108, 125–142 (2016)

    MathSciNet  MATH  Google Scholar 

  9. 9.

    Benner, P., Kürschner, P.: Computing real low-rank solutions of Sylvester equations by the factored ADI method. Comput. Math. Appl. 67(9), 1656–1672 (2014)

    MathSciNet  MATH  Google Scholar 

  10. 10.

    Benner, P., Kürschner, P., Saak, J.: An improved numerical method for balanced truncation for symmetric second order systems. Math. Comput. Model. Dyn. Sys. 19(6), 593–615 (2013)

    MathSciNet  MATH  Google Scholar 

  11. 11.

    Benner, P., Kürschner, P., Saak, J.: Efficient Handling of complex shift parameters in the low-rank Cholesky factor ADI method. Numer. Algorithms 62(2), 225–251 (2013)

    MathSciNet  MATH  Google Scholar 

  12. 12.

    Benner, P., Kürschner, P., Saak, J.: Self-generating and efficient shift parameters in ADI methods for large Lyapunov and Sylvester equations. Electron. Trans. Numer. Anal. 43, 142–162 (2014)

    MathSciNet  MATH  Google Scholar 

  13. 13.

    Benner, P., Li, R.-C., Truhar, N.: On the ADI method for Sylvester equations. J. Comput. Appl. Math. 233(4), 1035–1045 (2009)

    MathSciNet  MATH  Google Scholar 

  14. 14.

    Benner, P., Saak, J.: Numerical solution of large and sparse continuous time algebraic matrix Riccati and Lyapunov equations: a state of the art survey. GAMM Mitteilungen 36(1), 32–52 (2013)

    MathSciNet  MATH  Google Scholar 

  15. 15.

    Bergamaschi, L., Martínez, Á.: Two-stage spectral preconditioners for iterative eigensolvers. Numer. Linear Algebra Appl. 24(3), e2084 (2017)

    MathSciNet  MATH  Google Scholar 

  16. 16.

    Berljafa, M., Güttel, S.: Generalized rational Krylov decompositions with an application to rational approximation. SIAM J. Matrix Anal. Appl. 36(2), 894–916 (2015)

    MathSciNet  MATH  Google Scholar 

  17. 17.

    Bouras, A., Frayssé, V.: Inexact matrix–vector products in Krylov methods for solving linear systems: a relaxation strategy. SIAM J. Matrix Anal. Appl. 26(3), 660–678 (2005)

    MathSciNet  MATH  Google Scholar 

  18. 18.

    Canuto, C., Simoncini, V., Verani, M.: On the decay of the inverse of matrices that are sum of Kronecker products. Linear Algebra Appl. 452, 21–39 (2014)

    MathSciNet  MATH  Google Scholar 

  19. 19.

    Crouzeix, M., Palencia, C.: The numerical range is a \((1+\sqrt{2})\)-spectral set. SIAM J. Matrix Anal. Appl. 38(2), 649–655 (2017)

    MathSciNet  MATH  Google Scholar 

  20. 20.

    Druskin, V., Knizhnerman, L.A., Simoncini, V.: Analysis of the rational Krylov subspace and ADI methods for solving the Lyapunov equation. SIAM J. Numer. Anal. 49, 1875–1898 (2011)

    MathSciNet  MATH  Google Scholar 

  21. 21.

    Druskin, V., Simoncini, V.: Adaptive rational Krylov subspaces for large-scale dynamical systems. Syst. Control Lett. 60(8), 546–560 (2011)

    MathSciNet  MATH  Google Scholar 

  22. 22.

    Freitag, M.A., Spence, A.: Convergence theory for inexact inverse iteration applied to the generalised nonsymmetric eigenproblem. Electron. Trans. Numer. Anal. 28, 40–64 (2007/08)

  23. 23.

    Freitag, M.A., Spence, A.: Shift-invert Arnoldi’s method with preconditioned iterative solves. SIAM J. Matrix Anal. Appl. 31(3), 942–969 (2009)

    MathSciNet  MATH  Google Scholar 

  24. 24.

    Gaaf, S.W., Simoncini, V.: Approximating the leading singular triplets of a large matrix function. Appl. Numer. Math. 113, 26–43 (2017)

    MathSciNet  MATH  Google Scholar 

  25. 25.

    Grasedyck, L.: Existence of a low rank or \({\cal{H}}\)-matrix approximant to the solution of a Sylvester equation. Numer. Linear Algebra Appl. 11(4), 371–389 (2004)

    MathSciNet  MATH  Google Scholar 

  26. 26.

    Güttel, S.: Rational Krylov methods for operator functions. Ph.D. thesis, Technische Universität Bergakademie Freiberg, Germany. http://eprints.ma.man.ac.uk/2586/ (2010)

  27. 27.

    Güttel, S.: Rational Krylov approximation of matrix functions: numerical methods and optimal pole selection. GAMM-Mitteilungen 36(1), 8–31 (2013)

    MathSciNet  MATH  Google Scholar 

  28. 28.

    Jaimoukha, I.M., Kasenally, E.M.: Krylov subspace methods for solving large Lyapunov equations. SIAM J. Numer. Anal. 31, 227–251 (1994)

    MathSciNet  MATH  Google Scholar 

  29. 29.

    Kressner, D., Massei, S., Robol, L.: Low-rank updates and a divide-and-conquer method for linear matrix equations. SIAM J. Sci. Comput. 41(2), A848–A876 (2019)

    MathSciNet  MATH  Google Scholar 

  30. 30.

    Kürschner, P.: Efficient low-rank solution of large-scale matrix equations. Dissertation, Otto-von-Guericke-Universität, Magdeburg, Germany. http://hdl.handle.net/11858/00-001M-0000-0029-CE18-2 (2016)

  31. 31.

    Lehoucq, R.B., Meerbergen, K.: Using generalized Cayley transformations within an inexact rational Krylov sequence method. SIAM J. Matrix Anal. Appl. 20(1), 131–148 (1998)

    MathSciNet  MATH  Google Scholar 

  32. 32.

    Li, J.-R.:Model reduction of large linear systems via low rank system Gramians. Ph.D. thesis, Massachusettes Institute of Technology (2000)

  33. 33.

    Li, J.-R., White, J.: Low rank solution of Lyapunov equations. SIAM J. Matrix Anal. Appl. 24(1), 260–280 (2002)

    MathSciNet  MATH  Google Scholar 

  34. 34.

    Opmeer, M., Reis, T., Wollner, W.: Finite-rank ADI iteration for operator Lyapunov equations. SIAM J. Control Optim. 51(5), 4084–4117 (2013)

    MathSciNet  MATH  Google Scholar 

  35. 35.

    Palitta, D., Simoncini, V.: Numerical methods for large-scale Lyapunov equations with symmetric banded data. SIAM J. Sci. Comput. 40(5), A3581–A3608 (2018)

    MathSciNet  MATH  Google Scholar 

  36. 36.

    Penzl, T.: A cyclic low rank Smith method for large sparse Lyapunov equations. SIAM J. Sci. Comput. 21(4), 1401–1418 (2000)

    MathSciNet  MATH  Google Scholar 

  37. 37.

    Ruhe, Axel: The rational Krylov algorithm for nonsymmetric eigenvalue problems. III: complex shifts for real matrices. BIT 34, 165–176 (1994)

    MathSciNet  MATH  Google Scholar 

  38. 38.

    Saad, Y.: Numerical solution of large Lyapunov equation. In: Kaashoek, M.A., van Schuppen, J.H., Ran, A.C.M. (eds.) Signal Processing. Scattering, Operator Theory and Numerical Methods, pp. 503–511. Birkhäuser, Basel (1990)

    Google Scholar 

  39. 39.

    Saak, J.: Efficient numerical solution of large scale algebraic matrix equations in PDE control and model order reduction. Ph.D. thesis, TU Chemnitz. http://nbn-resolving.de/urn:nbn:de:bsz:ch1-200901642 (2009)

  40. 40.

    Sabino, J.: Solution of large-scale Lyapunov equations via the block modified Smith method. Ph.D. thesis, Rice University, Houston, TX. http://www.caam.rice.edu/tech_reports/2006/TR06-08.pdf (2007)

  41. 41.

    Shank, S.D., Simoncini, V., Szyld, D.B.: Efficient low-rank solution of generalized Lyapunov equations. Numer. Math. 134, 327–342 (2015)

    MathSciNet  MATH  Google Scholar 

  42. 42.

    Simoncini, V.: Variable accuracy of matrix–vector products in projection methods for eigencomputation. SIAM J. Numer. Anal. 43(3), 1155–1174 (2005)

    MathSciNet  MATH  Google Scholar 

  43. 43.

    Simoncini, V.: A new iterative method for solving large-scale Lyapunov matrix equations. SIAM J. Sci. Comput. 29(3), 1268–1288 (2007)

    MathSciNet  MATH  Google Scholar 

  44. 44.

    Simoncini, V.: Analysis of the rational Krylov subspace projection method for large-scale algebraic Riccati equations. SIAM J. Matrix Anal. Appl. 37(4), 1655–1674 (2016)

    MathSciNet  MATH  Google Scholar 

  45. 45.

    Simoncini, V.: Computational methods for linear matrix equations. SIAM Rev. 58(3), 377–441 (2016)

    MathSciNet  MATH  Google Scholar 

  46. 46.

    Simoncini, V., Eldén, L.: Inexact Rayleigh quotient-type methods for eigenvalue computations. BIT Numer. Math. 42(1), 159–182 (2002)

    MathSciNet  MATH  Google Scholar 

  47. 47.

    Simoncini, V., Szyld, D.B.: Theory of inexact Krylov subspace methods and applications to scientific computing. SIAM J. Sci. Comput. 25(2), 454–477 (2003)

    MathSciNet  MATH  Google Scholar 

  48. 48.

    Simoncini, V., Szyld, D.B., Monsalve, M.: On two numerical methods for the solution of large-scale algebraic Riccati equations. IMA J. Numer. Anal. 34(3), 904–920 (2014)

    MathSciNet  MATH  Google Scholar 

  49. 49.

    Soodhalter, K.: A block MINRES algorithm based on the banded Lanczos method. Numer. Algorithms 69, 473–494 (2015)

    MathSciNet  MATH  Google Scholar 

  50. 50.

    Sun, K.: Model order reduction and domain decomposition for large-scale dynamical systems. Ph.D. thesis, Rice University, Houston. http://search.proquest.com/docview/304507831 (2008)

  51. 51.

    Truhar, N., Veselić, K.: Bounds on the trace of a solution to the Lyapunov equation with a general stable matrix. Syst. Control Lett. 56(7–8), 493–503 (2007)

    MathSciNet  MATH  Google Scholar 

  52. 52.

    Truhar, N., Veselić, K.: An efficient method for estimating the optimal dampers’ viscosity for linear vibrating systems using Lyapunov equation. SIAM J. Matrix Anal. Appl. 31(1), 18–39 (2009)

    MathSciNet  MATH  Google Scholar 

  53. 53.

    van den Eshof, J., Sleijpen, G.: Inexact Krylov subspace methods for linear systems. SIAM J. Matrix Anal. Appl. 26(1), 125–153 (2004)

    MathSciNet  MATH  Google Scholar 

  54. 54.

    Wachspress, E.L.: The ADI Model Problem. Springer, New York (2013)

    Google Scholar 

  55. 55.

    Wolf, T., Panzer, H.K.-F.: The ADI iteration for Lyapunov equations implicitly performs \({\cal{H}}_2\) pseudo-optimal model order reduction. Int. J. Control 89(3), 481–493 (2016)

    MATH  Google Scholar 

Download references

Acknowledgements

The authors are grateful to Cost Action EU-MORNET (TD1307) and the Department of Mathematical Sciences at Bath, that provided funding for research visits of PK to the University of Bath, where substantial parts of this work have been conducted. This work was primarily generated while PK was affiliated with the Max Planck Institute for Dynamics of Complex Technical Systems in Magdeburg, Germany and MAF was affiliated with the University of Bath, United Kingdom. Furthermore, the authors thank Kirk Soodhalter (Trinity College Dublin) for insightful discussion regarding block Krylov subspace methods.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Patrick Kürschner.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by Daniel Kressner.

Appendices

Appendix A: Proof of Lemma 2.2

Proof

From \(\omega \underline{H}_j=0\) we have for \(k\le j\)

which, for \(k=j\), immediately leads to \(f_j^{(j)}=-\omega _{1:j}/(h_{j+1,j}\omega _{j+1})\), the first equality in (2.21a). Similarly, it is easy to show that for \(k<j\), a left null space vector \(\hat{\omega }\in \mathbb {C}^{1\times k+1}\) of \(\underline{H}_{k}\) is given by the first \(k+1\) entries of the null vector \(\omega \) of \(\underline{H}_j\). Hence, \(f^{(k)}_k=-\omega _{1:k}/(h_{k+1,k}\omega _{k+1})\) holds for all \(k\le j\).

For computing \(f_j^{(k)}= e_k^*H_j^{-1}\), \(k<j\), we use the following partition of \(H_j\) and consider the splitting \(f^{(k)}_j = [u,y]\), \(u\in \mathbb {C}^{1\times k}\), \(y\in \mathbb {C}^{1\times k-j}\):

$$\begin{aligned}&e_k^*=\left[ \begin{array}{ccc} 0_{1,k-1}\big |1\big |&0_{1,j-k} \end{array}\right] =f^{(k)}_jH_j=[u,y] \left[ \begin{array}{c} \begin{array}{c} H_{k-1}\\ e_{k-1}^*h_{k,k-1}\\ 0_{j-k,k-1} \end{array}\Bigg | H_{:,k:j} \end{array}\right] \\&=\left[ \begin{array}{cc} u\left[ \begin{array}{c} H_{k-1}\\ e_{k-1}^*h_{k,k-1} \end{array}\right]&\Bigg |[u,y]H_{:,k:j}\end{array}\right] . \end{aligned}$$

This structure enforces conditions on [uy], which we now explore. First, u has to be a multiple of \(\omega _{1:k}\). Here we exploited that due to the Hessenberg structure, \(\omega _{1:k}\underline{H}_{k-1}=0\). In particular,

$$\begin{aligned} u_{1:k-1}H_{k-1}+u_ke_{k-1}^*h_{k,k-1}=0, \end{aligned}$$

such that \(u_kh_{k,k-1}e_{k-1}^*H_{k-1}^{-1}=-u_{1:k-1}\). Since \(f^{(k-1)}_{k-1}=e_{k-1}^*H_{k-1}^{-1}\) we can infer \(u_{1:k-1}=-u_kh_{k,k-1}f^{(k-1)}_{k-1}\) and, consequently, (2.21c) for \(k>1\). Similarly, \([\omega _{1:k},y]\) has to satisfy

$$\begin{aligned} 0_{1,j-k}&=[\omega _{1:k},y]H_{:,k+1:j}=\omega _{1:k}H_{1:k,k+1:j}+yH_{k+1:j,k+1:j}=\omega H_{1:j+1,k+1:j}\\&=\omega _{1:k}H_{1:k,k+1:j}+\omega _{k+1:j+1}H_{k+1:j+1,k+1:j}, \end{aligned}$$

leading to

$$\begin{aligned} y=\omega _{k+1:j+1}H_{k+1:j+1,k+1:j}H_{k+1:j,k+1:j}^{-1}=\omega _{k+1:j} + \omega _{j+1} h_{j+1,j}e_{j-k}^*H_{k+1:j,k+1:j}^{-1}, \end{aligned}$$

where \(e_{j-k}\) is a canonical vector of length \(j-k\). Hence,

$$\begin{aligned} v_j^{(k)}=[\omega _{1:k},y]=\omega _{1:j}+[0_{1,k},[0_{1,j-k-1},h_{j+1,j}\omega _{j+1}]H_{k+1:j,k+1:j}^{-1}], \end{aligned}$$

leading to (2.21b). Finally, the normalization constant \(\phi _{j}^{(k)}\) is obtained by the requirement \([u,y]H_{1:j,k}=1\). \(\square \)

Appendix B: Setup of example msd

The example msd represents a variant of one realization of the series of examples in [52]. Set \(K:={\text {diag}}\!\left( I_3\otimes K_1,k_0+k_1+k_2+k_3\right) +k_+e_{3n}^T-e_{3n}k_+\in {\mathbb {R}}^{n_2\times n_2},~K_1:=\mathrm {tridiag}(-1,2,-1)\in {\mathbb {R}}^{n_1\times n_1}\) with \(k_0=0.1\), \(k_1=1\), \(k_2=2\), \(k_3=4\), \(n_2=3n_1+1\), and \(k_+:=[(k_1,k_2,k_3)\otimes e_{n_1}^T,0]^T\). Further, \(M_1:={\text {diag}}\!\left( m_1I_{n_1},m_2I_{n_1},m_3I_{n_1}m_0\right) \in {\mathbb {R}}^{n_2\times n_2}\) with \(m_0=1000\), \(m_1=10\), \(m_2=k_2\), \(m_3=k_3\) and \(D:=\alpha M+\beta (K+K(M^{-1}K)+K(M^{-1}K)^2+K(M^{-1}K)^3)+\nu [e_1,e_{n_1},e_{2n_1+1}][e_1,e_{n_1},e_{2n_1+1}]^T\), \(\alpha =0.8\), \(\beta =0.1\), \(\nu =16\). In [52] a different matrix D was used involving a term \(M^{\frac{1}{2}}\sqrt{M^{-\frac{1}{2}}KM^{-\frac{1}{2}}}M^{\frac{1}{2}}\) which was infeasible to set up in a large scale setting (the middle matrix square is a dense matrix). The version of D used here was similarly used in [10]. Construct the \(n\times n\), \(n:=2n_2\) block matrices \(\hat{A}:=\left[ \begin{array}{ll} 0&{}\quad I_{n_2}\\ -K&{}\quad -D\end{array}\right] \), \(\hat{M}:={\text {diag}}\!\left( I_{n_2},M_1\right) \) representing a linearization of the quadratic matrix pencil \(\lambda ^2M_1+\lambda D+K\). The right hand side factor is set up as \(\hat{B}=\left[ \begin{matrix} I_{2m}&{}0_{2m,n_2-m}&{}\left[ \begin{matrix}0_{m,n_2-m}&{}I_m\\ I_m&{}0_{m,n_2-m}\end{matrix}\right] \end{matrix}\right] ^T\in {\mathbb {R}}^{n\times 2m}\). In Sect. 4 we use \(n_1=4000\), \(m=2\) leading to \(n=24{,}002\), \(r=4\). Finally, \(A:=P^T\hat{A}P\), \(M:=P^T\hat{M}P\), \(B:=P^T\hat{B}\) as in [29, Section 5.6], where P is a perfect shuffle permutation: This leads to banded matrices AM, except for some sole off-band entries from the low-rank update of D, resulting in noticable computational savings when applying sparse direct solvers or computing sparse (incomplete) factorizations. An alternative to exploit the structure in \(\hat{A},\hat{M}\) within RKSM, LR-ADI is described in, e.g., [10, 30].

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kürschner, P., Freitag, M.A. Inexact methods for the low rank solution to large scale Lyapunov equations. Bit Numer Math (2020). https://doi.org/10.1007/s10543-020-00813-4

Download citation

Keywords

  • Lyapunov equation
  • Alternating direction implicit
  • Rational Krylov subspaces
  • Low-rank approximations

Mathematics Subject Classification

  • 15A24
  • 65F10
  • 65F30