Skip to main content
SpringerLink
Log in

Sparse Linear Least Squares Problems in Optimization

  • Chapter
Computational Issues in High Performance Software for Nonlinear Optimization

  • 145 Accesses

Abstract

Numerical and computational aspects of direct methods for large and sparse least squares problems are considered. After a brief survey of the most often used methods, we summarize the important conclusions made from a numerical comparison in matlab. Significantly improved algorithms have during the last 10–15 years made sparse QR factorization attractive, and competitive to previously recommended alternatives. Of particular importance is the multifrontal approach, characterized by low fill-in, dense subproblems and naturally implemented parallelism. We describe a Householder multifrontal scheme and its implementation on sequential and parallel computers. Available software has in practice a great influence on the choice of numerical algorithms. Less appropriate algorithms are thus often used solely because of existing software packages. We briefly survey software packages for the solution of sparse linear least squares problems. Finally, we focus on various applications from optimization, leading to the solution of large and sparse linear least squares problems. In particular, we concentrate on the important case where the coefficient matrix is a fixed general sparse matrix with a variable diagonal matrix below. Inner point methods for constrained linear least squares problems give, for example, rise to such subproblems. Important gains can be made by taking advantage of structure. Closely related is also the choice of numerical method for these subproblems. We discuss why the less accurate normal equations tend to be sufficient in many applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. M. Arioli, I. S. Duff, AND P. De RIJK, On the augmented system approach to sparse least-squares problems, Numcr. Math., 55 (1989), pp. 667–684.

    Article  MATH  Google Scholar 

  2. A. Björck, Stability analysis of the method of semi-normal equations for least squares problems. Linear Algebra Appl., 88/89 (1987), pp. 31–48.

    Article  Google Scholar 

  3. A. Björck, A direct method for sparse least squares problems with lower and upper bounds, Numer. Math., 54(1988). pp. 19–32.

    Article  MATH  MathSciNet  Google Scholar 

  4. A. Björck, Pivoting and stability in the augmented system method. Technical Report LiTH-MAT-R-1991-30, Department of Mathematics, Linköping University, June 1991.

    Google Scholar 

  5. A. Björck, Numerical Methods for Least Squares Problems, SIAM, Philadelphia, 1996.

    MATH  Google Scholar 

  6. J. R. Bunch AND L. Kaufman, Some stable methods for calculating inertia and solving symmetric-linear systems. Mathematics of Computation, 31 (1977), pp. 162–179.

    Article  MathSciNet  Google Scholar 

  7. E. C. H. Chu, J. A. George, J. Liu, AND E. Ng, SPARSPAK: Waterloo sparse matrix package user’s guide for SPARSPAK-A, Research Report CS-84-36, Dept. of Computer Science, University of Waterloo, 1984.

    Google Scholar 

  8. T. F. Coleman, A. Edenbrandt, AND J. R. Gilbert, Predicting fill for sparse orthogonal factorization, J. ACM, 33 (1986), pp. 517–532.

    Article  MATH  MathSciNet  Google Scholar 

  9. J. Dennis AND R. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice Hall, Englewood Cliffs, N.J., 1983.

    MATH  Google Scholar 

  10. I. S. Duff, Pivot selection and row orderings in Givens reduction on sparse matrices. Computing, 13 (1974), pp. 239–248.

    Article  MATH  MathSciNet  Google Scholar 

  11. I. S. Duff, N. I. M. Gould, J. K. Reid, J. A. Scott, AND K. Turner, The factorization of sparse symmetric indefinite matrices, IMA J. Numer. Anal., 11 (1991), pp. 181–204.

    Article  MATH  MathSciNet  Google Scholar 

  12. I. S. Duff, R. G. Grimes, AND J. G. Lewis, Sparse matrix test problems, ACM Trans. Math. Softw., 15(1989), pp. 1–14.

    Article  MATH  Google Scholar 

  13. I. S. Duff AND J. K. Reid, A comparison of some methods for the solution of sparse overdetermined systems of linear equations, J. Inst. Maths. Applies., 17 (1976), pp. 267–280.

    Article  MATH  MathSciNet  Google Scholar 

  14. I. S. Duff AND J. K. Reid, The multifrontal solution of indefinite sparse symmetric linear systems, ACM Trans. Math. Software, 9 (1983), pp. 302–325.

    Article  MATH  MathSciNet  Google Scholar 

  15. S. C. Eisenstat, M. H. Schultz, AND A. H. Sherman, Algorithms and data structures for sparse symmetric Gaussian elimination, SIAM J. Sci. Statist. Comput., 2 (1981), pp. 225–237.

    Article  MATH  MathSciNet  Google Scholar 

  16. L. Eldén, Algorithms for the regularizaron of ill-conditioned least squares problems, BIT, 17 (1977), pp. 134–145.

    Article  MATH  Google Scholar 

  17. L. V. Foster, Modifications of the normal equations method that are numerically stable, in Numerical Linear Algebra, Digital Signal Processing and Parallel Algorithms, G. H. Golub and P. V. Dooren, eds., NATO ASI Series, Berlin, 1991, Springer-Verlag, pp. 501–512.

    Google Scholar 

  18. C. F. Gauss, Theory of the Motion of the Heavenly Bodies Moving about the Sun in Conic Sections, Dover, New York (1963), 1809. C. H. Davis, Trans.

    MATH  Google Scholar 

  19. W. M. Gentleman, Basic procedures for large, sparse, or weighted linear least squares problems, Research report CSRR 2068, University of Waterloo, Waterloo, Ontario, Canada, July 1972.

    Google Scholar 

  20. W. M. Gentleman, Least squares computations by Givens transformations without square roots, J. Inst. Maths. Applies., 12 (1973), pp. 329–336.

    Article  MATH  MathSciNet  Google Scholar 

  21. W. M. Gentleman, Error analysis of QR decompositions by Givens transformations. Linear Algebra Appl., 10(1975), pp. 189–197.

    Article  MATH  MathSciNet  Google Scholar 

  22. W. M. Gentleman, Row elimination for solving sparse linear systems and least squares problems, in Proceedings the 6th Dundee Conference on Numerical Analysis, G. A. Watson, ed., Springer Verlag, 1976, pp. 122–133.

    Google Scholar 

  23. J. A George AND M. T. Heath, Solution of sparse linear least squares problems using Givens rotations. Linear Algebra Appl., 34 (1980), pp. 69–83.

    Article  MATH  MathSciNet  Google Scholar 

  24. J. A. George AND J. W.-H. Liu, Householder reflections versus Givens rotations in sparse orthogonal decomposition, Linear Algebra Appl., 88/89 (1987), pp. 223–238.

    Article  MathSciNet  Google Scholar 

  25. J. A. George AND E. G. Ng, On row and column orderings for sparse least squares problems, SIAM J. Numer. Anal., 20(1981), pp. 326–344.

    Article  MathSciNet  Google Scholar 

  26. J. A. George AND E. G. Ng, SPARSPAK: Waterloo sparse matrix package user’s guide for SPARSPAK-B, Research Report CS-84-37, Dept. of Computer Science, University of Waterloo, 1984.

    Google Scholar 

  27. J. R. Gilbert, C. Moler, AND R. Schreiber, Sparse matrices in MATLAB: Design and implementation, SIAM J. Matrix Anal. Appl., 13 (1992), pp. 333–356.

    Article  MATH  MathSciNet  Google Scholar 

  28. P. E. Gill AND W. Murray, Nonlinear least squares and nonlinearly constrained optimization, in In Proceedings Dundee Conference on Numerical Analysis 1975, Lecture Notes in Mathematics No. 506, Springer Verlag, 1976.

    Google Scholar 

  29. P. E. Gill AND W. Murray, The orthogonal factorization of a large sparse matrix, in Sparse Matrix Computations, J. R. Bunch and D. J. Rose, eds., New York, 1976, Academic Press, pp. 201–212.

    Google Scholar 

  30. P. E. Gill, W. Murray, AND M. H. Wright, Practical Optimization, Academic Press, London and New York, 1981.

    MATH  Google Scholar 

  31. G. H. Golub, Numerical methods for solving least squares problems, Numer. Math., 7 (1965), pp. 206–216.

    Article  MATH  MathSciNet  Google Scholar 

  32. C. C. Gonzaga, Path-following melhods for linear programming, SIAM Review, 34 (1992), pp. 167–224.

    Article  MATH  MathSciNet  Google Scholar 

  33. M. T. Heath, Numerical methods for large sparse linear least squares problems, SIAM J. Sci. Statist. Comput., 5 (1984), pp. 497–513.

    Article  MATH  MathSciNet  Google Scholar 

  34. G. B. Kolata, Geodesy: Dealing with an enormous computer tusk. Science, 200 (1978), pp. 421–422.

    Article  Google Scholar 

  35. C. L. Lawson AND R. J. Hanson, Solving Least Squares Problems, Prentice Hall, Englewood Cliffs, New Jersey. 1974.

    MATH  Google Scholar 

  36. A. M. Legendre, Nouvelle méthodes pour la détermination des orbites des comètes, Courcier, Paris, 1805.

    Google Scholar 

  37. K. Levenberg, A method for the solution of certain non-linear problems in least squares. Quart. Appl. Math., 2 (1944), pp. 164–168.

    MATH  MathSciNet  Google Scholar 

  38. J. W.-H. Liu, On general row merging schemes for sparse Givens transformations, SIAM J. Sci. Statist. Comput., 7 (1986), pp. 1190–1211.

    Article  MATH  MathSciNet  Google Scholar 

  39. J. W.-H. Llu, The role of elimination trees in sparse factorization, SIAM J. Matrix Anal. Appl., 11 (1990), pp. 134–172.

    Article  MathSciNet  Google Scholar 

  40. S. Lu AND J. L. Barlow, Multifrontal computation with the orthogonal factors of sparse matrices, SIAM J. Matr. Anal. 3 (1996), pp. 658–679.

    Article  MathSciNet  Google Scholar 

  41. I. Lustig, R. Marsten, AND D. Shanno, Computational experience with a primal-dual interior point method for linear programming, Linear Algebra Appl., (1991), pp. 191–222.

    Google Scholar 

  42. D. W. Marquardt, An algorithm for least-squares estimation of nonlinear parameters, Journal of the Society for Industrial and Applied Mathematics, 11 (1963), pp. 431–441.

    Article  MATH  MathSciNet  Google Scholar 

  43. P. Matstoms, The multifrontal solution of sparse linear least squares problems, Licentiat thesis, Linköping University, 1991.

    Google Scholar 

  44. P. Matstoms, QR27—Specification sheet, Tech. Report March 1992, Department of Mathematics, 1992.

    Google Scholar 

  45. P. Matstoms. Sparse QR factorization in MATLAB, ACM Trans. Math. Software, 20 (1994), pp 136–159.

    Article  MATH  Google Scholar 

  46. P. Matstoms, Sparse QR Factorization with Applications to Linear Least Squares Problems, PhD thesis, Linköping University, 1994.

    Google Scholar 

  47. P. Matstoms, Parallel sparse QR factorization on shared memory architectures. Parallel Computing, 21 (1995),pp. 473–486.

    Article  MATH  Google Scholar 

  48. J. J. Moré, The Levenberg-Marquardt algorithm: Implementation and theory, in G.A. Watson, Lecture Notes in Math. 630, Berlin, 1978, Springer Verlag, pp. 105–116.

    Google Scholar 

  49. U. Oreborn, A Direct Method for Sparse Nonnegative Least Squares Problems, licentiat thesis, Linköping University. 1986.

    Google Scholar 

  50. C. C. Paige AND M. A. Saunders, LSQR. an algorithm for sparse linear equations and sparse least squares, ACM Trans. Math. Software. 8 (1982), pp. 43–71.

    Article  MATH  MathSciNet  Google Scholar 

  51. D. J. Pierce AND J. G. Lewis, Sparse multifrontal rank revealing QR factorization, Technical Report MEA-TR-193-Revised, Boeing Information and Support Services, 1995.

    Google Scholar 

  52. L. F. Portugal, J. J. Judice. AND L. N. Vicente, Solution of large scale linear least-squares problems with nonnegativ variables, technical report. Departamento de ciêcias da terra, Universidade de Coimbra, 3000 Coimbra, Portugal, 1993.

    Google Scholar 

  53. C. Puglisi, QR factorization of large sparse overdetermined and square matrices with the multifrontal method in a multiprocessor environment, PhD thesis, CERFACS, 42 av. G. Coriolis, 31057 Toulouse Cedex, France.

    Google Scholar 

  54. J. K. Reid, A note on the least squares solution of a band system of linear equations by Householder reductions, Comput J., 10 (1967), pp. 188–189.

    MATH  MathSciNet  Google Scholar 

  55. C. Sun, Parallel sparse orthogonal factorization on distributed-memory multiprocessors, SIAM J. Sci. Comput., 17 (1996), p. to appear.

    Google Scholar 

  56. M. H. Wright, Interior methods for constrained optimization. Acta Numerica 1992. Cambridge University Press, 1992, pp. 341–407.

    Google Scholar 

  57. Z. Zlatev AND H. Nielsen, LLSS01-a Fortran subroutine for solving least squares problems (User’s guide), Technical Report 79-07, Institute of Numerical Analysis, Technical University of Denmark. Lyngby, Denmark, 1979.

    Google Scholar 

  58. Z. Zlatev AND H. Nielsen. Solving large and sparse linear least-squares problems by conjugate gradient algorithms, Comput. Math. Applic., 15 (1988), pp. 185–202.

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department oj Mathematics, Linköping University, S-581 82, Linköping, Sweden

    Pontus Matstoms

Authors
  1. Pontus Matstoms
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Universitià di Napoli “Federico II”, Italy

    Almerico Murli  & Gerardo Toraldo  & 

Rights and permissions

Reprints and permissions

Copyright information

© 1997 Kluwer Academic Publishers, Boston

About this chapter

Cite this chapter

Matstoms, P. (1997). Sparse Linear Least Squares Problems in Optimization. In: Murli, A., Toraldo, G. (eds) Computational Issues in High Performance Software for Nonlinear Optimization. Springer, Boston, MA. https://doi.org/10.1007/978-0-585-26778-4_6

Download citation

  • DOI: https://doi.org/10.1007/978-0-585-26778-4_6

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-0-7923-9862-2

  • Online ISBN: 978-0-585-26778-4

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation