Numerische Mathematik

, Volume 141, Issue 3, pp 743–789 | Cite as

Higher-order principal component analysis for the approximation of tensors in tree-based low-rank formats

  • Anthony NouyEmail author


This paper is concerned with the approximation of tensors using tree-based tensor formats, which are tensor networks whose graphs are dimension partition trees. We consider Hilbert tensor spaces of multivariate functions defined on a product set equipped with a probability measure. This includes the case of multidimensional arrays corresponding to finite product sets. We propose and analyse an algorithm for the construction of an approximation using only point evaluations of a multivariate function, or evaluations of some entries of a multidimensional array. The algorithm is a variant of higher-order singular value decomposition which constructs a hierarchy of subspaces associated with the different nodes of the tree and a corresponding hierarchy of interpolation operators. Optimal subspaces are estimated using empirical principal component analysis of interpolations of partial random evaluations of the function. The algorithm is able to provide an approximation in any tree-based format with either a prescribed rank or a prescribed relative error, with a number of evaluations of the order of the storage complexity of the approximation format. Under some assumptions on the estimation of principal components, we prove that the algorithm provides either a quasi-optimal approximation with a given rank, or an approximation satisfying the prescribed relative error, up to constants depending on the tree and the properties of interpolation operators. The analysis takes into account the discretization errors for the approximation of infinite-dimensional tensors. For a tensor with finite and known rank in a tree-based format, the algorithm is able to recover the tensor in a stable way using a number of evaluations equal to the storage complexity of the representation of the tensor in this format. Several numerical examples illustrate the main results and the behavior of the algorithm for the approximation of high-dimensional functions using hierarchical Tucker or tensor train tensor formats, and the approximation of univariate functions using tensorization.

Mathematics Subject Classification

15A69 41A05 41A63 65D15 65J99 



  1. 1.
    Bachmayr, M., Schneider, R., Uschmajew, A.: Tensor networks and hierarchical tensors for the solution of high-dimensional partial differential equations. Found. Comput. Math. 16(6), 1423–1472 (2016)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Ballani, J., Grasedyck, L., Kluge, M.: Black box approximation of tensors in hierarchical Tucker format. Linear Algebra Appl. 438(2), 639–657 (2013). Tensors and Multilinear AlgebraMathSciNetzbMATHGoogle Scholar
  3. 3.
    Blanchard, G., Bousquet, O., Zwald, L.: Statistical properties of kernel principal component analysis. Mach. Learn. 66(2–3), 259–294 (2007)zbMATHGoogle Scholar
  4. 4.
    Bungartz, H.-J., Griebel, M.: Sparse grids. Acta Numer. 13, 147–269 (2004)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Chevreuil, M., Lebrun, R., Nouy, A., Rai, P.: A least-squares method for sparse low rank approximation of multivariate functions. SIAM/ASA J. Uncertain. Quantif. 3(1), 897–921 (2015)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Cohen, A., DeVore, R.: Approximation of high-dimensional parametric pdes. Acta Numer. 24, 1–159 (2015)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Cohen, N., Sharir, O., Shashua, A.: On the expressive power of deep learning: a tensor analysis. In: Conference on Learning Theory, pp. 698–728 (2016)Google Scholar
  8. 8.
    De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl. 21(4), 1253–1278 (2000)MathSciNetzbMATHGoogle Scholar
  9. 9.
    de Silva, V., Lim, L.-H.: Tensor rank and ill-posedness of the best low-rank approximation problem. SIAM J. Matrix Anal. Appl. 30(3), 1084–1127 (2008)MathSciNetzbMATHGoogle Scholar
  10. 10.
    DeVore, R.A.: Nonlinear approximation. Acta Numer. 7, 51–150 (1998)zbMATHGoogle Scholar
  11. 11.
    Doostan, A., Validi, A., Iaccarino, G.: Non-intrusive low-rank separated approximation of high-dimensional stochastic models. Comput. Methods Appl. Mech. Eng. 263, 42–55 (2013)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Espig, M., Grasedyck, L., Hackbusch, W.: Black box low tensor-rank approximation using fiber-crosses. Constr. Approx. 30, 557–597 (2009)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Falcó, A., Hackbusch, W.: On minimal subspaces in tensor representations. Found. Comput. Math. 12, 765–803 (2012)MathSciNetzbMATHGoogle Scholar
  14. 14.
    Falco, A., Hackbusch, W., Nouy, A.: Geometric Structures in Tensor Representations (Final Release). ArXiv e-prints (2015)Google Scholar
  15. 15.
    Falcó, A., Hackbusch, W., Nouy, A.: On the Dirac–Frenkel variational principle on tensor Banach spaces. Found. Comput. Math. (2018).
  16. 16.
    Falcó, A., Hackbusch, W., Nouy, A.: Tree-based tensor formats. SeMA J. (2018).
  17. 17.
    Grasedyck, L.: Hierarchical singular value decomposition of tensors. SIAM J. Matrix Anal. Appl. 31, 2029–2054 (2010)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Grasedyck, L., Kressner, D., Tobler, C.: A literature survey of low-rank tensor approximation techniques. GAMM-Mitteilungen 36(1), 53–78 (2013)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Grelier, E., Nouy, A., Chevreuil, M.: Learning with tree-based tensor formats (2018). arXiv e-prints arXiv:1811.04455
  20. 20.
    Hackbusch, W.: Tensor Spaces and Numerical Tensor Calculus, Volume 42 of Springer Series in Computational Mathematics. Springer, Heidelberg (2012)Google Scholar
  21. 21.
    Hackbusch, W., Kuhn, S.: A new scheme for the tensor representation. J. Fourier Anal. Appl. 15(5), 706–722 (2009)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Hillar, C., Lim, L.-H.: Most tensor problems are np-hard. J. ACM (JACM) 60(6), 45 (2013)MathSciNetzbMATHGoogle Scholar
  23. 23.
    Holtz, S., Rohwedder, T., Schneider, R.: On manifolds of tensors of fixed tt-rank. Numer. Math. 120(4), 701–731 (2012)MathSciNetzbMATHGoogle Scholar
  24. 24.
    Jirak, M., Wahl, M.: A tight \(\sin \varTheta \) theorem for empirical covariance operators. ArXiv e-prints (2018)Google Scholar
  25. 25.
    Jirak, M., Wahl, M.: Relative perturbation bounds with applications to empirical covariance operators. ArXiv e-prints (2018)Google Scholar
  26. 26.
    Khoromskij, B.: O (dlog n)-quantics approximation of nd tensors in high-dimensional numerical modeling. Constr. Approx. 34(2), 257–280 (2011)MathSciNetzbMATHGoogle Scholar
  27. 27.
    Khoromskij, B.: Tensors-structured numerical methods in scientific computing: survey on recent advances. Chemometr. Intell. Lab. Syst. 110(1), 1–19 (2012)MathSciNetGoogle Scholar
  28. 28.
    Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455–500 (2009)MathSciNetzbMATHGoogle Scholar
  29. 29.
    Kressner, D., Steinlechner, M., Uschmajew, A.: Low-rank tensor methods with subspace correction for symmetric eigenvalue problems. SIAM J. Sci. Comput. 36(5), A2346–A2368 (2014)MathSciNetzbMATHGoogle Scholar
  30. 30.
    Lubich, C., Rohwedder, T., Schneider, R., Vandereycken, B.: Dynamical approximation by hierarchical tucker and tensor-train tensors. SIAM J. Matrix Anal. Appl. 34(2), 470–494 (2013)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Luu, T.H., Maday, Y., Guillo, M., Guérin, P.: A new method for reconstruction of cross-sections using Tucker decomposition. J. Comput. Phys. 345, 189–206 (2017)MathSciNetzbMATHGoogle Scholar
  32. 32.
    Maday, Y., Nguyen, N.C., Patera, A.T., Pau, G.S.H.: A general multipurpose interpolation procedure: the magic points. Commun. Pure Appl. Anal. 8(1), 383–404 (2009)MathSciNetzbMATHGoogle Scholar
  33. 33.
    Megginson, R.E: An Introduction to Banach Space Theory, Vol. 183. Springer, Berlin (2012)Google Scholar
  34. 34.
    Nouy, A.: Low-rank methods for high-dimensional approximation and model order reduction. In: Benner, P., Cohen, A., Ohlberger, M., Willcox, K. (eds.) Model Reduction and Approximation: Theory and Algorithms. SIAM, Philadelphia (2017)Google Scholar
  35. 35.
    Nouy, A.: Low-Rank Tensor Methods for Model Order Reduction, pp. 857–882. Springer, Cham (2017)Google Scholar
  36. 36.
    Orus, R.: A practical introduction to tensor networks: matrix product states and projected entangled pair states. Ann. Phys. 349, 117–158 (2014)MathSciNetzbMATHGoogle Scholar
  37. 37.
    Oseledets, I.: Tensor-train decomposition. SIAM J. Sci. Comput. 33(5), 2295–2317 (2011)MathSciNetzbMATHGoogle Scholar
  38. 38.
    Oseledets, I., Tyrtyshnikov, E.: Breaking the curse of dimensionality, or how to use svd in many dimensions. SIAM J. Sci. Comput. 31(5), 3744–3759 (2009)MathSciNetzbMATHGoogle Scholar
  39. 39.
    Oseledets, I., Tyrtyshnikov, E.: TT-cross approximation for multidimensional arrays. Linear Algebra Appl. 432(1), 70–88 (2010)MathSciNetzbMATHGoogle Scholar
  40. 40.
    Oseledets, I., Tyrtyshnikov, E.: Algebraic wavelet transform via quantics tensor train decomposition. SIAM J. Sci. Comput. 33(3), 1315–1328 (2011)MathSciNetzbMATHGoogle Scholar
  41. 41.
    Reiß, M., Wahl, M.: Non-asymptotic upper bounds for the reconstruction error of PCA (2016). arXiv preprint arXiv:1609.03779
  42. 42.
    Schneider, R., Uschmajew, A.: Approximation rates for the hierarchical tensor format in periodic Sobolev spaces. J. Complex. 30(2), 56–71 (2014). Dagstuhl 2012MathSciNetzbMATHGoogle Scholar
  43. 43.
    Temlyakov, V.: Nonlinear methods of approximation. Found. Comput. Math. 3(1), 33–107 (2003)MathSciNetzbMATHGoogle Scholar
  44. 44.
    Temlyakov, V.: Greedy Approximation. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, Cambridge (2011)Google Scholar
  45. 45.
    Uschmajew, A., Vandereycken, B.: The geometry of algorithms using hierarchical tensors. Linear Algebra Appl. 439(1), 133–166 (2013)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Centrale Nantes, LMJL, UMR CNRS 6629NantesFrance

Personalised recommendations