Skip to main content
SpringerLink
Log in

A Fast Algorithm for Simultaneous Sparse Approximation

  • Conference paper
  • First Online:
Advances in Knowledge Discovery and Data Mining (PAKDD 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12714))

Included in the following conference series:

Abstract

Simultaneous sparse approximation problems arise in several domains, such as signal processing and machine learning. Given a dictionary matrix X of size \(m {\times } n\) and a target matrix Y of size \(m {\times } N\), we consider the classical problem of selecting k columns from X that can be used to linearly approximate the entire matrix Y. The previous fastest nontrivial algorithms for this problem have a running time of O(mnN). We describe a significantly faster algorithm with a running time of \(O(km(n+N))\) with accuracy that compares favorably with the slower algorithms. We also derive bounds on the accuracy of the selections computed by our algorithm. These bounds show that our results are typically within a few percentage points of the optimal solution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Arai, H., Maung, C., Schweitzer, H.: Optimal column subset selection by A-star search. In: AAAI 2015, pp. 1079–1085. AAAI Press (2015)

    Google Scholar 

  2. Arai, H., Maung, C., Xu, K., Schweitzer, H.: Unsupervised feature selection by heuristic search with provable bounds on suboptimality. In: AAAI 2016 (2016)

    Google Scholar 

  3. Belmerhnia, L., Djermoune, E.H., Brie, D.: Greedy methods for simultaneous sparse approximation. In: 22nd European Signal Processing Conference (2014)

    Google Scholar 

  4. Çivril, A., Magdon-Ismail, M.: Column subset selection via sparse approximation of SVD. Theoret. Comput. Sci. 421, 1–14 (2012)

    Article  MathSciNet  Google Scholar 

  5. Chen, J., Huo, X.: Theoretical results of sparse representations of multiple measurement vectors. IEEE Trans. Signal Process. 54(12), 4634–4643 (2006)

    Article  Google Scholar 

  6. Cotter, S.F., Rao, B.D., Engen, K., Kreutz-Delgado, K.: Sparse solutions to linear inverse problems with multiple measurement vectors. ASP 53(7), 2477–2488 (2005)

    MathSciNet  MATH  Google Scholar 

  7. Daniel, J., Gragg, W., Kaufman, L., Stewart, G.: Reorthogonalization and stable algorithms for updating the Gram-Schmidt QR factorization. Math. Comput. 30, 772–795 (1976)

    MathSciNet  MATH  Google Scholar 

  8. Davis, G., Mallat, S., Avellaneda, M.: Adaptive greedy approximations. Constr. Approx. 13(1), 57–98 (1997)

    Article  MathSciNet  Google Scholar 

  9. Furnival, G., Wilson, R.: Regressions by leaps and bounds. Technometrics 42, 69–79 (1974)

    Article  Google Scholar 

  10. Golub, G.H., Van-Loan, C.F.: Matrix Computations, 4th edn. The Johns Hopkins University Press, Baltimore (2013)

    MATH  Google Scholar 

  11. Guha, T., Ward, R.K.: Learning sparse representations for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1576–1588 (2011)

    Article  Google Scholar 

  12. Halko, N., Martinsson, P., Tropp, J.: Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53, 217–288 (2011)

    Article  MathSciNet  Google Scholar 

  13. He, B., Shah, S., Maung, C., Arnold, G., Wan, G., Schweitzer, H.: Heuristic search algorithm for dimensionality reduction optimally combining feature selection and feature extraction. In: AAAI 2019, pp. 2280–2287. AAAI Press, California (2019)

    Google Scholar 

  14. Joneidi, M., et al.: Select to better learn: fast and accurate deep learning using data selection from nonlinear manifolds. In: CVPR, pp. 7819–7829 (2020)

    Google Scholar 

  15. Lerman, G., Maunu, T.: Fast, robust and non-convex subspace recovery. Inf. Inference J. IMA 7(2), 277–336 (2018)

    Article  MathSciNet  Google Scholar 

  16. Mairal, J., Bach, F., Ponce, J.: Sparse Modeling for Image and Vision Processing. Foundations and Trends in Computer Graphics and Vision (2014)

    Google Scholar 

  17. Malioutov, D., Cetin, M., Willsky, A.: A sparse signal reconstruction perspective for source localization with sensor arrays. IEEE Trans. Signal Process. 53, 3010–3022 (2005)

    Article  MathSciNet  Google Scholar 

  18. Mallat, S.: A Wavelet Tour of Signal Processing. Academic Press, Cambridge (1999)

    MATH  Google Scholar 

  19. Maung, C., Schweitzer, H.: Improved greedy algorithms for sparse approximation of a matrix in terms of another matrix. IEEE TKDE 27(3), 769–780 (2015)

    Google Scholar 

  20. Natarajan, B.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24, 227–234 (1995)

    Article  MathSciNet  Google Scholar 

  21. Qian, C., Yu, Y., Zhou, Z.: Subset selection by pareto optimization. In: Advances in Neural Information Processing Systems. Curran Associates, Inc. (2015)

    Google Scholar 

  22. Soussen, C., Gribonval, R., Idier, J., Herzet, C.: Joint k-step analysis of orthogonal matching pursuit and orthogonal least squares. IEEE Trans. Inf. Theory 59, 3158–3174 (2013)

    Article  MathSciNet  Google Scholar 

  23. Tibshirani, R.: Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. B 58, 267–288 (1996)

    MathSciNet  MATH  Google Scholar 

  24. Tropp, J.A.: Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inf. Theory 50(10), 2231–2242 (2004)

    Article  MathSciNet  Google Scholar 

  25. Tropp, J.A., Gilbert, A.C., Strauss, M.J.: Algorithms for simultaneous sparse approximation. Part I: greedy pursuit. Signal Process. 86(3), 572–588 (2006)

    MATH  Google Scholar 

  26. Wan, G., Schweitzer, H.: A new robust subspace recovery algorithm (student abstract). In: the 35th National Conference on Artificial Intelligence (AAAI) (2021)

    Google Scholar 

  27. Wright, J., Ma, Y., Mairal, J., Sapiro, G., Huang, T., Yan, S.: Sparse representation for computer vision and pattern recognition. Proc. IEEE 98, 1031–1044 (2010)

    Article  Google Scholar 

  28. Xu, X., Shi, Z.: Multi-objective based spectral unmixing for hyperspectral images. ISPRS J. Photogramm. Remote Sens. 124, 54–69 (2017)

    Article  Google Scholar 

  29. Xu, Y., Li, Z., Yang, J., Zhang, D.: A survey of dictionary learning algorithms for face recognition. IEEE Access 5, 8502–8514 (2017)

    Article  Google Scholar 

  30. Zaeemzadeh, A., Joneidi, M., Rahnavard, N., Shah, M.: Iterative projection and matching: finding structure-preserving representatives and its application to computer vision. In: CVPR 2019 (2019)

    Google Scholar 

  31. Zhang, Q., Liu, Y., Blum, R., Han, J., Tao, D.: Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: a review. Inf. Fusion 40, 57–75 (2018)

    Article  Google Scholar 

  32. Zhang, T.: Adaptive forward-backward greedy algorithm for sparse learning with linear models. In: Advances in Neural Information Processing Systems (2009)

    Google Scholar 

  33. Zhang, Z., Xu, Y., Yang, J., Li, X., Zhang, D.: A survey of sparse representation: algorithms and applications. IEEE Access 3, 490–530 (2015)

    Article  Google Scholar 

  34. Zhu, X., Hu, R., Lei, C., Thung, K.H., Zheng, W., Wang, C.: Low-rank hypergraph feature selection for multi-output regression. World Wide Web 22(2), 517–531 (2017). https://doi.org/10.1007/s11280-017-0514-5

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, The University of Texas at Dallas, Richardson, TX, 75080, USA

    Guihong Wan & Haim Schweitzer

Authors
  1. Guihong Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haim Schweitzer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guihong Wan .

Editor information

Editors and Affiliations

  1. IIIT, Hyderabad, Hyderabad, India

    Kamal Karlapalem

  2. Chinese University of Hong Kong, Shatin, Hong Kong

    Hong Cheng

  3. Virginia Tech, Arlington, VA, USA

    Naren Ramakrishnan

  4. Jawaharlal Nehru University, New Delhi, India

    R. K. Agrawal

  5. IIIT Hyderabad, Hyderabad, India

    P. Krishna Reddy

  6. University of Minnesota, Minneapolis, MN, USA

    Jaideep Srivastava

  7. IIIT Delhi, New Delhi, India

    Tanmoy Chakraborty

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wan, G., Schweitzer, H. (2021). A Fast Algorithm for Simultaneous Sparse Approximation. In: Karlapalem, K., et al. Advances in Knowledge Discovery and Data Mining. PAKDD 2021. Lecture Notes in Computer Science(), vol 12714. Springer, Cham. https://doi.org/10.1007/978-3-030-75768-7_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-75768-7_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-75767-0

  • Online ISBN: 978-3-030-75768-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation