Skip to main content
SpringerLink
Log in

Nonlinear Dimensionality Reduction by Local Orthogonality Preserving Alignment

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

We present a new manifold learning algorithm called Local Orthogonality Preserving Alignment (LOPA). Our algorithm is inspired by the Local Tangent Space Alignment (LTSA) method that aims to align multiple local neighborhoods into a global coordinate system using affine transformations. However, LTSA often fails to preserve original geometric quantities such as distances and angles. Although an iterative alignment procedure for preserving orthogonality was suggested by the authors of LTSA, neither the corresponding initialization nor the experiments were given. Procrustes Subspaces Alignment (PSA) implements the orthogonality preserving idea by estimating each rotation transformation separately with simulated annealing. However, the optimization in PSA is complicated and multiple separated local rotations may produce globally contradictive results. To address these difficulties, we first use the pseudo-inverse trick of LTSA to represent each local orthogonal transformation with the unified global coordinates. Second the orthogonality constraints are relaxed to be an instance of semi-definite programming (SDP). Finally a two-step iterative procedure is employed to further reduce the errors in orthogonal constraints. Extensive experiments show that LOPA can faithfully preserve distances, angles, inner products, and neighborhoods of the original datasets. In comparison, the embedding performance of LOPA is better than that of PSA and comparable to that of state-of-the-art algorithms like MVU and MVE, while the runtime of LOPA is significantly faster than that of PSA, MVU and MVE.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Tenenbaum J B, de Silva V, Langford J C. A global geometric framework for nonlinear dimensionality reduction. Science, 2000, 290(5500): 2319–2323.

    Article  Google Scholar 

  2. Roweis S T, Saul L K. Nonlinear dimensionality reduction by locally linear embedding. Science, 2000, 290(5500): 2323–2326.

    Article  Google Scholar 

  3. Saul L K, Roweis S T. Think globally, fit locally: Unsupervised learning of low dimensional manifolds. The Journal of Machine Learning Research, 2003, 4: 119–155.

    MathSciNet  MATH  Google Scholar 

  4. Seung H S, Lee D D. The manifold ways of perception. Science, 2000, 290(5500): 2268–2269.

    Article  Google Scholar 

  5. Donoho D L. High-dimensional data analysis: The curses and blessings of dimensionality. In Proc. AMS Mathematical Challenges of the 21st Century, Aug. 2000.

  6. Pei Y, Huang F, Shi F, Zha H. Unsupervised image matching based on manifold alignment. IEEE Trans. Pattern Analysis and Machine Intelligence (TPAMI), 2012, 34(8): 1658–1664.

  7. Zhang J, Huang H, Wang J. Manifold learning for visualizing and analyzing high-dimensional data. IEEE Intelligent Systems, 2010, 25(4): 54–61.

    Google Scholar 

  8. Weinberger K Q, Saul L K. Unsupervised learning of image manifolds by semidefinite programming. International Journal of Computer Vision, 2006, 70(1): 77–90.

    Article  Google Scholar 

  9. Weinberger K Q, Sha F, Saul L K. Learning a kernel matrix for nonlinear dimensionality reduction. In Proc. the 21st Int. Conf. Machine Learning, July 2004.

  10. Shaw B, Jebara T. Minimum volume embedding. In Proc. the 11th Int. Conf. Artificial Intelligence and Statistics, Mar. 2007, pp.460-467.

  11. Lin T, Zha H. Riemannian manifold learning. TPAMI, 2008, 30(5): 796–809.

    Article  Google Scholar 

  12. Sha F, Saul L K. Analysis and extension of spectral methods for nonlinear dimensionality reduction. In Proc. the 22nd Int. Conf. Machine Learning, Aug. 2005, pp.784-791.

  13. Donoho D L, Grimes C. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proc. the National Academy of Sciences, 2003, 100(10): 5591–5596.

    Article  MathSciNet  MATH  Google Scholar 

  14. Belkin M, Niyogi P. Laplacian Eigenmaps for dimensionality reduction and data representation. Neural Computation, 2003, 15(6): 1373–1396.

    Article  MATH  Google Scholar 

  15. Zhang Z, Zha H. Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM Journal of Scientific Computing, 2004, 26(1): 313–338.

    Article  MathSciNet  MATH  Google Scholar 

  16. Qiao H, Zhang P, Wang D, Zhang B. An explicit nonlinear mapping for manifold learning. IEEE Trans. Cybernetics, 2013, 43(1): 51–63.

    Article  Google Scholar 

  17. Yang L. Alignment of overlapping locally scaled patches for multidimensional scaling and dimensionality reduction. TPAMI, 2008, 30(3): 438–450.

    Article  Google Scholar 

  18. Wang R, Shan S, Chen X, Chen J, Gao W. Maximal linear embedding for dimensionality reduction. TPAMI, 2011, 33(9): 1776–1792.

    Article  Google Scholar 

  19. Gashler M, Ventura D, Martinez T. Manifold learning by graduated optimization. IEEE Trans. Systems, Man, and Cybernetics – Part B: Cybernetics, 2011, 41(6): 1458–1470.

    Article  Google Scholar 

  20. Venna J, Peltonen J, Nybo K, Aidos H, Kaski S. Information retrieval perspective to nonlinear dimensionality reduction for data visualization. The Journal of Machine Learning Research, 2010, 11: 451–490.

    MathSciNet  MATH  Google Scholar 

  21. Goldberg Y, Ritov Y. Local Procrustes for manifold embedding: A measure of embedding quality and embedding algorithms. Machine Learning, 2009, 77(1): 1–25.

    Article  Google Scholar 

  22. do Carmo M P. Riemannian Geometry (1st edition). Birkhäuser, Boston, 1992.

  23. Perraul-Joncas D, Meila M. Non-linear dimensionality reduction: Riemannian metric estimation and the problem of geometric discovery. arXiv:1305.7255, 2013.

  24. Kokiopoulou E, Saad Y. Orthogonal neighborhood preserving projections: A projection-based dimensionality reduction technique. TPAMI, 2007, 29(12): 2143–2156.

    Article  Google Scholar 

  25. Wen Z, Yin W. A feasible method for optimization with orthogonality constraints. Mathematical Programming, 2013, 142(1/2): 397–434.

    Article  MathSciNet  MATH  Google Scholar 

  26. Borchers B. CSDP, A C library for semidefinite programming. Optimization Methods and Software, 1999, 11(1/2/3/4): 613–623.

  27. Toh K, Todd M, Tütüncü R. SDPT3 — A Matlab software package for semidefinite programming, version 1.3. Optimization Methods and Software, 1999, 11(1–4): 545–581.

  28. Sturm J. Using SeDuMi 1.02, a Matlab toolbox for optimization over symmetric cones. Optimization Methods and Software, 1999, 11(1/2/3/4): 625–653.

  29. Vandenberghe L, Boyd S. Semidefinite programming. SIAM Review, 1996, 38(1): 49–95.

    Article  MathSciNet  MATH  Google Scholar 

  30. Golub G H, van Van Loan C F. Matrix Computations (3rd edition). The Johns Hopkins University Press, 1996.

  31. van der Maaten L, Postma E, van den Herik J. Dimensionality reduction: A comparative review. Technical Report, TiCC TR 2009–005, Tilburg Centre for Creative Computing, Tilburg University, Oct. 2009.

Download references

Author information

Authors and Affiliations

  1. The Key Laboratory of Machine Perception (Ministry of Education), School of Electrical Engineering and Computer Sciences, Peking University, Beijing, 100871, China

    Tong Lin, Yao Liu, Bo Wang, Li-Wei Wang & Hong-Bin Zha

Authors
  1. Tong Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Li-Wei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hong-Bin Zha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tong Lin.

Additional information

Special Section of CVM 2016

This work was supported by the National Basic Research 973 Program of China under Grant No. 2011CB302202, the National Natural Science Foundation of China under Grant Nos. 61375051 and 61075119, and the Seeding Grant for Medicine and Information Sciences of Peking University of China under Grant No. 2014-MI-21.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, T., Liu, Y., Wang, B. et al. Nonlinear Dimensionality Reduction by Local Orthogonality Preserving Alignment. J. Comput. Sci. Technol. 31, 512–524 (2016). https://doi.org/10.1007/s11390-016-1644-4

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-016-1644-4

Keywords

Search

Navigation