Journal of Statistical Physics

, Volume 167, Issue 3–4, pp 462–475 | Cite as

PCA Meets RG

Article

Abstract

A system with many degrees of freedom can be characterized by a covariance matrix; principal components analysis focuses on the eigenvalues of this matrix, hoping to find a lower dimensional description. But when the spectrum is nearly continuous, any distinction between components that we keep and those that we ignore becomes arbitrary; it then is natural to ask what happens as we vary this arbitrary cutoff. We argue that this problem is analogous to the momentum shell renormalization group. Following this analogy, we can define relevant and irrelevant operators, where the role of dimensionality is played by properties of the eigenvalue density. These results also suggest an approach to the analysis of real data. As an example, we study neural activity in the vertebrate retina as it responds to naturalistic movies, and find evidence of behavior controlled by a nontrivial fixed point. Applied to financial data, our analysis separates modes dominated by sampling noise from a smaller but still macroscopic number of modes described by a non-Gaussian distribution.

Keywords

Renormalization group Neural networks Financial markets 

Notes

Acknowledgements

We thank D Amodei, MJ Berry II, and O Marre for making available the data of Ref. [23] and M Marsili for the data of Ref. [27]. We are especially grateful to G Biroli, J–P Bouchaud, MP Brenner, CG Callan, A Cavagna, I Giardina, MO Magnasco, A Nicolis, SE Palmer, G Parisi, and DJ Schwab for helpful discussions and comments on the manuscript. Work at CUNY was supported in part by the Swartz Foundation. Work at Princeton was supported in part by Grants from the National Science Foundation (PHY-1305525, PHY-1451171, and CCF-0939370) and the Simons Foundation.

References

  1. 1.
    Pearson, K.: On lines and planes of closest fit to systems of points in space. Philos. Mag. 2, 559–572 (1901)MATHCrossRefGoogle Scholar
  2. 2.
    Shlens, J.: A tutorial on principal components analysis. arXiv:1404.1100 [cs.LG] (2014)
  3. 3.
    Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA 79, 2554–2558 (1982)ADSMathSciNetCrossRefGoogle Scholar
  4. 4.
    Wilson, K.G.: Problems in physics with many scales of length. Sci. Am. 241, 158–179 (1979)CrossRefGoogle Scholar
  5. 5.
    Jona-Lasinio, G.: The renormalization group: a probabilistic view. Il Nuovo Cimento 26B, 99–119 (1975)ADSMathSciNetCrossRefGoogle Scholar
  6. 6.
    Wilson, K.G., Kogut, J.: The renormalization group and the \(\epsilon \) expansion. Phys. Rep. 12, 75–200 (1974)ADSCrossRefGoogle Scholar
  7. 7.
    Kadanoff, L.P.: Scaling laws for Ising models near \(T_c\). Physics 2, 263–272 (1966)Google Scholar
  8. 8.
    Kadanoff, L.P.: From simulation model to public policy: an examination of Forrester’s “Urban Dynamics”. Simulation 16, 261–268 (1971)CrossRefGoogle Scholar
  9. 9.
    Kadanoff, L.P., Weinblatt, H.: Public policy conclusions from urban growth models. IEEE Trans. Syst. Man Cybern. SMC–2, 139–165 (1972)Google Scholar
  10. 10.
    Bensimon, D., Kadanoff, L.P., Liang, S., Shraiman, B.I., Tang, C.: Viscous flows in two dimensions. Rev. Mod. Phys. 58, 977–999 (1986)ADSMATHCrossRefGoogle Scholar
  11. 11.
    Halsey, T.C., Jensen, M.H., Kadanoff, L.P., Procaccia, I., Shraiman, B.I.: Fractal measures and their singularities: the characterization of strange sets. Phys. Rev. A33, 1141–1151 (1986); erratum 34, 1601 (1986)Google Scholar
  12. 12.
    Constantin, P., Kadanoff, L.P.: Singularities in complex interfaces. Philos. Trans. R. Soc. Lond. Ser. A 333, 379–389 (1990)ADSMathSciNetMATHCrossRefGoogle Scholar
  13. 13.
    Bertozzi, A., Brenner, M., Dupont, T.F., Kadanoff, L.P.: Singularities and similarities in interface flows. In: Sirovich, L.P. (ed.) Trends and Perspectives in Applied Mathatematics. Springer Verlag Applied Math Series Vol. 100, pp. 155–208 (1994)Google Scholar
  14. 14.
    Coppersmith, S.N., Blank, R.D., Kadanoff, L.P.: Analysis of a population genetics model with mutation, selection, and pleitropy. J. Stat. Phys. 97, 429–459 (1999)ADSMATHCrossRefGoogle Scholar
  15. 15.
    Povinelli, M.L., Coppersmith, S.N., Kadanoff, L.P., Nagel, S.R., Venkataramani, S.C.: Noise stabilization of self-organized memories. Phys. Rev. E 59, 4970–4982 (1999)ADSCrossRefGoogle Scholar
  16. 16.
    Kadanoff, L.P.: More is the same: mean field theory and phase transitions. J. Stat. Phys. 137, 777–797 (2009)ADSMathSciNetMATHCrossRefGoogle Scholar
  17. 17.
    Kadanoff, L.P.: Relating theories via renormalization. Stud. Hist. Philos. Sci. B 44, 22–39 (2013)MathSciNetMATHGoogle Scholar
  18. 18.
    Kadanoff, L.P.: Reflections on Gibbs: from statistical physics to the Amistad. J. Stat. Phys. 156, 1–9 (2014)ADSMathSciNetMATHCrossRefGoogle Scholar
  19. 19.
    Kadanoff, L.P.: Innovations in statistical physics. Annu. Rev. Cond. Matter Phys. 6, 1–14 (2015)ADSCrossRefGoogle Scholar
  20. 20.
    Wilson, K.G., Fisher, M.E.: Critical exponents in 3.99 dimensions. Phys. Rev. Lett. 28, 240–243 (1972)ADSCrossRefGoogle Scholar
  21. 21.
    Amit, D.J., Martin-Mayor, V.: Field Theory, the Renormalization Group, and Critical Phenomena. Graphs to Computers, 3rd edn. World Scientific, Singapore (2005)MATHCrossRefGoogle Scholar
  22. 22.
    Binder, K.: Finite size scaling analysis of Ising model block distribution functions. Z. Phys. B 43, 119–140 (1981)ADSCrossRefGoogle Scholar
  23. 23.
    Tkačik, G., Marre, O., Amodei, D., Schneidman, E., Bialek, W., Berry II, M.J.: Searching for collective behavior in a large network of sensory neurons. PLoS Comput. Biol. 10, e1003408 (2014)ADSCrossRefGoogle Scholar
  24. 24.
    Abarbanel, H.D.I., Brown, R., Sidorowich, J.J., Tsimring, L.S.: The analysis of observed chaotic data in physical systems. Rev. Mod. Phys. 65, 1331–1392 (1993)ADSMathSciNetCrossRefGoogle Scholar
  25. 25.
    Mora, T., Bialek, W.: Are biological systems poised at criticality? J. Stat. Phys. 144, 268–302 (2011)ADSMathSciNetMATHCrossRefGoogle Scholar
  26. 26.
    Tkačik, G., Mora, T., Marre, O., Amodei, D., Palmer, S.E., Berry II, M.J., Bialek, W.: Thermodynamics and signatures of criticality in a network of neurons. Proc. Natl. Acad. Sci. USA 112, 11508–11513 (2015)ADSCrossRefGoogle Scholar
  27. 27.
    Marsili, M.: Dissecting financial markets: sectors and states. Quant. Financ. 2, 297–302 (2002)CrossRefGoogle Scholar
  28. 28.
    Lillo, F., Mantegna, R.N.: Variety and volatility in financial markets. Phys. Rev. E 62, 6126–6134 (2000)ADSCrossRefGoogle Scholar
  29. 29.
    Bouchaud, J.P., Potters, M: Financial applications. In: Akemann, G., Baik, J., Di Francesco, P. (eds.) The Oxford Handbook of Random Matrix Theory. Oxford University Press, Oxford (2011). arXiv:0910.1205 [q–fin.ST] (2009)
  30. 30.
    Bun, J., Allez, R., Bouchaud, J.P., Potters, M.: Rotational invariant estimator for general noisy matrices. arXiv:1502.06736 [cond–mat.stat–mech] (2015)
  31. 31.
    Bun, J., Bouchaud, J.-P., Potters, M.: Cleaning large correlation matrices: tools from random matrix theory. arXiv:1610.08104 [cond–mat.stat–mech] (2016)
  32. 32.
    Aygün, E., Erzan, A.: Spectral renormalization group theory on networks. J. Phys. Conf. Ser. 319, 012007 (2011)CrossRefGoogle Scholar
  33. 33.
    Castellana, M.: Real-space renormalization group analysis of a non-mean-field spin-glass. EPL 95, 47014 (2011)CrossRefGoogle Scholar
  34. 34.
    Angelini, M.C., Parisi, G., Ricci-Tersenghi, F.: Ensemble renormalization group for disordered systems. Phys. Rev. B 87, 134201 (2013)ADSCrossRefGoogle Scholar
  35. 35.
    Angelini, M.C., Biroli, G.: Spin glass in a field: a new zero-temperature fixed point in finite dimensions. Phys. Rev. Lett. 114, 095701 (2015)ADSCrossRefGoogle Scholar
  36. 36.
    Brown, K.S., Hill, C.C., Calero, G.A., Myers, C.R., Lee, K.H., Sethna, J.P., Cerione, R.A.: The statistical mechanics of complex signaling networks: Nerve growth factor aignaling. Phys. Biol. 1, 184–195 (2004)ADSCrossRefGoogle Scholar
  37. 37.
    Waterfall, J.J., Casey, F.P., Gutenkunst, R.N., Brown, K.S., Myers, C.R., Brouwer, P.W., Elser, V., Sethna, J.P.: Sloppy model universality class and the Vandermonde matrix. Phys. Rev. Lett. 97, 150601 (2006)ADSCrossRefGoogle Scholar
  38. 38.
    Gutenkunst, R.N., Waterfall, J.J., Casey, F.P., Brown, K.S., Myers, C.R., Sethna, J.P.: Universally sloppy parameter sensitivities in systems biology. PLoS Comput. Biol. 3, e189 (2007)ADSMathSciNetCrossRefGoogle Scholar
  39. 39.
    Transtrum, M.K., Machta, B.B., Sethna, J.P.: Geometry of nonlinear least squares with applications to sloppy models and optimization. Phys. Rev. E 83, 036701 (2011)ADSCrossRefGoogle Scholar
  40. 40.
    Matcha, B.B., Chachra, R., Transtrum, M.K., Sethna, J.P.: Parameter space compression underlies emergent theories and predictive models. Science 342, 604–607 (2013)ADSCrossRefGoogle Scholar
  41. 41.
    Mehta, P., Schwab, D.J.: An exact mapping between the variational renormalization group and deep learning. arXiv:1410.3831 [stat.ML] (2014)

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Initiative for the Theoretical Sciences, The Graduate CenterCity University of New YorkNew YorkUSA
  2. 2.Joseph Henry Laboratories of Physics, and Lewis–Sigler Institute for Integrative GenomicsPrinceton UniversityPrincetonUSA

Personalised recommendations