Skip to main content
SpringerLink
Log in

Notion of Information and Independent Component Analysis

  • Published:
Applications of Mathematics Aims and scope Submit manuscript

Abstract

Partial orderings and measures of information for continuous univariate random variables with special roles of Gaussian and uniform distributions are discussed. The information measures and measures of non-Gaussianity including the third and fourth cumulants are generally used as projection indices in the projection pursuit approach for the independent component analysis. The connections between information, non-Gaussianity and statistical independence in the context of independent component analysis is discussed in detail.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. A. R. Barron: Entropy and the central limit theorem. Ann. Probab. 14 (1986), 336–342.

    Article  MathSciNet  Google Scholar 

  2. A. J. Bell, T. J. Sejnowski: An information-maximization approach to blind separation and blind deconvolution. Neural Comput. 7 (1995), 1129–1159.

    Article  Google Scholar 

  3. P. J. Bickel, E. L. Lehmann: Descriptive statistics for nonparametric models. II: Location. Ann. Stat. 3 (1975), 1045–1069.

    MathSciNet  MATH  Google Scholar 

  4. P. J. Bickel, E. L. Lehmann: Descriptive statistics for nonparametric models. III: Dispersion. Ann. Stat. 4 (1976), 1139–1158.

    MathSciNet  MATH  Google Scholar 

  5. M. Bilodeau, D. Brenner: Theory of Multivariate Statistics. Springer Texts in Statistics, Springer, New York, 1999.

    MATH  Google Scholar 

  6. H. Chernoff, I. R. Savage: Asymptotic normality and efficiency of certain nonparametric test statistics. Ann. Math. Stat. 29 (1958), 972–994.

    Article  MathSciNet  Google Scholar 

  7. T. M. Cover, J. A. Thomas: Elements of Information Theory. Wiley Series in Telecommunications, John Wiley & Sons, New York, 1991.

    Book  Google Scholar 

  8. L. Faivishevsky, J. Goldberger: ICA based on a smooth estimation of the differential entropy. NIPS’08: Proceedings of the 21st International Conference on Neural Information Processing Systems. Curran Associates, New York, 2008, pp. 433–440.

    Google Scholar 

  9. J. L. Hodges, Jr., E. L. Lehmann: The efficiency of some nonparametric competitors of the t-test. Ann. Math. Stat. 27 (1956), 324–335.

    Article  MathSciNet  Google Scholar 

  10. P. J. Huber: Projection pursuit. Ann. Stat. 13 (1985), 435–475.

    Article  MathSciNet  Google Scholar 

  11. A. Hyvärinen: New approximations of differential entropy for independent component analysis and projection pursuit. NIPS’ 97: Proceedings of the 1997 Conference on Advances in Neural Information Processing Systems 10. MIT Press, Cambridge, 1998, pp. 273–279.

    Google Scholar 

  12. A. Hyvärinen, J. Karhunen, E. Oja: Independent Component Analysis. John Wiley & Sons, New York, 2001.

    Book  Google Scholar 

  13. M. C. Jones, R. Sibson: What is projection pursuit? J. R. Stat. Soc., Ser. A 150 (1987), 1–36.

    Article  MathSciNet  Google Scholar 

  14. K. Kim, G. Shevlyakov: Why Gaussianity? IEEE Signal Processing Magazine 25 (2008), 102–113.

    Article  Google Scholar 

  15. E. Kristiansson: Decreasing Rearrangement and Lorentz L(p, q) Spaces. Master Thesis. Department of Mathematics, Lulea University of Technology, Lulea, 2002; Available at http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.111.1244&rep=rep1&type=pdf.

    Google Scholar 

  16. S. Kullback: Information Theory and Statistics. Wiley Publication in Mathematical Statistics, John Wiley & Sons, New York, 1959.

    Google Scholar 

  17. E. G. Learned-Miller, J. W. Fisher III: ICA using spacings estimates of entropy. J. Mach. Learn. Res. 4 (2004), 1271–1295.

    MathSciNet  MATH  Google Scholar 

  18. B. G. Lindsay, W. Yao: Fisher information matrix: A tool for dimension reduction, projection pursuit, independent component analysis, and more. Can. J. Stat. 40 (2012), 712–730.

    Article  MathSciNet  Google Scholar 

  19. A. W. Marshall, I. Olkin: Inequalities: Theory of Majorization and Its Applications. Mathematics in Science and Engineering 143, Academic Press, New York, 1979.

    Google Scholar 

  20. J. Miettinen, K. Nordhausen, H. Oja, S. Taskinen: Deflation-based FastICA with adaptive choices of nonlinearities. IEEE Trans. Signal Process 62 (2014), 5716–5724.

    Article  MathSciNet  Google Scholar 

  21. J. Miettinen, K. Nordhausen, H. Oja, S. Taskinen, J. Virta: The squared symmetric FastICA estimator. Signal Process. 131 (2017), 402–411.

    Article  Google Scholar 

  22. J. Miettinen, S. Taskinen, K. Nordhausen, H. Oja: Fourth moments and independent component analysis. Stat. Sci. 30 (2015), 372–390.

    Article  MathSciNet  Google Scholar 

  23. K. Nordhausen, H. Oja: Independent component analysis: A statistical perspective. WIREs Comput. Stat. 10 (2018), Article ID e1440, 23 pages.

  24. K. Nordhausen, H. Oja: Robust nonparametric inference. Annu. Rev. Stat. Appl. 5 (2018), 473–500.

    Article  MathSciNet  Google Scholar 

  25. H. Oja: On location, scale, skewness and kurtosis of univariate distributions. Scand. J. Stat., Theory Appl. 8 (1981), 154–168.

    MathSciNet  MATH  Google Scholar 

  26. J. E. Pečarić, F. Proschan, Y. L. Tong: Convex Functions, Partial Orderings, and Statistical Applications. Mathematics in Science and Engineering 187, Academic Press, Boston, 1992.

    Google Scholar 

  27. J. V. Ryff: On the representation of doubly stochastic operators. Pac. J. Math. 13 (1963), 1379–1386.

    Article  MathSciNet  Google Scholar 

  28. R. Serfling: Asymptotic relative efficiency in estimation. International Encyclopedia of Statistical Science. Springer, Berlin, 2011, pp. 68–72.

    Chapter  Google Scholar 

  29. C. E. Shannon: A mathematical theory of communication. Bell Syst. Tech. J. 27 (1948), 379–423.

    Article  MathSciNet  Google Scholar 

  30. R. G. Staudte: The shapes of things to come: Probability density quantiles. Statistics 51 (2017), 782–800.

    Article  MathSciNet  Google Scholar 

  31. R. G. Staudte, A. Xia: Divergence from, and convergence to, uniformity of probability density quantiles. Entropy 20 (2018), Article ID 317, 10 pages.

  32. W. R. van Zwet: Convex Transformations of Random Variables. Mathematical Centre Tracts 7, Mathematisch Centrum, Amsterdam, 1964.

    Google Scholar 

  33. V. Vigneron, C. Jutten: Fisher information in source separation problems. International Conference on Independent Component Analysis and Signal Separation. Lecture Notes in Computer Science 3195, Springer, Berlin, 2004, pp. 168–176.

    Chapter  Google Scholar 

  34. J. Virta: On characterizations of the covariance matrix. Available at https://arxiv.org/abs/1810.01147 (2018), 11 pages.

  35. J. Virta, K. Nordhausen: On the optimal non-linearities for Gaussian mixtures in FastICA. Latent Variable Analysis and Signal Separation. Theoretical Computer Science and General Issues, Springer, Berlin, 2017, pp. 427–437.

    Chapter  Google Scholar 

Download references

Acknowledgement

The authors wish to express their gratitude to the anonymous referee, whose insightful comments greatly helped improving the quality of the manuscript.

Author information

Authors and Affiliations

  1. CSTAT-Computational Statistics, Institute of Statistics & Mathematical Methods in Economics, Vienna University of Technology, Wiedner Hauptstraße 7, 1040, Vienna, Austria

    Una Radojičić & Klaus Nordhausen

  2. Department of Mathematics and Statistics, University of Turku, 20014, Turku, Finland

    Hannu Oja

Authors
  1. Una Radojičić
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Klaus Nordhausen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hannu Oja
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Una Radojičić.

Additional information

The work of K. Nordhausen has been supported by the Austrian Science Fund (FWF) Grant number P31881-N32.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Radojičić, U., Nordhausen, K. & Oja, H. Notion of Information and Independent Component Analysis. Appl Math 65, 311–330 (2020). https://doi.org/10.21136/AM.2020.0326-19

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.21136/AM.2020.0326-19

Keywords

MSC 2020

Search

Navigation