Notion of Information and Independent Component Analysis

Abstract

Partial orderings and measures of information for continuous univariate random variables with special roles of Gaussian and uniform distributions are discussed. The information measures and measures of non-Gaussianity including the third and fourth cumulants are generally used as projection indices in the projection pursuit approach for the independent component analysis. The connections between information, non-Gaussianity and statistical independence in the context of independent component analysis is discussed in detail.

This is a preview of subscription content, log in to check access.

References

  1. [1]

    A. R. Barron: Entropy and the central limit theorem. Ann. Probab. 14 (1986), 336–342.

    MathSciNet  Article  Google Scholar 

  2. [2]

    A. J. Bell, T. J. Sejnowski: An information-maximization approach to blind separation and blind deconvolution. Neural Comput. 7 (1995), 1129–1159.

    Article  Google Scholar 

  3. [3]

    P. J. Bickel, E. L. Lehmann: Descriptive statistics for nonparametric models. II: Location. Ann. Stat. 3 (1975), 1045–1069.

    MathSciNet  MATH  Google Scholar 

  4. [4]

    P. J. Bickel, E. L. Lehmann: Descriptive statistics for nonparametric models. III: Dispersion. Ann. Stat. 4 (1976), 1139–1158.

    MathSciNet  MATH  Google Scholar 

  5. [5]

    M. Bilodeau, D. Brenner: Theory of Multivariate Statistics. Springer Texts in Statistics, Springer, New York, 1999.

    Google Scholar 

  6. [6]

    H. Chernoff, I. R. Savage: Asymptotic normality and efficiency of certain nonparametric test statistics. Ann. Math. Stat. 29 (1958), 972–994.

    MathSciNet  Article  Google Scholar 

  7. [7]

    T. M. Cover, J. A. Thomas: Elements of Information Theory. Wiley Series in Telecommunications, John Wiley & Sons, New York, 1991.

    Google Scholar 

  8. [8]

    L. Faivishevsky, J. Goldberger: ICA based on a smooth estimation of the differential entropy. NIPS’08: Proceedings of the 21st International Conference on Neural Information Processing Systems. Curran Associates, New York, 2008, pp. 433–440.

    Google Scholar 

  9. [9]

    J. L. Hodges, Jr., E. L. Lehmann: The efficiency of some nonparametric competitors of the t-test. Ann. Math. Stat. 27 (1956), 324–335.

    MathSciNet  Article  Google Scholar 

  10. [10]

    P. J. Huber: Projection pursuit. Ann. Stat. 13 (1985), 435–475.

    MathSciNet  Article  Google Scholar 

  11. [11]

    A. Hyvärinen: New approximations of differential entropy for independent component analysis and projection pursuit. NIPS’ 97: Proceedings of the 1997 Conference on Advances in Neural Information Processing Systems 10. MIT Press, Cambridge, 1998, pp. 273–279.

    Google Scholar 

  12. [12]

    A. Hyvärinen, J. Karhunen, E. Oja: Independent Component Analysis. John Wiley & Sons, New York, 2001.

    Google Scholar 

  13. [13]

    M. C. Jones, R. Sibson: What is projection pursuit? J. R. Stat. Soc., Ser. A 150 (1987), 1–36.

    MathSciNet  Article  Google Scholar 

  14. [14]

    K. Kim, G. Shevlyakov: Why Gaussianity? IEEE Signal Processing Magazine 25 (2008), 102–113.

    Article  Google Scholar 

  15. [15]

    E. Kristiansson: Decreasing Rearrangement and Lorentz L(p, q) Spaces. Master Thesis. Department of Mathematics, Lulea University of Technology, Lulea, 2002; Available at http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.111.1244&rep=rep1&type=pdf.

    Google Scholar 

  16. [16]

    S. Kullback: Information Theory and Statistics. Wiley Publication in Mathematical Statistics, John Wiley & Sons, New York, 1959.

    Google Scholar 

  17. [17]

    E. G. Learned-Miller, J. W. Fisher III: ICA using spacings estimates of entropy. J. Mach. Learn. Res. 4 (2004), 1271–1295.

    MathSciNet  MATH  Google Scholar 

  18. [18]

    B. G. Lindsay, W. Yao: Fisher information matrix: A tool for dimension reduction, projection pursuit, independent component analysis, and more. Can. J. Stat. 40 (2012), 712–730.

    MathSciNet  Article  Google Scholar 

  19. [19]

    A. W. Marshall, I. Olkin: Inequalities: Theory of Majorization and Its Applications. Mathematics in Science and Engineering 143, Academic Press, New York, 1979.

    Google Scholar 

  20. [20]

    J. Miettinen, K. Nordhausen, H. Oja, S. Taskinen: Deflation-based FastICA with adaptive choices of nonlinearities. IEEE Trans. Signal Process 62 (2014), 5716–5724.

    MathSciNet  Article  Google Scholar 

  21. [21]

    J. Miettinen, K. Nordhausen, H. Oja, S. Taskinen, J. Virta: The squared symmetric FastICA estimator. Signal Process. 131 (2017), 402–411.

    Article  Google Scholar 

  22. [22]

    J. Miettinen, S. Taskinen, K. Nordhausen, H. Oja: Fourth moments and independent component analysis. Stat. Sci. 30 (2015), 372–390.

    MathSciNet  Article  Google Scholar 

  23. [23]

    K. Nordhausen, H. Oja: Independent component analysis: A statistical perspective. WIREs Comput. Stat. 10 (2018), Article ID e1440, 23 pages.

  24. [24]

    K. Nordhausen, H. Oja: Robust nonparametric inference. Annu. Rev. Stat. Appl. 5 (2018), 473–500.

    MathSciNet  Article  Google Scholar 

  25. [25]

    H. Oja: On location, scale, skewness and kurtosis of univariate distributions. Scand. J. Stat., Theory Appl. 8 (1981), 154–168.

    MathSciNet  MATH  Google Scholar 

  26. [26]

    J. E. Pečarić, F. Proschan, Y. L. Tong: Convex Functions, Partial Orderings, and Statistical Applications. Mathematics in Science and Engineering 187, Academic Press, Boston, 1992.

    Google Scholar 

  27. [27]

    J. V. Ryff: On the representation of doubly stochastic operators. Pac. J. Math. 13 (1963), 1379–1386.

    MathSciNet  Article  Google Scholar 

  28. [28]

    R. Serfling: Asymptotic relative efficiency in estimation. International Encyclopedia of Statistical Science. Springer, Berlin, 2011, pp. 68–72.

    Google Scholar 

  29. [29]

    C. E. Shannon: A mathematical theory of communication. Bell Syst. Tech. J. 27 (1948), 379–423.

    MathSciNet  Article  Google Scholar 

  30. [30]

    R. G. Staudte: The shapes of things to come: Probability density quantiles. Statistics 51 (2017), 782–800.

    MathSciNet  Article  Google Scholar 

  31. [31]

    R. G. Staudte, A. Xia: Divergence from, and convergence to, uniformity of probability density quantiles. Entropy 20 (2018), Article ID 317, 10 pages.

  32. [32]

    W. R. van Zwet: Convex Transformations of Random Variables. Mathematical Centre Tracts 7, Mathematisch Centrum, Amsterdam, 1964.

    Google Scholar 

  33. [33]

    V. Vigneron, C. Jutten: Fisher information in source separation problems. International Conference on Independent Component Analysis and Signal Separation. Lecture Notes in Computer Science 3195, Springer, Berlin, 2004, pp. 168–176.

    Google Scholar 

  34. [34]

    J. Virta: On characterizations of the covariance matrix. Available at https://arxiv.org/abs/1810.01147 (2018), 11 pages.

  35. [35]

    J. Virta, K. Nordhausen: On the optimal non-linearities for Gaussian mixtures in FastICA. Latent Variable Analysis and Signal Separation. Theoretical Computer Science and General Issues, Springer, Berlin, 2017, pp. 427–437.

    Google Scholar 

Download references

Acknowledgement

The authors wish to express their gratitude to the anonymous referee, whose insightful comments greatly helped improving the quality of the manuscript.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Una Radojičić.

Additional information

The work of K. Nordhausen has been supported by the Austrian Science Fund (FWF) Grant number P31881-N32.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Radojičić, U., Nordhausen, K. & Oja, H. Notion of Information and Independent Component Analysis. Appl Math 65, 311–330 (2020). https://doi.org/10.21136/AM.2020.0326-19

Download citation

Keywords

  • dispersion
  • entropy
  • kurtosis
  • partial ordering

MSC 2020

  • 62B10
  • 94A17
  • 62H99