A shrinkage approach to joint estimation of multiple covariance matrices

Abstract

In this paper, we propose a shrinkage framework for jointly estimating multiple covariance matrices by shrinking the sample covariance matrices towards the pooled sample covariance matrix. This framework allows us to borrow information across different groups. We derive the optimal shrinkage parameters under the Stein and quadratic loss functions, and prove that our derived estimators are asymptotically optimal when the sample size or the number of groups tends to infinity. Simulation studies demonstrate that our proposed shrinkage method performs favorably compared to the existing methods.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. Barnard J, McCulloch R, Meng X (2000) Modeling covariance matrices in terms of standard deviations and correlations, with application to shrinkage. Statistica Sinica 10:1281–1311

    MathSciNet  MATH  Google Scholar 

  2. Boik RJ (2002) Spectral models for covariance matrices. Biometrika 89:159–182

    MathSciNet  Article  Google Scholar 

  3. Boik RJ (2003) Principal component models for correlation matrices. Biometrika 90:679–701

    MathSciNet  Article  Google Scholar 

  4. Cai T, Li H, Liu W, Xie J (2016) Joint estimation of multiple high-dimensional precision matrices. Statistica Sinica 26:445–464

    MathSciNet  MATH  Google Scholar 

  5. Danaher P, Wang P, Witten DM (2014) The joint graphical lasso for inverse covariance estimation across multiple classes. J R Stat Soc Ser B 76:373–397

    MathSciNet  Article  Google Scholar 

  6. Daniels MJ (2006) Bayesian modeling of several covariance matrices and some results on propriety of the posterior for linear regression with correlated or heterogeneous errors. J Multivar Anal 97:1185–1207

    MathSciNet  Article  Google Scholar 

  7. Ferguson TS (1996) A course in large sample theory. Chapman & Hall, London

    Google Scholar 

  8. Friedman JH (1989) Regularized discriminant analysis. J Am Stat Assoc 84:165–175

    MathSciNet  Article  Google Scholar 

  9. Gaskins J, Daniels M (2016) Covariance partition priors: a Bayesian approach to simultaneous covariance estimation for longitudinal data. J Comput Gr Stat 25:167–186

    MathSciNet  Article  Google Scholar 

  10. Guionnet A (2009) Large random matrices: lectures on macroscopic asymptotics. Springer, Berlin

    Google Scholar 

  11. Guo J, Levina E, Michailidis G, Zhu J (2011) Joint estimation of multiple graphical models. Biometrika 98:1–15

    MathSciNet  Article  Google Scholar 

  12. Haff L (1980) Empirical Bayes estimation of the multivariate normal covariance matrix. Ann Stat 8:586–597

    MathSciNet  Article  Google Scholar 

  13. Haff L (1991) The variational form of certain Bayes estimators. Ann Stat 19:1163–1190

    MathSciNet  Article  Google Scholar 

  14. Hoff PD (2009) A hierarchical eigenmodel for pooled covariance estimation. J R Stat Soc Ser B 71:971–992

    MathSciNet  Article  Google Scholar 

  15. Ledoit O, Wolf M (2004) A well-conditioned estimator for large-dimensional covariance matrices. J Multivar Anal 88:365–411

    MathSciNet  Article  Google Scholar 

  16. Ledoit O, Wolf M (2012) Nonlinear shrinkage estimation of large-dimensional covariance matrices. Ann Stat 40:1024–1060

    MathSciNet  Article  Google Scholar 

  17. Le Y, Hastie T (2016) Sparse quadratic discriminant analysis and community Bayes, arXiv preprint arXiv:1407.4543

  18. Letac G, Massam H (2004) All invariant moments of the Wishart distribution. Scand J Stat 31:295–318

    MathSciNet  Article  Google Scholar 

  19. Manly BF, Rayner J (1987) The comparison of sample covariance matrices using likelihood ratio tests. Biometrika 74:841–847

    MathSciNet  Article  Google Scholar 

  20. Pourahmadi M, Daniels MJ, Park T (2007) Simultaneous modelling of the Cholesky decomposition of several covariance matrices. J Multivar Anal 98:568–587

    MathSciNet  Article  Google Scholar 

  21. Price BS, Geyer CJ, Rothman AJ (2015) Ridge fusion in statistical learning. J Comput Gr Stat 24:439–454

    MathSciNet  Article  Google Scholar 

  22. Ramey JA, Stein CK, Young PD, Young DM (2016) High-dimensional regularized discriminant analysis, arXiv preprint arXiv:1602.01182

  23. Schäfer J, Strimmer K (2005) A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Stat Appl Genet Mol Biol 4:1175–1189

    MathSciNet  Article  Google Scholar 

  24. Stein C (1956) Inadmissibility of the usual estimator of the mean of a multivariate normal distribution. Proc Third Berkeley Symp Math Stat Prob 1:197–206

    MathSciNet  MATH  Google Scholar 

  25. Tao T (2012) Topics in random matrix theory. American Mathematical Society, Providence

    Google Scholar 

  26. Tong T, Wang Y (2007) Optimal shrinkage estimation of variances with applications to microarray data analysis. J Am Stat Assoc 102:113–122

    MathSciNet  Article  Google Scholar 

  27. Tong T, Jang H, Wang Y (2012) James–Stein type estimators of variances. J Multivar Anal 107:232–243

    MathSciNet  Article  Google Scholar 

  28. Yang R, Berger JO (1994) Estimation of a covariance matrix using the reference prior. Ann Stat 22:1195–1211

    MathSciNet  Article  Google Scholar 

Download references

Acknowledgements

Zongliang Hu’s research was supported by the Natural Science Foundation grant (No. 2019083) of Shenzhen University, and the Foundation and Applied Basic Research Programs (No. 2019A1515110449) of Guangdong Province of China. Zhishui Hu’s research was partially supported by the National Natural Science Foundation of China grant (No. 11671373). Tiejun Tong’s research was supported by the General Research Fund (No. HKBU12303918), the National Natural Science Foundation of China (No. 11671338), and the Initiation Grant for Faculty Niche Research Areas (No. RC-IG-FNRA/17-18/13) of Hong Kong Baptist University. Yuedong Wang’s research was partially supported by the National Science Foundation grant (No. DMS-1507620). The authors also thank the editor, the associate editor, and two reviewers for their constructive comments that have led to a substantial improvement of this paper.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Tiejun Tong.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 173 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hu, Z., Hu, Z., Dong, K. et al. A shrinkage approach to joint estimation of multiple covariance matrices. Metrika (2020). https://doi.org/10.1007/s00184-020-00781-3

Download citation

Keywords

  • Covariance matrices
  • Joint estimation
  • Optimal estimator
  • Quadratic loss function
  • Shrinkage parameter
  • Stein loss function