Sparse Kalman filtering approaches to realized covariance estimation from high frequency financial data

  • Michael HoEmail author
  • Jack Xin
Full Length Paper Series B


Estimation of the covariance matrix of asset returns from high frequency data is complicated by asynchronous returns, market microstructure noise and jumps. One technique for addressing both asynchronous returns and market microstructure is the Kalman-Expectation-Maximization (KEM) algorithm. However the KEM approach assumes log-normal prices and does not address jumps in the return process which can corrupt estimation of the covariance matrix. In this paper we extend the KEM algorithm to price models that include jumps. We propose a sparse Kalman filtering approach to this problem. In particular we develop a Kalman Expectation Conditional Maximization algorithm to determine the unknown covariance as well as detecting the jumps. In order to promote a sparse estimate of the jumps,we consider both Laplace and the spike and slab jump priors. Numerical results using simulated data show that each of these approaches provide for improved covariance estimation relative to the KEM method in a variety of settings where jumps occur.


Spike and slab ECM Kalman filtering \(\ell _{1}\) regularization 

Mathematics Subject Classification

90C26 62P05 


Supplementary material


  1. 1.
    Aravkin, A., Bell, B., Burke, J., Pilonetto, G.: An \(\ell _{1}\) Laplace robust Kalman smoother. IEEE Trans. Autom. Control 56(12), 2898–2911 (2011)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Aït-Sahalia, Y., Fan, J., Xiu, D.: High-frequency covariance estimates with noisy and asynchronous financial data. J. Am. Stat. Assoc. 105(492), 1504–1517 (2010)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Aït-Sahalia, Y., Myklank, P., Zhang, L.: How often to sample a continuous-time process in the presence of market microstructure noise. Rev. Financ. Stud. 100, 1394–1411 (2005)Google Scholar
  4. 4.
    Bandi, F., Russell, J.: Separating microstructure noise from volatility. J. Financ. Econ. 79, 655–692 (2006)Google Scholar
  5. 5.
    Barndorff-Nielsen, O., Hansen, P., Lunde, A., Shephard, N.: Multivariate realised kernels: consistent positive semi-definite estimators of the covariation of equity prices with noise and non-synchronous trading. J. Econom. 162, 149–169 (2011)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Barry, C.B.: Portfolio analysis under uncertain means, variances and covariances. J. Finance 29, 515–522 (1974)Google Scholar
  7. 7.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Imaging Sci. 2(1), 183–202 (2009)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Bollerslev, T.: Modeling the coherence in short-run nominal exchange rates: a multivariate generalized ARCH approach. Rev. Econ. Stat. 72, 498–505 (1990)Google Scholar
  9. 9.
    Boudt, K., Croux, C., Laurent, S.: Outlyingness weighted covariation. J. Financ. Econom. 9(4), 657–684 (2011)Google Scholar
  10. 10.
    Boudt, K., Zhang, J.: Jump robust two time scale covariance estimation and realized volatility budgets. Quant. Finance 15(6), 1041–1054 (2015)MathSciNetzbMATHGoogle Scholar
  11. 11.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)zbMATHGoogle Scholar
  12. 12.
    Campbell, J., Lo, A., MacKinlay, A.C.: The Econometrics of Financial Markets. Princeton University Press, Princeton (1996)zbMATHGoogle Scholar
  13. 13.
    Candès, E., Wakin, M., Boyd, S.: Enhancing sparsity by reweighted \(\ell _{1}\) minimzation. J. Fourier Anal. Appl. 14, 877–905 (2008)MathSciNetzbMATHGoogle Scholar
  14. 14.
    Chan, W., Maheu, J.: Conditional jump dynamics in stock market returns. J. Bus. Econ. Stat. 20(3), 377–389 (2002)MathSciNetGoogle Scholar
  15. 15.
    Corsi, F., Peluso, S., Audrino, F.: Missing in asynchronicity: a Kalman-EM approach for multivariate realized covariance estimation. J. Appl. Econom. 30(3), 377–397 (2015)MathSciNetGoogle Scholar
  16. 16.
    DeMiguel, V., Garlappi, L., Uppal, R.: Optimal versus Naive diversification: how inefficient is the 1/n portfolio strategy? Rev. Financ. Stud. 22(5), 1915–1953 (2009)Google Scholar
  17. 17.
    Dempster, A., Laird, N., Rubin, D.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Ser. B 39(1), 1–38 (1977)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Fan, J., Li, Y., Yu, K.: Vast volatility matrix estimation using high frequency data for portfolio selection. J. Am. Stat. Assoc. 107(497), 412–428 (2012)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Fan, J., Wang, Y.: Multi-scale jump and volatility analysis for high-frequency financial data. J. Am. Stat. Assoc. 102(480), 1349–1362 (2007)MathSciNetzbMATHGoogle Scholar
  20. 20.
    Fink, D.: A compendium of conjugate priors. Technical Report, Montana State Univeristy (1997)Google Scholar
  21. 21.
    Ghahramani, Z., Hinton, G.E.: Variational learning for switching state-space models. Neural Comput. 12(4), 963–996 (2000)Google Scholar
  22. 22.
    Goldstein, T., Osher, S.: The split Bregman method for \(\ell _{1}\) regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009)MathSciNetzbMATHGoogle Scholar
  23. 23.
    Jobson, J.D., Korkie, B.: Estimation for Markowitz efficient portfolios. J. Am. Stat. Assoc. 75, 544–554 (1980)MathSciNetzbMATHGoogle Scholar
  24. 24.
    Karpoff, J.: The relation between price changes and trading volume: a survey. J. Financ. Quant. Anal. 22, 109–126 (1987)Google Scholar
  25. 25.
    Liu, C., Tang, C.Y.: A quasi-maximum likelihood approach for integrated covariance matrix estimation with high frequency data. J. Econom. 180, 217–232 (2014)MathSciNetzbMATHGoogle Scholar
  26. 26.
    Lo, A., MacKinlay, A.C.: An econometric analysis of nonsynchronous trading. J. Econom. 45, 181–211 (1990)MathSciNetzbMATHGoogle Scholar
  27. 27.
    Lunenberger, D., Ye, Y.: Linear and Nonlinear Programming. Addison-Wesley, New York (2008)Google Scholar
  28. 28.
    Maheu, J.M., McCurdy, T.H.: News arrival, jump dynamics, and volatility components for individual stock returns. J. Finance 59(2), 755–793 (2004)Google Scholar
  29. 29.
    Mattingley, J., Boyd, S.: Real-time convex optimization in signal processing. IEEE Signal Process. Mag. 27, 50–61 (2010)zbMATHGoogle Scholar
  30. 30.
    Meng, X.L., Rubin, D.: Maximum likelihood estimation via the ECM algorithm: a general framework. Biometrika 80, 267–278 (1993)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Peluso, S., Corsi, F., Mira, A.: A Bayesian high-frequency estimator of the multivariate covariance of noisy and asynchronous returns. J. Financ. Econom. 13(3), 665–697 (2015)Google Scholar
  32. 32.
    Roll, R.: A simple implicit measure of the effective bid-ask spread in an efficient market. J. Finance 39(4), 1127–1139 (1984)Google Scholar
  33. 33.
    Seeger, M.W.: Bayesian inference and optimal design for the sparse linear model. J. Mach. Learn. Res. 9, 759–813 (2008)MathSciNetzbMATHGoogle Scholar
  34. 34.
    Shumway, R., Stoffer, D.: Time Series Analysis and its Applications with R Examples. Springer, Berlin (2011)zbMATHGoogle Scholar
  35. 35.
    Shumway, R.H., Stoffer, D.S.: An approach to time series smoothing and forecasting using the EM algorithm. J. Time Ser. Anal. 3(4), 253–264 (1982)zbMATHGoogle Scholar
  36. 36.
    Wu, C.F.: On the convergence properties of the EM algorithm. Ann. Stat. 11(1), 95–103 (1983)MathSciNetzbMATHGoogle Scholar
  37. 37.
    Zangwill, W.: Nonlinear Programming: A Unified Approach. Prentice-Hall, Upper Saddle River (1969)zbMATHGoogle Scholar
  38. 38.
    Zhang, L.: Estimating covariation: Epps effect and microstructure noises. J. Econ. 160(1), 33–47 (2011)MathSciNetzbMATHGoogle Scholar
  39. 39.
    Zhang, M., Russel, J., Tsay, R.: Determinants of bid and ask quotes and implications for the cost of trading. J. Empir. Finance 15(4), 656–678 (2008)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature and Mathematical Optimization Society 2019

Authors and Affiliations

  1. 1.Department of MathematicsUniversity of California at IrvineIrvineUSA

Personalised recommendations