Implicit Sampling, with Application to Data Assimilation



There are many computational tasks in which it is necessary to sample a given probability density function (or pdf for short), i.e., to use a computer to construct a sequence of independent random vectors x i (i=1,2,…), whose histogram converges to the given pdf. This can be difficult because the sample space can be huge, and more importantly, because the portion of the space where the density is significant, can be very small, so that one may miss it by an ill-designed sampling scheme. Indeed, Markov-chain Monte Carlo, the most widely used sampling scheme, can be thought of as a search algorithm, where one starts at an arbitrary point and one advances step-by-step towards the high probability region of the space. This can be expensive, in particular because one is typically interested in independent samples, while the chain has a memory. The authors present an alternative, in which samples are found by solving an algebraic equation with a random right-hand side rather than by following a chain; each sample is independent of the previous samples. The construction is explained in the context of numerical integration, and it is then applied to data assimilation.


Importance sampling Bayesian estimation Particle filter Implicit filter Data assimilation 

Mathematics Subject Classification

11K45 34K60 62M20 65C50 93E11 


  1. 1.
    Doucet, A., de Freitas, N., Gordon, N.: Sequential Monte Carlo Methods in Practice. Springer, New York (2001) CrossRefzbMATHGoogle Scholar
  2. 2.
    Chorin, A.J., Hald, O.H.: Stochastic Tools in Mathematics and Science, 2nd edn. Springer, New York (2009) CrossRefzbMATHGoogle Scholar
  3. 3.
    Morzfeld, M., Tu, X., Atkins, E., Chorin, A.J.: A random map implementation of implicit filters. J. Comput. Phys. 231, 2049–2066 (2012) MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Morzfeld, M., Chorin, A.J.: Implicit particle filtering for models with partial noise, and an application to geomagnetic data assimilation. Nonlinear Process. Geophys. 19, 365–382 (2012) CrossRefGoogle Scholar
  5. 5.
    Kloeden, P.E., Platen, E.: Numerical Solution of Stochastic Differential Equations, 3rd edn. Springer, New York (1999) Google Scholar
  6. 6.
    Chorin, A.J., Tu, X.: Implicit sampling for particle filters. Proc. Natl. Acad. Sci. USA 106, 17249–17254 (2009) CrossRefGoogle Scholar
  7. 7.
    Chorin, A.J., Morzfeld, M., Tu, X.: Implicit particle filters for data assimilation. Commun. Appl. Math. Comput. Sci. 5(2), 221–240 (2010) MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Arulampalam, M.S., Maskell, S., Gordon, N., Clapp, T.: A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Trans. Signal Process. 10, 197–208 (2002) Google Scholar
  9. 9.
    Bickel, P., Li, B., Bengtsson, T.: Sharp failure rates for the bootstrap particle filter in high dimensions. In: Pushing the Limits of Contemporary Statistics: Contributions in Honor of Jayanta K. Ghosh, pp. 318–329 (2008) CrossRefGoogle Scholar
  10. 10.
    Snyder, C.C., Bengtsson, T., Bickel, P., Anderson, J.: Obstacles to high-dimensional particle filtering. Mon. Weather Rev. 136, 4629–4640 (2008) CrossRefGoogle Scholar
  11. 11.
    Gordon, N.J., Salmon, D.J., Smith, A.F.M.: Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEE Proc., F, Radar Signal Process. 140, 107–113 (1993) CrossRefGoogle Scholar
  12. 12.
    Doucet, A., Godsill, S., Andrieu, C.: On sequential Monte Carlo sampling methods for Bayesian filtering. Stat. Comput. 50, 174–188 (2000) Google Scholar
  13. 13.
    Del Moral, P.: Feynman-Kac Formulae. Springer, New York (2004) CrossRefzbMATHGoogle Scholar
  14. 14.
    Del Moral, P.: Measure-valued processes and interacting particle systems. Application to nonlinear filtering problems. Ann. Appl. Probab. 8(2), 438–495 (1998) MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Zaritskii, V.S., Shimelevich, L.I.: Monte Carlo technique in problems of optimal data processing. Autom. Remote Control 12, 95–103 (1975) MathSciNetGoogle Scholar
  16. 16.
    Kalman, R.E.: A new approach to linear filtering and prediction theory. J. Basic Eng. 82, 35–48 (1960) CrossRefGoogle Scholar
  17. 17.
    Kalman, R.E., Bucy, R.S.: New results in linear filtering and prediction theory. J. Basic Eng. 83, 95–108 (1961) MathSciNetCrossRefGoogle Scholar
  18. 18.
    Evensen, G.: Data Assimilation. Springer, New York (2007) zbMATHGoogle Scholar
  19. 19.
    Zakai, M.: On the optimal filtering of diffusion processes. Z. Wahrscheinlichkeitstheor. Verw. Geb. 11, 230–243 (1969) MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Talagrand, O., Courtier, P.: Variational assimilation of meteorological observations with the adjoint vorticity equation. I. Theory. Q. J. R. Meteorol. Soc. 113, 1311–1328 (1987) CrossRefGoogle Scholar
  21. 21.
    Bennet, A.F., Leslie, L.M., Hagelberg, C.R., Powers, P.E.: A cyclone prediction using a barotropic model initialized by a general inverse method. Mon. Weather Rev. 121, 1714–1728 (1993) CrossRefGoogle Scholar
  22. 22.
    Courtier, P., Thepaut, J.N., Hollingsworth, A.: A strategy for operational implementation of 4D-var, using an incremental appoach. Q. J. R. Meteorol. Soc. 120, 1367–1387 (1994) CrossRefGoogle Scholar
  23. 23.
    Courtier, P.: Dual formulation of four-dimensional variational assimilation. Q. J. R. Meteorol. Soc. 123, 2449–2461 (1997) CrossRefGoogle Scholar
  24. 24.
    Talagrand, O.: Assimilation of observations, an introduction. J. Meteorol. Soc. Jpn. 75(1), 191–209 (1997) MathSciNetGoogle Scholar
  25. 25.
    Tremolet, Y.: Accounting for an imperfect model in 4D-var. Q. J. R. Meteorol. Soc. 621(132), 2483–2504 (2006) CrossRefGoogle Scholar
  26. 26.
    Atkins, E., Morzfeld, M., Chorin, A.J.: Implicit particle methods and their connection to variational data assimilation. Mon. Weather Rev. (2013, in press) Google Scholar
  27. 27.
    Kuramoto, Y., Tsuzuki, T.: On the formation of dissipative structures in reaction-diffusion systems. Prog. Theor. Phys. 54, 687–699 (1975) CrossRefGoogle Scholar
  28. 28.
    Sivashinsky, G.: Nonlinear analysis of hydrodynamic instability in laminar flames. Part I. Derivation of basic equations. Acta Astronaut. 4, 1177–1206 (1977) MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Chorin, A.J., Krause, P.: Dimensional reduction for a Bayesian filter. Proc. Natl. Acad. Sci. USA 101, 15013–15017 (2004) MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Jardak, M., Navon, I.M., Zupanski, M.: Comparison of sequential data assimilation methods for the Kuramoto-Sivashinsky equation. Int. J. Numer. Methods Fluids 62, 374–402 (2009) MathSciNetGoogle Scholar
  31. 31.
    Lord, G.J., Rougemont, J.: A numerical scheme for stochastic PDEs with Gevrey regularity. IMA J. Numer. Anal. 24, 587–604 (2004) MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Jentzen, A., Kloeden, P.E.: Overcoming the order barrier in the numerical approximation of stochastic partial differential equations with additive space-time noise. Proc. R. Soc. A 465, 649–667 (2009) MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Fletcher, R.: Practical Methods of Optimization. Wiley, New York (1987) zbMATHGoogle Scholar
  34. 34.
    Nocedal, J., Wright, S.T.: Numerical Optimization, 2nd edn. Springer, New York (2006) zbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.Department of MathematicsUniversity of CaliforniaBerkeleyUSA
  2. 2.Lawrence Berkeley National LaboratoryBerkeleyUSA
  3. 3.Department of MathematicsUniversity of KansasLawrenceUSA

Personalised recommendations