Statistics and Computing

, Volume 29, Issue 1, pp 23–32 | Cite as

Double-Parallel Monte Carlo for Bayesian analysis of big data

  • Jingnan Xue
  • Faming LiangEmail author


This paper proposes a simple, practical, and efficient MCMC algorithm for Bayesian analysis of big data. The proposed algorithm suggests to divide the big dataset into some smaller subsets and provides a simple method to aggregate the subset posteriors to approximate the full data posterior. To further speed up computation, the proposed algorithm employs the population stochastic approximation Monte Carlo algorithm, a parallel MCMC algorithm, to simulate from each subset posterior. Since this algorithm consists of two levels of parallel, data parallel and simulation parallel, it is coined as “Double-Parallel Monte Carlo.” The validity of the proposed algorithm is justified mathematically and numerically.


Embarrassingly parallel Divide-and-combine MCMC Pop-SAMC Subset posterior aggregation 



Liang’s research was supported in part by the Grants DMS-1545202, DMS-1612924 and R01-GM117597. The authors thank the Editor, Associate Editor and two referees for their constructive comments which has led to significant improvement of this paper.


  1. Andrieu, C., Moulines, E., Priouret, P.: Stability of stochastic approximation under verifiable conditions. SIAM J. Control Optim. 44, 283–312 (2005). MathSciNetCrossRefzbMATHGoogle Scholar
  2. Carvalho, C.M., Polson, N.G., Scott, J.G.: The horseshoe estimator for sparse signals. Biometrika 97, 465–480 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  3. Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans. Pattern Anal. Mach. Intell. 6, 721–741 (1984). CrossRefzbMATHGoogle Scholar
  4. Hasting, W.: Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57, 97–109 (1970). MathSciNetCrossRefGoogle Scholar
  5. Kass, R.E., Tierney, L., Kadane, J.B.: The validity of posterior expansions based on Laplace’s method. In: Geisser, S., Hodges, J.S., Press, S.J., ZeUnerv, A. (eds.) Bayesian and Likelihood Methods in Statistics and Econometrics: Essays in Honor of George A. Barnard, vol. 7, pp. 473–488. North Holland, Amsterdam (1990)Google Scholar
  6. Li, C., Srivastava, S., Dunson, D.B.: Simple, scalable and accurate posterior interval estimation. Biometrika 104, 665–680 (2017)Google Scholar
  7. Liang, F.: On the use of stochastic approximation Monte Carlo for Monte Carlo integration. Stat. Probab. Lett. 79, 581–587 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  8. Liang, F., Liu, C., Carroll, R.J.: Stochastic approximation in Monte Carlo computation. J. Am. Stat. Assoc. 102, 305–320 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  9. Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21, 1087 (1953)CrossRefGoogle Scholar
  10. Minsker, S., Srivastava, S., Lin, L., Dunson, D.B.: Robust and scalable Bayes via a median of subset posterior measures. ArXiv e-prints (2014)Google Scholar
  11. Neiswanger, W., Wang, C., Xing, E.P.: Asymptotically exact, embarrassingly parallel MCMC, In: Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI’14, pp. 623–632, AUAI Press, Arlington. (2014)
  12. Ripley, B.D.: Stochastic Simulation. Wiley, New York (1987)CrossRefzbMATHGoogle Scholar
  13. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 22, 400–407 (1951). MathSciNetCrossRefzbMATHGoogle Scholar
  14. Roe, B.P., Yang, H.-J., Zhu, J., Liu, Y., Stancu, I., McGregor, G.: Boosted decision trees as an alternative to artificial neural networks for particle identification. Nucl. Instrum. Methods Phys. Res. Sect. A Accel. Spectrom. Detect. Assoc. Equip. 543, 577–584 (2005)CrossRefGoogle Scholar
  15. Scott, S.L., Blocker, A.W., Bonassi, F.V.: Bayes and big data: the consensus Monte Carlo algorithm. Int. J. Manag. Sci. Eng. Manag. 11, 78–88 (2016)Google Scholar
  16. Song, Q., Wu, M., Liang, F.: Weak convergence rates of population versus single-chain stochastic approximation MCMC algorithms. Adv. Appl. Probab. 46, 1059–1083 (2014). MathSciNetCrossRefzbMATHGoogle Scholar
  17. Srivastava, S., Cevher, V., Tran-Dinh, Q., Dunson, D.B.: WASP: scalable Bayes via barycenters of subset posteriors. In: AISTATS, volume 38 of JMLR: Workshop and Conference Proceedings ( (2015a)Google Scholar
  18. Srivastava, S., Li, C., Dunson, D.B.: Scalable Bayes via barycenter in Wasserstein space. ArXiv e-prints (2015b)Google Scholar
  19. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. (Ser. B) 58, 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  20. Wand, M.: KernSmooth: functions for kernel smoothing supporting Wand & Jones (1995). r package version 2.23-15 (2015)Google Scholar
  21. Wang, X., Dunson, D.: Parallelizing MCMC via Weierstrass sampler. ArXiv e-prints (2013)Google Scholar
  22. Wang, X., Guo, F., Heller, K.A., Dunson, D.B.: parallelizing MCMC with random partition trees. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (Eds.) Advances in Neural Information Processing Systems, vol. 28, pp. 451–459. Curran Associates, Inc. (2015)

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2017

Authors and Affiliations

  1. 1.Department of StatisticsTexas A&M UniversityCollege StationUSA
  2. 2.Department of StatisticsPurdue UniversityWest LafayetteUSA

Personalised recommendations