Decentralized Replica Exchange Parallel Tempering: An Efficient Implementation of Parallel Tempering Using MPI and SPRNG

  • Yaohang Li
  • Michael Mascagni
  • Andrey Gorin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4707)


Parallel Tempering (PT), also known as Replica Exchange, is a powerful Markov Chain Monte Carlo sampling approach which aims at reducing the relaxation time in simulations of physical systems. In this paper, we present a novel implementation of PT, so-called decentralized replica exchange PT, using MPI and the Scalable Parallel Random Number Generators (SPRNG) libraries. By adjusting the replica exchange operations in the original PT algorithm, and taking advantage of the characteristics of pseudorandom number generators, this implementation minimizes the overhead caused by interprocessor communication in replica exchange in PT. This enables one to efficiently apply PT to large-scale massively parallel systems. The efficiency of this implementation has been demonstrated in the context of various benchmark energy functions, such as the high-dimensional Rosenbrock function, and a rugged funnel-like function.


Parallel Tempering Monte Carlo Methods Parallel Programming 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Marinari, E., Parisi, G.: Simulated Tempering: a New Monte Carlo Scheme. Europhysics Letters 19, 451–458 (1992)CrossRefGoogle Scholar
  2. 2.
    Geyer, C.J., Thompson, E.A.: Annealing Markov Chain Monte Carlo with Applications to Ancestral Inference. Journal of the American Statistical Association 90, 909–920 (1995)zbMATHCrossRefGoogle Scholar
  3. 3.
    Falcioni, M., Deem, M.W.: A Biased Monte Carlo Scheme for Zeolite Structure Solution. J. Chem. Phys 110, 1754–1766 (1999)CrossRefGoogle Scholar
  4. 4.
    Schug, A., Herges, T., Verma, A., Wenzel, W.: Investigation of the parallel tempering method for protein folding. J. Phys: Condens. Matter 17, 1641–1650 (2005)CrossRefGoogle Scholar
  5. 5.
    Sikorski, A.: Properties of Star-Branched Polymer Chains – Application of the Replica Exchange Monte Carlo Method. Macromolecules 35(18), 7132–7137 (2002)CrossRefGoogle Scholar
  6. 6.
    Schug, A., Wenzel, W.: Predictive in-silico all atom folding of a four helix protein with a free energy model. J. Am. Chem. Soc. 126, 16737 (2004)CrossRefGoogle Scholar
  7. 7.
    Li, Y., Strauss, C.E.M., Gorin, A.: Parallel Tempering in Rosetta Practice. In: Zhang, D., Jain, A.K. (eds.) ICBA 2004. LNCS, vol. 3072, Springer, Heidelberg (2004)Google Scholar
  8. 8.
    Mascagni, M., Srinivasan, A.: Algorithm 806: SPRNG: A Scalable Library for Pseudorandom Number Generation. ACM Transactions on Mathematical Software 26, 436–461 (2000)CrossRefGoogle Scholar
  9. 9.
    Bu, L., Straub, J.E.: Simulating Vibrational Energy Flow in Proteins: Relaxation Rate and Mechanism for Heme Cooling in Cytochrome c. J. Phys. Chem. B 107, 12339–12345 (2003)CrossRefGoogle Scholar
  10. 10.
    Li, Y., Protopopescu, V.A., Gorin, A.: Accelerated Simulated Tempering. Physics Letters A 328(4), 274–283 (2004)CrossRefGoogle Scholar
  11. 11.
    Liu, J.S., Liang, F., Wong, W.H.: The Use of Multiple-Try Method and Local Optimization in Metropolis Sampling. Technical Report, Department of Statistics, Stanford University (1998)Google Scholar
  12. 12.
    Srinivasan, A., Mascagni, M., Ceperley, D.: Testing Parallel Random Number Generators. Parallel Computing 29, 69–94 (2003)CrossRefGoogle Scholar
  13. 13.
    Li, Y., Clark, J., Zhang, X.: Parallel Implementation of the Accelerated Simulated Tempering Method. In: Proceedings of 3rd NPSC Conference, Atlanta (2006)Google Scholar
  14. 14.
    Li, Y., Protopopescu, V.A., Arnold, N., Zhang, X., Gorin, A.: Hybrid Parallel Tempering/Simulated Annealing Method. submitted to Physical Review E, 2006Google Scholar
  15. 15.
    Du, Z., Li, S., Li, S., Wu, M., Zhu, J.: Massively parallel simulated annealing embedded with downhill – a SPMD algorithm for cluster computing. In: Proceedings of 1st IEEE Computer Society International Workshop on Cluster Computing (1999)Google Scholar
  16. 16.
    Drugan, M.M., Thierens, D.: Evolutionary Markov Chain Monte Carlo. Technical Report UU-CS-2003-047, Utrecht university (2003)Google Scholar
  17. 17.
    Liu, J.S., Sabatti, C.: Simulated Sintering: Markov Chain Monte Carlo with Spaces of Varying Dimensions. In: Bayesian Statistics 6, pp. 389–413. Oxford University Press, Oxford (1999)Google Scholar
  18. 18.
    Cercueil, A., Francois, O.: Monte Carlo simulation and population-based optimization. In: Congress on Evolutionary Computation, pp. 191–198 (2001)Google Scholar
  19. 19.
    Laskey, K.B., Myers, J.W.: Population Markov Chain Monte Carlo. Machine Learning, 175–196 (2003)Google Scholar
  20. 20.
    Mahfoud, S.W., Goldberg, D.E.: Parallel Recombinative Simulated Annealing: a Genetic Algorithm. Parallel Computing, 1–28 (1995)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Yaohang Li
    • 1
  • Michael Mascagni
    • 2
  • Andrey Gorin
    • 3
  1. 1.Department of Computer Science, North Carolina A&T State University, Greensboro, NC 27411 
  2. 2.Department of Computer Science, Florida State University, Tallahassee, FL 32306 
  3. 3.Division of Computer Science and Mathematics, Oak Ridge National Laboratory, Oak Ridge, TN 37831 

Personalised recommendations