Highly Scalable Dynamic Load Balancing in the Atmospheric Modeling System COSMO-SPECS+FD4

  • Matthias Lieber
  • Verena Grützun
  • Ralf Wolke
  • Matthias S. Müller
  • Wolfgang E. Nagel
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7133)


To study the complex interactions between cloud processes and the atmosphere, several atmospheric models have been coupled with detailed spectral cloud microphysics schemes. These schemes are computationally expensive, which limits their practical application. Additionally, our performance analysis of the model system COSMO-SPECS (atmospheric model of the Consortium for Small-scale Modeling coupled with SPECtral bin cloud microphysicS) shows a significant load imbalance due to the cloud model. To overcome this issue and enable dynamic load balancing, we propose the separation of the cloud scheme from the static partitioning of the atmospheric model. Using the framework FD4 (Four-Dimensional Distributed Dynamic Data structures), we show that this approach successfully eliminates the load imbalance and improves the scalability of the model system. We present a scalability analysis of the dynamic load balancing and coupling for two different supercomputers. The observed overhead is 6% on 1600 cores of an SGI Altix 4700 and less than 7% on a BlueGene/P system at 64Ki cores.


atmospheric modeling spectral bin cloud microphysics scalability dynamic load balancing model coupling 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Builtjes, P., Fowler, D., Feichter, J., Lewis, A., Monks, P., Borrell, P. (eds.): The Impact of Climate Change on Air Quality. The 4th ACCENT Barnsdale Expert Workshop (2008)Google Scholar
  2. 2.
    Burstedde, C., Burtscher, M., Ghattas, O., Stadler, G., Tu, T., Wilcox, L.C.: ALPS: A framework for parallel adaptive PDE solution. J. Phys. Conf. Ser. 180(1), 012009 (2009)CrossRefGoogle Scholar
  3. 3.
    Chin, M., Kahn, R.A., Schwartz, S.E. (eds.): Atmospheric Aerosol Properties and Climate Impacts. U.S. Climate Change Science Program and the Subcommittee on Global Change Research (2009)Google Scholar
  4. 4.
    Devine, K., Boman, E., Heaphy, R., Hendrickson, B., Vaughan, C.: Zoltan Data Management Services for Parallel Dynamic Applications. Comput. Sci. Eng. 4(2), 90–97 (2002)CrossRefGoogle Scholar
  5. 5.
    Fahey, K.M., Pandis, S.N.: Size-resolved aqueous-phase atmospheric chemistry in a three-dimensional chemical transport model. J. Geophys. Res. 108(D22), 4690 (2003)CrossRefGoogle Scholar
  6. 6.
    Grützun, V., Knoth, O., Simmel, M.: Simulation of the influence of aerosol particle characteristics on clouds and precipitation with LM–SPECS: Model description and first results. Atmos. Res. 90, 233–242 (2008)CrossRefGoogle Scholar
  7. 7.
    Jacobson, M.Z., Ginnebaugh, D.L.: Global-through-urban nested three-dimensional simulation of air pollution with a 13,600-reaction photochemical mechanism. J. Geophys. Res. 115, D14304 (2010)CrossRefGoogle Scholar
  8. 8.
    Larson, J., Jacob, R., Ong, E.: The Model Coupling Toolkit: A New Fortran90 Toolkit for Building Multiphysics Parallel Coupled Models. Int. J. High Perf. Comput. Appl. 19, 277–292 (2005)CrossRefGoogle Scholar
  9. 9.
    Lieber, M., Grützun, V., Wolke, R., Müller, M.S., Nagel, W.E.: FD4: A Framework for Highly Scalable Load Balancing and Coupling of Multiphase Models. In: AIP Conf. Proc., vol. 1281(1), pp. 1639–1642 (2010)Google Scholar
  10. 10.
    Lieber, M., Wolke, R.: Optimizing the coupling in parallel air quality model systems. Environ. Modell. Softw. 23(2), 235–243 (2008)CrossRefGoogle Scholar
  11. 11.
    Lieber, M., Wolke, R., Grützun, V., Müller, M.S., Nagel, W.E.: A framework for detailed multiphase cloud modeling on HPC systems. In: Parallel Computing, vol. 19, pp. 281–288. IOS Press (2010)Google Scholar
  12. 12.
    Lynn, B., Khain, A., Rosenfeld, D., Woodley, W.L.: Effects of aerosols on precipitation from orographic clouds. J. Geophys. Res. 112, D10225 (2007)CrossRefGoogle Scholar
  13. 13.
    Lynn, B.H., Khain, A.P., Dudhia, J., Rosenfeld, D., Pokrovsky, A., Seifert, A.: Spectral (Bin) Microphysics Coupled with a Mesoscale Model (MM5). Part I: Model Description and First Results. Mon. Weather Rev. 133, 44–58 (2005)CrossRefGoogle Scholar
  14. 14.
    Message Passing Interface Forum: MPI-2: Extensions to the Message-Passing Interface (1997),
  15. 15.
    Michalakes, J.: MM90: A scalable parallel implementation of the Penn State/NCAR Mesoscale Model (MM5). Parallel Computing 23(14), 2173–2186 (1997)CrossRefzbMATHGoogle Scholar
  16. 16.
    Milbrandt, J.A., Yau, M.K.: A Multimoment Bulk Microphysics Parameterization. Part I: Analysis of the Role of the Spectral Shape Parameter. J. Atmos. Sci. 62(9), 3051–3064 (2005)CrossRefGoogle Scholar
  17. 17.
    Pinar, A., Aykanat, C.: Fast optimal load balancing algorithms for 1D partitioning. J. Parallel Distrib. Comput. 64(8), 974–996 (2004)CrossRefzbMATHGoogle Scholar
  18. 18.
    Planche, C., Wobrock, W., Flossmann, A.I., Tridon, F., Van Baelen, J., Pointin, Y., Hagen, M.: The influence of aerosol particle number and hygroscopicity on the evolution of convective cloud systems and their precipitation: A numerical study based on the COPS observations on 12 August 2007. Atmos. Res. 98(1), 40–56 (2010)CrossRefGoogle Scholar
  19. 19.
    Redler, R., Valcke, S., Ritzdorf, H.: OASIS4 - a coupling software for next generation earth system modelling. Geosci. Model Dev. 3(1), 87–104 (2010)CrossRefGoogle Scholar
  20. 20.
    Sagan, H.: Space-filling curves. Springer, Heidelberg (1994)CrossRefzbMATHGoogle Scholar
  21. 21.
    Seifert, A., Beheng, K.D.: A two-moment cloud microphysics parameterization for mixed-phase clouds. Part 1: Model description. Meteorol. Atmos. Phys. 92, 45–66 (2006)CrossRefGoogle Scholar
  22. 22.
    Seifert, A., Khain, A., Pokrovsky, A., Beheng, K.D.: A comparison of spectral bin and two-moment bulk mixed-phase cloud microphysics. Atmos. Res. 80, 46–66 (2006)CrossRefGoogle Scholar
  23. 23.
    Simmel, M., Wurzler, S.: Condensation and activation in sectional cloud microphysical models. Atmos. Res. 80, 218–236 (2006)CrossRefGoogle Scholar
  24. 24.
    Solomon, S., et al. (eds.): Climate Change 2007: The Physical Science Basis. Contribution of Working Group I to the Forth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC). Cambridge University Press (2007)Google Scholar
  25. 25.
    Teresco, J.D., Devine, K.D., Flaherty, J.E.: Partitioning and Dynamic Load Balancing for the Numerical Solution of Partial Differential Equations. In: Numerical Solution of Partial Differential Equations on Parallel Computers, pp. 55–88. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  26. 26.
    Tremback, C.J., Walko, R.L.: The Regional Atmospheric Modeling System (RAMS): Development for parallel processing computer architectures. In: 3rd RAMS Users Workshop (1997)Google Scholar
  27. 27.
    Van Straalen, B., Shalf, J., Ligocki, T., Keen, N., Yang, W.S.: Scalability challenges for massively parallel AMR applications. In: IPDPS 2009, pp. 1–12. IEEE Computer Society (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Matthias Lieber
    • 1
  • Verena Grützun
    • 2
  • Ralf Wolke
    • 3
  • Matthias S. Müller
    • 1
  • Wolfgang E. Nagel
    • 1
  1. 1.Center for Information Services and High Performance ComputingTU DresdenDresdenGermany
  2. 2.Max Planck Institute for MeteorologyHamburgGermany
  3. 3.Leibniz Institute for Tropospheric ResearchLeipzigGermany

Personalised recommendations