Summary
Most high-performance simulation codes are not written from scratch but begin as desktop experiments and are subsequently migrated to a scalable, parallel paradigm. This transition can be painful, however, because the restructuring required in conversion forces most authors to abandon their serial code and begin an entirely new parallel code. Starting a parallel code from scratch has many disadvantages, such as the loss of the original test suite and the introduction of new bugs. We present a disciplined, incremental approach to parallelization of existing scientific code using the PETSc framework. In addition to the parallelization, it allows the addition of more physics (in this case strong nonlinearities) without the user having to program anything beyond the new pieces of discretization code. Our approach permits users to easily develop and experiment on the desktop with the same code that scales efficiently to large clusters with excellent parallel performance. As a motivating example, we present work integrating PETSc into an existing plate tectonic subduction code.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
M. F. Adams, H. H. Bayraktar, T.M. Keaveny, and P. Papadopoulos. Ultrascalable implicit finite element analyses in solid mechanics with over a half a billion degrees of freedom. In Proceedings of SC04, 2004. Winner of Gordon Bell Special Prize at SC2004: Large scale trabecular bone finite element modeling.
V. Akcelik, J. Bielak, G. Biros, I. Epanomeritakis, A. F. O. Ghattas, E. J. Kim, D. O’Hallaron, and T. Tu. High resolution forward and inverse earthquake modeling on terascale computers. In Proceedings of SC2003, 2003. A winner of the Gordon Bell Prize for special achievement at SC2003.
P. R. Amestoy, I. S. Duff, J.-Y. L’Excellent, and J. Koster. A fully asynchronous multifrontal solver using distributed dynamic scheduling. SIAM Journal on Matrix Analysis and Applications, 23(1):15–41, 2001.
W. K. Anderson, W. D. Gropp, D. K. Kaushik, D. E. Keyes, and B. F. Smith. Achieving high sustained performance in an unstructured mesh CFD application. In Proceedings of SC 99, 1999.
S. Balay, K. Buschelman, V. Eijkhout, W. D. Gropp, D. Kaushik, M. G. Knepley, L. C. McInnes, B. F. Smith, and H. Zhang. PETSc Web page. http://www.mcs.anl.gov/petsc.
S. Balay, W. D. Gropp, L. C. McInnes, and B. F. Smith. Efficient management of parallelism in object oriented numerical software libraries. In E. Arge, A. M. Bruaset, and H. P. Langtangen, editors, Modern Software Tools in Scientific Computing, pages 163–202. Birkhäuser Press, 1997.
S. Carbotte, C. Small, and K. Donnelly. The influence of ridge migration on the magmatic segmentation of mid-ocean ridges. Nature, 429:743–746, 2004.
J. Conder, D. Wiens, and J. Morris. On the decompression melting structure at volcanic arcs and back-arc spreading centers. Geophys. Res. Letts., 29, 2002.
J. W. Demmel, J. R. Gilbert, and X. S. Li. SuperLU user’s guide. Technical Report LBNL-44289, Lawrence Berkeley National Laboratory, October 2003.
M. Eberle, O. Grasset, and C. Sotin. A numerical study of the interaction of the mantle wedge, subducting slab, and overriding plate. Phys. Earth Planet. In., 134:191–202, 2002.
P. England, R. Engdahl, and W. Thatcher. Systematic variation in the depth of slabs beneath arc volcanos. Geophys. J. Int., 156(2):377–408, 2003.
Y. Furukawa. Depth of the decoupling plate interface and thermal structure under arcs. J. Geophys. Res., 98:20005–20013, 1993.
V. E. Henson and U. M. Yang. BoomerAMG: A parallel algebraic multigrid solver and preconditioner. Technical Report UCRL-JC-133948, Lawrence Livermore National Laboratory, 2000.
G. Hirth and D. Kohlstedt. Rheology of the upper mantle and the mantle wedge: A view from the experimentalists. In Inside the Subduction Factory, volume 138 of Geophysical Monograph. American Geophysical Union, 2003.
P. Hovland, B. Norris, and B. Smith. Making automatic differentiation truly automatic: Coupling PETSc with ADIC. In Proceedings of ICCS2002, 2002.
D. Hysom and A. Pothen. A scalable parallel algorithm for incomplete factor preconditioning. SIAM Journal on Scientific Computing, 22:2194–2215, 2001.
R. Katz, M. Spiegelman, and S. Carbotte. Ridge migration, asthenospheric flow and the origin of magmatic segmentation in the global mid-ocean ridge system. Geophys. Res. Letts., 31, 2004.
P. Kelemen, J. Rilling, E. Parmentier, L. Mehl, and B. Hacker. Thermal structure due to solid-state flow in the mantle wedge beneath arcs. In Inside the Subduction Factory, volume 138 of Geophysical Monograph. American Geophysical Union, 2003.
S. Peacock and K. Wang. Seismic consequences of warm versus cool subduction metamorphism: Examples from southwest and northeast Japan. Science, 286:937–939, 1999. sau]20._PETSc SNES Example 30. http://www.mcs.anl.gov/petsc/petsc-2/snapshots/petsc-current/src/snes/examples/tutorials/ex30.c.html.
PETSc SNES Example 5. http://www.mcs.anl.gov/petsc/petsc-2/snapshots/petsc-current/src/snes/examples/tutorials/ex5f90.F.html.
PETSc Solvers. http://www.mcs.anl.gov/petsc/petsc-2/documentation/linearsolvertable.html.
B. Smith et al. Scientific Applications Using PETSc. http://www.mcs.anl.gov/petsc/petsc-2/publications.
R. Stern. Subduction zones. Rev. Geophys., 40(4), 2002.
P. van Keken, B. Kiefer, and S. Peacock. High-resolution models of subduction zones: Implications for mineral dehydration reactions and the transport of water into the deep mantle. Geochem. Geophys. Geosys., 3(10), 2003.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Knepley, M.G., Katz, R.F., Smith, B. (2006). Developing a Geodynamics Simulator with PETSc. In: Bruaset, A.M., Tveito, A. (eds) Numerical Solution of Partial Differential Equations on Parallel Computers. Lecture Notes in Computational Science and Engineering, vol 51. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-31619-1_12
Download citation
DOI: https://doi.org/10.1007/3-540-31619-1_12
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29076-6
Online ISBN: 978-3-540-31619-0
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)