An Example of Porting PETSc Applications to Heterogeneous Platforms with OpenACC

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10732)


In this paper, we document the workflow of our practice to port a PETSc application with OpenACC to a supercomputer, Titan, at Oak Ridge National Laboratory. Our experience shows a few lines of code modifications with OpenACC directives can give us a speedup of 1.34x in a PETSc-based Poisson solver (conjugate gradient method with algebraic multigrid preconditioner). This demonstrates the feasibility of enabling GPU capability in PETSc with OpenACC. We hope our work can serve as a reference to those who are interested in porting their legacy PETSc applications to modern heterogeneous platforms.


OpenACC PETSc GPU computing 



This research used resources of the Oak Ridge Leadership Computing Facility at Oak Ridge National Laboratory, which is supported by the Office of Science of the Department of Energy under Contract DE-AC05-00OR22725.


  1. 1.
    Balay, S., Abhyankar, S., Adams, M.F., Brown, J., Brune, P., Buschelman, K., Dalcin, L., Eijkhout, V., Gropp, W.D., Kaushik, D., Knepley, M.G., McInnes, L.C., Rupp, K., Smith, B.F., Zampini, S., Zhang, H., Zhang, H.: PETSc Web page (2016).
  2. 2.
    PETSc, Applications and Publications. Accessed 31 Aug 2017
  3. 3.
    Minden, V., Smith, B.F., Knepley, M.G.: Preliminary implementation of PETSc using GPUs. In: Proceedings of the 2010 International Workshop of GPU Solutions to Multiscale Problems in Science and Engineering (2010)Google Scholar
  4. 4.
    PETSc. Accessed 31 Aug 2017
  5. 5.
    OLCF, Titan Cray XK7. Accessed 31 Aug 2017
  6. 6.
    Chuang, P.-Y., Barba, L.A.: Using AmgX to Accelerate PETSc-Based CFD Codes, figshare (2017).
  7. 7.
    Chuang, P.-Y., Barba, L.A.: AmgXWrapper: An interface between PETSc and the NVIDIA AmgX library. J. Open Source Softw. 2(16), 280 (2017)CrossRefGoogle Scholar
  8. 8.
    Larkin, J.: Chapter 2: Profile-guided development with OpenACC. In: Farber, R. (ed.) Parallel Programing with OpenACC. Elsevier (2017)Google Scholar
  9. 9.
    Kraus, J.: Multi-GPU Programming with MPI. In: GTC 2017, San Jose (2017)Google Scholar
  10. 10.
    OLCF, PETSc-OpenACC. Accessed 31 Aug 2017
  11. 11.
    Knüpfer, A., et al.: Score-P: a joint performance measurement run-time infrastructure for periscope, scalasca, TAU, and vampir. In: Brunst, H., Müller, M., Nagel, W., Resch, M. (eds.) Tools for High Performance Computing 2011. Springer, Heidelberg (2012). Google Scholar
  12. 12.
    NVIDIA, Profiler User’s Guide. Accessed 24 Aug 2017
  13. 13.
    Mesnard, O., Barba, L.A.: Reproducible and replicable computational fluid dynamics: it’s harder than you think. Comput. Sci. Eng. 19(4), 44–55 (2017)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.The George Washington UniversityWashington DCUSA
  2. 2.Oak Ridge National LaboratoryOak RidgeUSA

Personalised recommendations