GPU Support for Automatic Generation of Finite-Differences Stencil Kernels

  • Vitor Hugo Mickus RodriguesEmail author
  • Lucas Cavalcante
  • Maelso Bruno Pereira
  • Fabio Luporini
  • István Reguly
  • Gerard Gorman
  • Samuel Xavier de Souza
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1087)


The growth of data to be processed in the Oil & Gas industry matches the requirements imposed by evolving algorithms based on stencil computations, such as Full Waveform Inversion and Reverse Time Migration. Graphical processing units (GPUs) are an attractive architectural target for stencil computations because of its high degree of data parallelism. However, the rapid architectural and technological progression makes it difficult for even the most proficient programmers to remain up-to-date with the technological advances at a micro-architectural level. In this work, we present an extension for an open source compiler designed to produce highly optimized finite difference kernels for use in inversion methods named Devito©. We embed it with the Oxford Parallel Domain Specific Language (OP-DSL) in order to enable automatic code generation for GPU architectures from a high-level representation. We aim to enable users coding in a symbolic representation level to effortlessly get their implementations leveraged by the processing capacities of GPU architectures. The implemented backend is evaluated on a NVIDIA® GTX Titan Z, and on a NVIDIA® Tesla V100 in terms of operational intensity through the roof-line model for varying space-order discretization levels of 3D acoustic isotropic wave propagation stencil kernels with and without symbolic optimizations. It achieves approximately 63% of V100’s peak performance and 24% of Titan Z’s peak performance for stencil kernels over grids with 2563 points. Our study reveals that improving memory usage should be the most efficient strategy for leveraging the performance of the implemented solution on the evaluated architectures.


GPU Domain Specific Languages Finite-differences Stencil kernels Parallel architectures Devito OPS 


  1. 1.
    Cárdenas, A.F., Karplus, W.J.: PDEL-a language for partial differential equations. Commun. ACM 13(3), 184–191 (1970). Scholar
  2. 2.
    Cook Jr., G.O.: ALPAL: a tool for the development of large-scale simulation codes, Lawrence Livermore National Lab., C.U. (1988)Google Scholar
  3. 3.
    van Engelen, R., Wolters, L., Cats, G.: CTADEL: a generator of multi-platform high performance codes for PDE-based scientific applications. In: Proceedings of the 10th International Conference on Supercomputing - ICS 1996, pp. 86–93. ACM Press, New York (2003).
  4. 4.
    Hawick, K., Playne, D.P.: Simulation software generation using a domain-specific language for partial differential field equations. In: Proceedings of the International Conference on Software Engineering Research and Practice (SERP). The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp), p. 7(2013). ProQuest document ID: 1491419516Google Scholar
  5. 5.
    Henretty, T., Veras, R., Franchetti, F., Pouchet, L.N., Ramanujam, J., Sadayappan, P.: A stencil compiler for short-vector SIMD architectures. In: Proceedings of the 27th International ACM Conference on International Conference on Supercomputing - ICS 2013. p. 13. ACM Press, New York (2013).
  6. 6.
    Logg, A., Olgaard, K.B., Rognes, M.E., Wells, G.N.: FFC: the FEniCS Form Compiler. In: Logg, A., Mardal, K.A., Wells, G.N. (eds.) Automated Solution of Differential Equations by the Finite Element Method, Lecture Notes in Computational Science and Engineering, Vol. 84, Chap. 11. Springer (2012).
  7. 7.
    Louboutin, M., et al.: Devito (v3.1.0): an embedded domain-specific language for finite differences and geophysical exploration. Geosci. Model Dev. 12(3), 1165–1187 (2019). Scholar
  8. 8.
    Luporini, F.,et al.: Architecture and performance of Devito, a system for automated stencil computation, July 2018Google Scholar
  9. 9.
    Membarth, R., Hannig, F., Teich, J., Kostler, H.: Towards domain-specific computing for stencil codes in HPC. In: Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012, pp. 1133–1138. IEEE, November 2012.
  10. 10.
    Mojica, O.F., Kukreja, N.: Towards automatically building starting models for full-waveform inversion using global optimization methods: a PSO approach via DEAP + Devito, May 2019Google Scholar
  11. 11.
    Rathgeber, F., et al.: Firedrake: automating the finite element method by composing abstractions, January 2015.
  12. 12.
    Reguly, I.Z., Mudalige, G.R., Giles, M.B., Curran, D., McIntosh-Smith, S.: The OPS domain specific abstraction for multi-block structured grid computations. In: Proceedings of WOLFHPC 2014: 4th International Workshop on Domain-Specific Languages and High-Level Frameworks for High Performance Computing - Held in Conjunction with SC 2014: The International Conference for High Performance Computing, Networking, Storage, pp. 58–67 (2014).
  13. 13.
    Williams, S., Waterman, A., Patterson, D.: Roofline. Commun. ACM 52(4), 65 (2009). Scholar
  14. 14.
    Witte, P.A., et al.: A large-scale framework for symbolic implementations of seismic inversion algorithms in Julia. Geophysics 84(3), F57–F71 (2019). Scholar
  15. 15.
    Yount, C., Tobin, J., Breuer, A., Duran, A.: YASK - yet another stencil kernel: a framework for HPC stencil code-generation and tuning. In: 2016 Sixth International Workshop on Domain-Specific Languages and High-Level Frameworks for High Performance Computing (WOLFHPC), pp. 30–39. IEEE, November 2017.
  16. 16.
    Zhang, Y., Mueller, F.: Auto-generation and auto-tuning of 3D stencil codes on GPU clusters. In: Proceedings of the Tenth International Symposium on Code Generation and Optimization - CHO 2012, p. 155. ACM Press, New York (2012).

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Vitor Hugo Mickus Rodrigues
    • 1
    Email author
  • Lucas Cavalcante
    • 1
  • Maelso Bruno Pereira
    • 1
  • Fabio Luporini
    • 2
  • István Reguly
    • 3
  • Gerard Gorman
    • 2
  • Samuel Xavier de Souza
    • 1
  1. 1.Universidade Federal do Rio Grande do NorteNatalBrazil
  2. 2.Imperial College LondonLondonUK
  3. 3.Pazmany Peter Catholic UniversityBudapestHungary

Personalised recommendations