Abstract
During the software crisis of the 1960s, Dijkstra’s famous thesis “goto considered harmful’’ paved the way for structured programming of sequential computers. This short communication suggests that many current difficulties and challenges of parallel programming based on message passing are caused by poorly structured, pair-wise communication, which is a consequence of using low-level send-receive primitives. We argue that, like goto in sequential programs, send-receive should be avoided as far as possible. A viable alternative in the setting of message passing are collective operations, already present in MPI (Message Passing Interface). We dispute some widely held opinions about the apparent superiority of unstructured pair-wise communication over well-structured collective operations, and we present substantial theoretical and empirical evidence to the contrary in the context of the MPI framework.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bernashi, M., Iannello, G., & Lauria, M. (1999). Experimental results about MPI collective communication operations. High-Performance Computing and Networking. Lecture Notes in Computer Science, 775–783.
Bilardi, G., Herley, K. T., Pietracaprina, A., Pucci, G., & Spirakis, P. (1996). BSP vs LogP. Proceedings of the Eighth Annual ACM Symposium on Parallel Algorithms and Architectures (pp. 25–32).
Böhm, C., & Jacopini, G. (1966). Flow diagrams, Turing machines and languages with only two formation rules. Communications of the ACM, 9, 366–371.
Dahl, O., Dijkstra, E. W., & Hoare, C. (1975). Structured Programming: Academic Press.
Dijkstra, E. W. (1968). Go to statement considered harmful. Communications of the ACM, 11(3), 147–148.
Fischer, J., & Gorlatch, S. (2002). Turing universality of morphisms for parallel programming. Parallel Processing Letters, 12(2), 229–246.
Gorlatch, S. (2000). Towards formally-based design of message passing programs. TSE, 26(3), 276–288.
Gorlatch, S. (2001). Send-Recv considered harmful? Myths and truths about parallel programming. In Parallel Computing Technologies (pp. 243–258).
Gorlatch, S., & Cole, M. (2011). Parallel skeletons, encyclopedia of parallel computing. Boston: Springer.
Gorlatch, S., Wedler, C., & Lengauer, C. (1999). Optimization rules for programming with collective operations. In Proceedings of IPPS’99 (pp. 492–499).
Goudreau, M., Lang, K., Rao, S., Suel, T., & Tsantilas, T. (n.d.). Towards efficiency and portablility: Programming with the BSP model (pp. 1–12).
Goudreau, M., & Rao, S. (1999). Single-message vs.\batch communication. In Algorithms for parallel processing (pp. 61–74).
Gropp, W., Lusk, E., & Skjellum, A. (1994). Using MPI: MIT Press. Scientific and Engineering Computation Series.
Hagedorn, B., Steuwer, M., & Gorlatch, S. (2018). A transformation-based approach to developing high-performance GPU programs. In 11th International A.Ershov Memorial Conference on Perspectives of System Informatics, PSI 2017. Springer.
Hagedorn, B., Stoltzfus, L., Steuwer, M., Gorlatch, S., & Dubach, C. (2018). High performance stencil code generation with Lift. In Proceedings of the 2018 International Symposium on Code Generation and Optimization-CGO 2018.
Hwang, K., & Xu, Z. (1998). Scalable parallel computing. McGraw Hill.
Kielmann, T., Bal, H., & Gorlatch, S. (2000). Bandwidth-efficient collective communication for clustered wide area systems. In Parallel and Distributed Processing Symposium (pp. 492–499).
Kielmann, T., Hofman, R. F. H., Bal, H. E., Plaat, A., & Bhoedjang, R. a F. (1999). MagPIe: MPI’s collective communication operations for clustered wide area systems. In PPOPP’99 (pp. 131–140).
Kumar, V. (1994). Introduction to parallel computing. Benjamin Cummings.
Pacheco, P. S. (1997). Parallel programming with MPI. Morgan Kaufmann.
Park, J., Choi, H., Nupairoj, N., & Ni, L. (1996). Construction of optimal multicast trees based on the parameterized communication model (pp. 180–187).
Schneider, F. (1997). On concurrent programming. Springer.
Steuwer, M., & Gorlatch, S. (2014). SkelCL: a high-level extension of OpenCL for multi-GPU systems. The Journal of Supercomputing, 69(1), 25–33.
Vadhiyar, S. S., Fagg, G. E., & Dongarra, J. (2000). Automatically tuned collective communications. In 2000 ACM/IEEE Conference on Supercomputing.
Valiant, L. (1990). General purpose parallel architectures. Handbook of Theoretical Computer Science, 943–971.
van de Geijn, R. (1997). Using PLAPACK: Parallel linear algebra package. MIT Press (Scientific and Engineering Computation Series).
Acknowledgements
I am grateful to many colleagues in the field of parallel computing, whose research provided necessary theoretical and experimental evidence to support the ideas presented here. It is my pleasure to acknowledge the very helpful comments of Chris Lengauer, Robert van de Geijn, Murray Cole, Jan Prins, Thilo Kielmann, Holger Bischof, and Phil Bacon on the preliminary version of the manuscript.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Gorlatch, S. (2019). Send-Receive Considered Harmful: Toward Structured Parallel Programming. In: Bergener, K., Räckers, M., Stein, A. (eds) The Art of Structuring. Springer, Cham. https://doi.org/10.1007/978-3-030-06234-7_48
Download citation
DOI: https://doi.org/10.1007/978-3-030-06234-7_48
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-06233-0
Online ISBN: 978-3-030-06234-7
eBook Packages: Business and ManagementBusiness and Management (R0)