Parallel Numerical Algorithms: An Introduction

Abstract

The rapidly evolving field of parallel computing has seen four eras, substantially overlapping, yet progressive. The first, which we could label the Stone Age, was characterized by having speed-up as its primary objective (“Solve my problem faster…”). The tyranny of Amdahl’s Law, combined with architectural convergence to systems that added memory in fixed proportion to processors, led to the Bronze Age, characterized by pursuit of scaled speed-up (“Solve my problem bigger, in constant time…”). The cost associated with special-purpose high-performance hardware and the even greater cost of developing reliable special purpose system-level software led to the Iron Age of commodity processors — clusters of full-function computers connected by either a special purpose network or a commodity network like Ethernet or FDDI (“Solve my problem cheaper…”). Though the influence of technology and market forces on the state-of-the-art of parallel tool-building and parallel applications have not waned, we can now point to a growing companion influence on the envelope of computability due to algorithmic development and software environment development (“Solve my problem smarter…”). Perhaps, through continued progress in architecture-adaptive algorithmic design that improves upon “best fit” flop-for-flop algorithmic porting, we will soon be justified in referring to a Golden Age of parallel computing.

This work was supported by the National Aeronautics and Space Administration under NASA Contract NAS1-19480 while the author was in residence at the Institute for Computer Applications in Science and Engineering (ICASE), NASA Langley Research Center, Hampton, VA 23681-0001.