The Performance Wall of Large Parallel Computing Systems

Abstract

Since the beginning of using parallelized computing units it is known that the actual performance is less than the possible nominal performance: the unproductive part of the computing performance remains “dark”. It is also known that the amount of dark performance strongly depends on the number of parallelly working processing units, so its role must be crucial for supercomputers, where in the coming exa-scale models millions of processors are utilized, as well as for the exa-scale applications they are running, like brain simulation and Earth simulation. Although the effects affecting parallel performance are known from the beginning, their relative weights have been considerably changed with the development of the field and strongly depend on the type of application. For large computer systems the “dark performance” represents a new major obstacle, in addition the former ones, like “heat wall”, “memory wall”, “dark silicon”, etc. The careful reconsideration discovers that in contrast with the general belief, supercomputer performance has an upper limit, and reaching that limit explains some strange and mysterious events, like canceling projects immediately before their target date or that the special-purpose brain simulator cannot outperform the many-thread simulator running on a general-purpose supercomputer.