Enterprise HPC on the Clouds

Abstract

In the past few decades, the use of high-performance computing (HPC) has become more and more relevant in the enterprise. From aeronautics to the car industry, and from large computer manufacturers to Internet start-ups, everybody has the need to process enormous amounts of data in order to reduce costs and cope with the speed that technology is evolving today. Companies know that the need for an HPC solution is paramount to their success and the viability of their business in the future. While large enterprises have the required funds for an in-house HPC system, many smaller companies do not have the budget to deploy such solutions, although their needs for data processing may be equally high. Through commoditization of hardware, the need for supercomputers in HPC has evaporated; clusters of servers can nowadays provide the same functionality and performance, at a much lower cost. The latter has led to the advent of “cloud computing” which constitutes a major paradigm shift in how we, as users, can have access to large-scale computing infrastructure. “Clouds” offer virtually limitless resources, on-demand, at a relatively low cost. In the future, this can lead to a complete outsourcing of enterprise HPC and demolish the need for in-house solutions. In this chapter, we are going to discuss the major issues that must be addressed in order to make clouds viable for enterprise HPC, and review research, based on existing or simulated cloud systems, that hints as to how the problems can be solved.