Affordable Scalability Using Multi-Cubes

Abstract

This chapter presents an analysis of the scalability of Scali systems. A Scali high-performance server consists of compute nodes, interconnected by a high-speed, low-latency SCI interconnect. The topology addressed in this paper will be direct networks based on r-ary f-cubes, or multi-dimensional tori, also called multi-cubes. The focus is on the scalability of a specific implementation of the SCI architecture, namely the PCI to SCI adapter boards from Dolphin Interconnect Solutions [5]. The adapter boards are enhanced with more than one SCI link controller (LC) in order to increase the number of supported dimensions (or fan-outs).

We show how the SCI ringlets and the internal bus in each adapter limit scalability of the interconnect, and how the two relate to each other. The internal bus (the B-Link) is used to take packets between different dimensions and between each dimension and the PCI interface towards the local node.

The resulting analysis can serve as a guide to select the right topology for a given system size.We discuss how the interconnect scales with respect to the amount of traffic each node can generate, which is limited to the bandwidth of a single PCI bus, and argue that these topologies scale to 512 nodes using state-of-the-art technology.