NIX

Abstract

The NIX project targeted manycore architectures in which only a small subset of the cores could run kernel code. Consider N cores on a die arranged in a grid, with an on-die network providing a totally symmetric network and access to common memory. Let us assume that only $\sqrt{N}$ of the cores can run a kernel and the rest can only run user mode code. NIX was intended to provide a kernel for such a system. The hypothetical target was a 1,024 core system, as several vendors hinted that was reasonable in the 2016 time frame. We picked this number as it was large enough to far exceed any possible number of user processes: the Plan 9 systems on which NIX was based use only a small fraction of this number. Further, the realities of memory bandwidth will limit the number of active cores to small single digits. We also wanted to have a system with good support for Go, the then-new language from Google. At the time, Go was a statically linked language and a typical binary was about 1 MiB. This influenced the design of the virtual memory system in NIX. The base page size was expanded from 4,096 bytes to 2 MiB. We further extended Plan 9’s virtual memory model to transparently support GiB pages, but via a simple heuristic that made it far less complex than existing systems. The target system had a 32:1 ratio of application cores to kernel cores (a close match to the Blue Gene/Q system at 16:1), and it was designed for space sharing, not time sharing, and had a minimum page size of 2 MiB; supported GiB pages; and yet, at the same time, looked in all other ways like a standard Plan 9 system.