Performance Evaluation of HPC Benchmarks on VMware’s ESXi Server

  • Qasim Ali
  • Vladimir Kiriansky
  • Josh Simons
  • Puneet Zaroo
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7155)


A major obstacle to virtualizing HPC workloads is a concern about the performance loss due to virtualization. We will demonstrate that new features significantly enhance the performance and scalability of virtualized HPC workloads on VMware’s virtualization platform. Specifically, we will discuss VMware’s ESXi Server performance for virtual machines with up to 64 virtual CPUs as well as support for exposing virtual NUMA topology to guest operating systems, enabling the operating system and applications to make intelligent NUMA aware decisions about memory allocation and process/thread placement. NUMA support is especially important for large VMs which necessarily span host NUMA nodes on all modern hardware. We will show how the virtual NUMA topology is chosen to closely match physical host topology, while preserving the now expected virtualization benefits of portability and load balancing. We show that the benefit of exposing the virtual NUMA topology can lead to performance gains of up to 167%. Overall, we will show close to native performance on applications from SPEC MPI V2.0 and SPEC OMP V3.2 benchmarks virtualized on our prototype VMware’s ESXi Server.


Non Uniform Memory Architecture (NUMA) ESXi High Performance Computing (HPC) virtual NUMA (vNUMA) virtualization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Advanced Configuration and Power Interface specification, rev 4.0a (2009),
  2. 2.
    Agesen, O., Garthwaite, A., Sheldon, J., Subrahmanyam, P.: The evolution of an x86 virtual machine monitor. Operating Systems Review 44(4) (2010)Google Scholar
  3. 3.
    Scalable Infrastructure with the CPU scheduler in VMware ESX 4.1,
  4. 4.
    Gavrilovska, A., Kumar, S., et al.: High-Performance Hypervisor Architectures: Virtualization in HPC Systems. In: HPCVirt 2007: 1st Workshop on System-level Virtualization for High Performance Computing (2007)Google Scholar
  5. 5.
    Huang, W., Gao, Q., Liu, J., Panda, D.K.: High performance virtual machine migration with RDMA over modern interconnects. In: Proceedings of the 2007 IEEE International Conference on Cluster Computing, CLUSTER 2007, pp. 11–20. IEEE Computer Society, Washington, DC (2007)CrossRefGoogle Scholar
  6. 6.
    Mergen, M.F., Uhlig, V., Krieger, O., Xenidis, J.: Virtualization for high-performance computing. SIGOPS Oper. Syst. Rev. 40, 8–11 (2006)CrossRefGoogle Scholar
  7. 7.
    Nagarajan, A.B., Mueller, F.: Proactive fault tolerance for HPC with Xen Virtualization. In: Proceedings of the 21st Annual International Conference on Supercomputing (ICS 2007), pp. 23–32. ACM Press (2007)Google Scholar
  8. 8.
    Open MPI: Open Source High Performance Computing (2011),
  9. 9.
    Ranadive, A., Kesavan, M., Gavrilovska, A., Schwan, K.: Performance implications of virtualizing multicore cluster machines. In: Proceedings of the 2nd Workshop on System-Level Virtualization for High Performance Computing, HPCVirt 2008, pp. 1–8. ACM, New York (2008)CrossRefGoogle Scholar
  10. 10.
    Rao, D., Schwan, K.: vNUMA-mgr: Managing VM memory on NUMA platforms. In: 2010 International Conference on High Performance Computing (HiPC), pp. 1–10 (December 2010)Google Scholar
  11. 11.
    Simons, J.E., Buell, J.: Virtualizing high performance computing. SIGOPS Oper. Syst. Rev. 44, 136–145 (2010)CrossRefGoogle Scholar
  12. 12.
    Valle, G., Engelmann, C., Scott, S.L., Naughton, T., Ong, H.: System-Level Virtualization for High Performance Computing, February 13-15 (2008)Google Scholar
  13. 13.
    Waldspurger, C.A.: Memory resource management in VMware ESX server. In: OSDI 2002: Proceedings of the 5th Symposium on Operating Systems Design and Implementation, pp. 181–194. ACM Press, New York (2002)CrossRefGoogle Scholar
  14. 14.
    Youseff, L., Seymour, K., You, H., Dongarra, J., Wolski, R.: The impact of paravirtualized memory hierarchy on linear algebra computational kernels and software. In: Proceedings of the 17th International Symposium on High Performance Distributed Computing, HPDC 2008, pp. 141–152. ACM, New York (2008)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Qasim Ali
    • 1
  • Vladimir Kiriansky
    • 1
  • Josh Simons
    • 1
  • Puneet Zaroo
    • 1
  1. 1.VMwareUSA

Personalised recommendations