Cluster Computing

, Volume 12, Issue 3, pp 309–322 | Cite as

Full-system simulation of distributed memory multicomputers

  • Fco. Javier Ridruejo
  • Jose Miguel-Alonso
  • Javier Navaridas


In this paper we discuss environments for the full-system simulation of multicomputers. These environments are composed of a large collection of modules that simulate the compute nodes and the network, plus additional linking elements that perform communication and synchronization. We present our own environment, in which we integrate Simics with INSEE. We reuse as many Simics modules as possible to reduce the effort of hardware modeling, and also to simulate standard machines running unmodified operating systems. This way we avoid the error-prone effort of developing drivers and libraries. The environment we propose in this paper enables us to show some of the difficulties we found when integrating diverse tools, and how we were able to overcome them. Furthermore we show some important details to have into account in order to do a valid full-system simulation of multicomputers, mostly related with synchronization and timing. Thus, a trade-off has to be found between simulation speed and accuracy of results.


Full-system simulation Interconnection networks Multicomputers Clusters 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Adiga, N.R., et al.: Blue Gene/L torus interconnection network. IBM J. Res. Dev. 49(2/3) (2005) Google Scholar
  2. 2.
    Binkert, N.L., Hallnor, E.G., Reinhardt, S.K.: Network-oriented full-system simulation using M5. In: Sixth Workshop on Computer Architecture Evaluation using Commercial Workloads (CAECW), Feb. 2003 Google Scholar
  3. 3.
    Binkert, N.L., Dreslinski, R.G., Hsu, L.R., Lim, K.T., Saidi, A.G., Reinhardt, S.K.: The M5 simulator: modeling networked systems. IEEE Micro 26(4), 52–60 (2006) CrossRefGoogle Scholar
  4. 4.
    Boden, N.J., Cohen, D., Felderman, R.E., Kulawik, A.E., Seitz, C.L., : Myrinet. A gigabit per second local area network. IEEE Micro 15(1), 29–36 (1995) CrossRefGoogle Scholar
  5. 5.
    García, P.J., Quiles, F.J., Flich, J., Duato, J., Jhonson, I., Naven, F.: Efficient, scalable congestion management for interconnection networks. IEEE Micro 26(5), 52–66 (2006) CrossRefGoogle Scholar
  6. 6.
    IBM: IBM full-system simulator for the cell broadband engine processor. Available at (May 2008)
  7. 7.
    Jacobson, V.: Congestion avoidance and control. Comput. Commun. Rev. 18(4), 314–329 (1988) CrossRefGoogle Scholar
  8. 8.
    Jain, R.: Congestion control in computer networks: issues and trends. IEEE Netw. 4(3), 24–30 (1990) CrossRefGoogle Scholar
  9. 9.
    Karlin, S., Peterson, L.: Maximum packet rates for full-duplex ethernet. Technical Report TR–645–02, Princeton University (February 2002) Google Scholar
  10. 10.
    LA-MPI Home Page: The Los Alamos message passing interface. Available at (May 2008)
  11. 11.
    LAM/MPI Home Page: LAM/MPI parallel computing. Available at (Apr. 2008)
  12. 12.
    Liu, J., Wu, J., Panda, D.K.: High performance RDMA-based MPI implementation over infiniBand, Int. J. Parallel Program. (2004) Google Scholar
  13. 13.
    Magnusson, P.S., Christensson, M., Eskilson, J., Forsgren, D., Hallberg, G., Hogberg, J., Larsson, F., Moestedt, A., Werner, B.: Simics: a full system simulation platform. IEEE Comput. 35(2), 50–58 (2002) Google Scholar
  14. 14.
    Martin, M.M.K., : Multifacet’s general execution-driven multiprocessor simulator (GEMS) toolset. Sigarch Comput. Archit. News 33(4), 92–99 (2005) CrossRefGoogle Scholar
  15. 15.
    Mauer, C.J., Hill, M.D., Wood, D.A.: Full-system timing-first simulation. In: ACM SIGMETRICS, June 2002 Google Scholar
  16. 16.
    Miguel-Alonso, J., Izu, C., Gregorio, J.A.: Improving the performance of large interconnection networks using congestion-control mechanisms. Perform. Eval. 65, 203–211 (2008) CrossRefGoogle Scholar
  17. 17.
    Miguel-Alonso, J., Navaridas, J., Ridruejo, F.J.: Interconnection network simulation using traces of MPI applications. Int. J. Parallel. Program. (to appear). DOI  10.1007/s10766-008-0089-y
  18. 18.
    MPI Forum: MPICH home page. Available at (May 2008)
  19. 19.
    Myricom Documentation and Software Downloads. Available at (May 2008)
  20. 20.
    NASA Advanced Supercomputing (NAS) division: NAS Parallel Benchmarks. Available at (May 2008)
  21. 21.
    Navaridas, J., Ridruejo, F.J., Miguel-Alonso, J.: Evaluation of interconnection networks using full-system simulators: lessons learned. In: Proc. 40th Annual Simulation Symposium, Norfolk, VA, March 26–28, 2007 Google Scholar
  22. 22.
    Navaridas, J., Miguel-Alonso, J., Ridruejo, F.J.: On synthesizing workloads emulating MPI applications. In: The 9th IEEE International Workshop on Parallel and Distributed Scientific and Engineering Computing (PDSEC-08). April 14–18, 2008, Miami, Florida, USA Google Scholar
  23. 23.
    OPNET Technologies, Inc. corporate web page, available at (May 2008)
  24. 24.
    Pai, V.S., Ranganathan, P., Adve, S.V.: RSIM: an execution-driven simulator for ILP-based shared-memory multiprocessors and uniprocessors. In: IEEE TCCA New., Oct. 1997 Google Scholar
  25. 25.
    Pfister, G.F.: Aspects of the InfiniBand(tm) architecture. In: Third IEEE International Conference on Cluster Computing (CLUSTER’01), October 2001, pp. 369 Google Scholar
  26. 26.
    Puente, V., Izu, C., Gregorio, J.A., Beivide, R., Vallejo, F.: The adaptive bubble router. J. Parallel Distrib. Comput. 61(9), 1180–1208 (2001) MATHCrossRefGoogle Scholar
  27. 27.
    Puente, V., Gregorio, J.A., Beivide, R.: SICOSYS: an integrated framework for studying interconnection network in multiprocessor systems. In: Proceedings of the IEEE 10th Euromicro Workshop on Parallel and Distributed Processing, Gran Canaria, Spain (2002) Google Scholar
  28. 28.
    Puente, V., Gregorio, J.A., Vallejo, F., Beivide, R.: Immunet: a cheap and robust fault-tolerant packet routing mechanism. In: International Symposium on Computer Architecture (ISCA), June 2004, pp. 198–211 Google Scholar
  29. 29.
    Ridruejo, F.J., Miguel-Alonso, J.: INSEE: an interconnection network simulation and evaluation environment. In: Proc. Euro-Par 2005. Lecture Notes in Computer Science, vol. 3648, pp. 1014–1023 Google Scholar
  30. 30.
    Ridruejo, F.J., Gonzalez, A., Miguel-Alonso, J.: TrGen: a traffic generation system for interconnection network simulators. In: International Conference on Parallel Processing, 2005. 1st. Int. Workshop on Performance Evaluation of Networks for Parallel, Cluster and Grid Computing Systems (PEN-PCGCS’05). ICPP 2005 Workshops, 14–17 June 2005, pp. 547–553 Google Scholar
  31. 31.
    Rosenblum, M., : Complete computer system simulation: the SimOS approach. IEEE Parallel Distrib. Tech. 3(4), 34–43 (1995) CrossRefGoogle Scholar
  32. 32.
    Schaelicke, L., Parker, M.: ML-RSIM reference manual. Tech. Report 02-10, Department of Computer Science and Engineering, Univ. of Notre Dame, Notre Dame, ID (2002) Google Scholar
  33. 33.
    SMART group at the U. of Southern California. FlexSim 1.2. Available at (May 2008)
  34. 34.
    The Chaotic Routing Project at the U. of Washington. Chaos Router Simulator. Available at (May 2008)
  35. 35.
    The Network Simulator ns-2. Available at (May 2008)
  36. 36.
    Thottethodi, M., Lebeck, A.R., Mukherjee, S.S.: Exploiting global knowledge to achieve self-tuned congestion control for k-ary n-cube networks. IEEE Trans. Parallel Distrib. Syst. 15(3), 257–272 (2004) CrossRefGoogle Scholar
  37. 37.
    Wenisch, T.F., Wunderlich, R.E., Ferdman, M., Ailamaki, A., Falsafi, B., Hoe, J.C.: SimFlex: statistical sampling of computer system simulation. IEEE Micro 26(4), 18–31 (2006) CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Fco. Javier Ridruejo
    • 1
  • Jose Miguel-Alonso
    • 1
  • Javier Navaridas
    • 1
  1. 1.Dep. of Computer Architecture and TechnologyThe University of the Basque CountryDonostia-San SebastianSpain

Personalised recommendations