Energy-Aware Checkpointing Strategies

  • Guillaume Aupy
  • Anne BenoitEmail author
  • Mohammed El Mehdi Diouri
  • Olivier Glück
  • Laurent Lefèvre
Part of the Computer Communications and Networks book series (CCN)


Future extreme-scale supercomputers will gather several millions of cores. The main problem that we address in this chapter is the energy consumption of these systems. Fault-tolerant methods must be deployed in such extreme-scale systems and these methods have a dramatic impact on total energy consumption. Fault-tolerant protocols have different energy consumption rates, depending on parameters such as platform characteristics, application features, and number of processes used in the execution. Currently, in order to evaluate the power consumption of fault-tolerant protocols in a given execution context, the only approach is to run the application with the different versions of fault-tolerant protocols and to monitor energy consumption. In order to avoid this time and energy consuming process, we describe in this chapter a methodology to estimate the energy consumption of the fault-tolerant protocols used for HPC applications. This methodology relies on an energy calibration of the supercomputer and a user description of the execution setting. We evaluate the accuracy of the estimations with applications and scenarios executed on a real platform with energy consumption monitoring. Results show that the energy estimations provided before the execution are highly accurate, and allow users to select the less energy consuming fault-tolerant protocol without pre-running their applications.


Energy Consumption Power Consumption Execution Time Energy Estimation Mean Time Between Failure 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was supported in part by the ANR RESCUE project and the Joint Laboratory for Petascale Computing between Inria and the University of Illinois at Urbana Champaign. Some experiments presented in this chapter were carried out using the Grid’5000 experimental testbed, being developed under the INRIA ALADDIN development action with support from CNRS, RENATER, and several Universities as well as other funding bodies (see


  1. 1.
    Alonso P, Dolz MF, Mayo R, Quintana-Ortí ES (2012) Energy-efficient execution of dense linear algebra algorithms on multi-core processors. Cluster Comput, May 2012Google Scholar
  2. 2.
    Alonso P, Dolz MF, Igual FD, Mayo R, Quintana-Ortí ES (2012) DVFS-control techniques for dense linear algebra operations on multi-core processors. Comput Sci—R&D 27(4):289–298Google Scholar
  3. 3.
    Aupy G, Benoit A, Melhem R, Renaud-Goud P, Robert Y (2013) Energy-aware checkpointing of divisible tasks with soft or hard deadlines. In: International green computing conference (IGCC), IEEE, p 1–8Google Scholar
  4. 4.
    Bosilca G, Bouteiller A, Brunet E, Cappello F, Dongarra J, Guermouche A, Herault T, Robert Y, Vivien F, Zaidouni D (2013) Unified model for assessing checkpointing protocols at extreme-scale. Concurr Comput: Pract Exp, Oct 2013, to be published. Also available as INRIA research report 7950 at
  5. 5.
    Bouteiller A, Bosilca G, Dongarra J (2010) Redesigning the message logging model for high performance. Concurr Comput: Pract Exp 22(16):2196–2211CrossRefGoogle Scholar
  6. 6.
    Cappello F, Caron E, Daydé MJ, Desprez F, Jégou Y, Primet PV-B, Jeannot E, Lanteri S, Leduc J, Melab N, Mornet G, Namyst R, Quétier B, Richard O (2005) Grid’5000: a large scale, Reconfigurable, controlable and monitorable grid platform. In IEEE/ACM Grid 2005, Seattle, Washington, Nov 2005Google Scholar
  7. 7.
    Cappello F, Casanova H, Robert Y (2011) Preventive migration vs. preventive checkpointing for extreme scale supercomputers. Parallel Process Lett 21(2):111–132MathSciNetCrossRefGoogle Scholar
  8. 8.
    Cappello F, Geist A, Gropp B, Kale S, Kramer B, Snir M (2009) Toward exascale resilience. Int J High Perform Comput Appl 23:374–388Google Scholar
  9. 9.
    Chandy KM, Lamport L (1985) Distributed snapshots: determining global states of distributed systems. Trans Comput Syst 3(1)63–75. ACM, Feb 1985Google Scholar
  10. 10.
    Daly JT (2004) A higher order estimate of the optimum checkpoint interval for restart dumps. FGCS 22(3):303–312CrossRefGoogle Scholar
  11. 11.
    Daly JT (2006) A higher order estimate of the optimum checkpoint interval for restart dumps. Future Generat Comp Syst 22(3):303–312CrossRefGoogle Scholar
  12. 12.
    Dias de Assuncao M, Gelas J-P, Lefèvre L, Orgerie A-C (2010) The green grid5000: Instrumenting a grid with energy sensors. In: 5th international workshop on distributed cooperative laboratories: instrumenting the grid (IN-GRID 2010). Poznan, Poland, May 2010Google Scholar
  13. 13.
    Dias de Assuncao M, Orgerie A-C, Lefèvre L (2010) An analysis of power consumption logs from a monitored grid site. In: IEEE/ACM international conference on green computing and communications (GreenCom-2010). Hangzhou, China, p 61–68, Dec 2010Google Scholar
  14. 14.
    Diouri MEM, Dolz MF, Glück O, Lefèvre L, Alonso P, Catalán S, Mayo R, Quintana-Ortí ES (2013) Solving some mysteries in power monitoring of servers: take care of your wattmeters! In: Energy efficiency in large scale distributed systems (EE-LSDS). Vienna, Austria, April 2013, 22–24Google Scholar
  15. 15.
    Diouri MEM, Glück O, Lefèvre L (2013) Your cluster is not power homogeneous: take care when designing green schedulers. In: 4th IEEE international green computing conference (IGCC). Arlington, Virginia, June 2013Google Scholar
  16. 16.
    Diouri MEM, Gluck O, Lefèvre L, Cappello F (2012) Energy considerations in checkpointing and fault tolerance protocols. In: IEEE DSNW, pp 1–6Google Scholar
  17. 17.
    Diouri MEM, Glück O, Lefèvre L, Cappello F (2013) ECOFIT: a framework to estimate energy consumption of fault tolerance protocols during HPC executions. In: 13th IEEE/ACM international symposium on cluster, cloud and grid computing (CCGrid). Delft, Netherlands, May 2013Google Scholar
  18. 18.
    Diouri MEM, Gluck O, Lefevre L Cappello F (2013) Ecofit: a framework to estimate energy consumption of fault tolerance protocols for HPC applications. In: IEEE CCGRID, pp 522–529Google Scholar
  19. 19.
    Diouri MEM, Tsafack Chetsa GL, Glück O, Lefèvre L, Pierson J-M, Stolf P, Da Costa G (2013) Energy efficiency in high-performance computing with and without knowledge of applications and services. Intern J High Perform Comput Appl (IJHPCA) (to appear)Google Scholar
  20. 20.
    Dongarra J et al (2011) The international exascale software project roadmap. Int J High Perform Comput Appl 25(1)Google Scholar
  21. 21.
    Dongarra J, Beckman P, Aerts P, Cappello F, Lippert T, Matsuoka S, Messina P, Moore T, Stevens R, Trefethen A, Valero M (2009) The international exascale software project: a call to cooperative action by the global high-performance community. Int J High Perform Comput Appl 23(4):309–322Google Scholar
  22. 22.
    Dongarra J, Herault T, Robert Y (2013) Revisiting the double checkpointing algorithm. In: 15th workshop on advances in parallel and distributed computational models APDCM. IEEE Computer Society PressGoogle Scholar
  23. 23.
    Elnozahy ENM, Alvisi L, Wang Y-M, Johnson DB (2002) A survey of rollback-recovery protocols in message-passing systems. ACM Comput Surv 34(3):375–408CrossRefzbMATHGoogle Scholar
  24. 24.
    Etinski M, Corbalan J, Labarta J, Valero M (2010) Utilization driven power-aware parallel job scheduling. Comput Sci—Res Dev 25(3–4):207–216CrossRefGoogle Scholar
  25. 25.
    Feng W-C, Feng X, Ge R (2008) Green supercomputing comes of age. IT Prof 10(1):17–23CrossRefGoogle Scholar
  26. 26.
    Ferreira K, Stearley J, Laros JHI, Oldfield R, Pedretti K, Brightwell R, Riesen R, Bridges PG, Arnold D (2011) Evaluating the viability of process replication reliability for exascale systems. In: Proceedings of the ACM/IEEE conference on supercomputingGoogle Scholar
  27. 27.
    Freeh VW, Lowenthal DK, Pan F, Kappiah N, Springer R, Rountree B, Femal ME (2007) Analyzing the energy-time trade-off in high-performance computing applications. IEEE Trans Parallel Distrib Syst 18(6):835–848CrossRefGoogle Scholar
  28. 28.
    Hermenier F, Loriant N, Menaud J-M (2006) Power management in grid computing with xen. In: frontiers of high performance computing and networking—ISPA 2006 international workshops, FHPCN, XHPC, S-GRACE, GridGIS, HPC-GTP, PDCE, ParDMCom, WOMP, ISDF and UPW. Lecture Notes in Computer Science, vol 4331, pp 407–416, Sorrento, Italy, 4–7 Dec 2006Google Scholar
  29. 29.
    Hlavacs H, da Costa G, Pierson J-M (2009) Energy consumption of residential and professional switches. In IEEE CSEGoogle Scholar
  30. 30.
    Hotta Y, Sato M, Kimura H, Matsuoka S, Boku T, Takahashi D (2006) Profile-based optimization of power performance by using dynamic voltage scaling on a PC cluster. In: Proceedings of the 20th international in parallel and distributed processing symposium (IPDPS)Google Scholar
  31. 31.
    Hsing Hsu C, chun Feng W, Archuleta JS (2005) Towards efficient supercomputing: a quest for the right metric. In: Proceedings of the high performance power-aware computing workshopGoogle Scholar
  32. 32.
    Hsu C-Hchun Feng W (2005) A power-aware run-time system for high-performance computing. In Supercomputing, 2005. Proceedings of the ACM/IEEE SC 2005 Conference, 2005Google Scholar
  33. 33.
    Laros III JH, Pedretti KT, Kelly SM, Shu W, Vaughan CT (2012) Energy based performance tuning for large scale high performance computing systems. In: Proceedings of the 2012 symposium on high performance computing, HPC’12. San Diego, CA, pp 6:1–6:10Google Scholar
  34. 34.
    Mahadevan P, Sharma P, Banerjee S, Ranganathan P (2009) A power benchmarking framework for network devices. In: Networking 2009 conference, Aachen, Germany, 11–15 May 2009, p 795–808Google Scholar
  35. 35.
    Meneses E, Sarood O, Kalé LV (2012) Assessing energy efficiency of fault tolerance protocols for HPC systems. In: Proceedings of the 2012 IEEE 24th international symposium on computer architecture and high performance computing (SBAC-PAD 2012), New York, Oct 2012Google Scholar
  36. 36.
    Netzer RHB, Xu J (1995) Necessary and sufficient conditions for consistent global snapshots. IEEE Trans Parallel Distrib Syst 6(2):165–169CrossRefGoogle Scholar
  37. 37.
    Ni X, Meneses E, Kalé LV (2012) Hiding checkpoint overhead in HPC applications with a semi-blocking algorithm. In: Proceedings of the IEEE International Conference on Cluster Computing. IEEE Computer SocietyGoogle Scholar
  38. 38.
    Orgerie A-C, Lefevre L, Gelas J-P (2008) Save watts in your grid: green strategies for energy-aware framework in large scale distributed systems. In ICPADS 2008: The 14th IEEE international conference on parallel and distributed systems, Melbourne, Australia, Dec 2008Google Scholar
  39. 39.
    Pinheiro E, Bianchini R, Carrera EV, Heath T (2001) Load balancing and unbalancing for power and performance in cluster-based systems. In: Workshop on compilers and operating systems for low powerGoogle Scholar
  40. 40.
    Rajachandrasekar R, Moody A, Mohror K, Panda DKD (2013) A 1 PB/s file system to checkpoint three million MPI tasks. In: Proceedings of the 22nd international symposium on high-performance parallel and distributed computing, HPDC ’13, ACM. New York, pp 143–154Google Scholar
  41. 41.
    Rao C, Toutenburg H, Fieger H, Heumann C, Nittner T, Scheid S (1999) Linear models: least squares and alternatives. Springer series in statisticsGoogle Scholar
  42. 42.
    Sarkar V et al (2009) Exascale software study: software challenges in extreme scale systems. White paper.
  43. 43.
    Shalf J, Dosanjh S, Morrison J (2011) Exascale computing technology challenges. In: VECPAR’10, the 9th International Conference on high performance computing for computational science, LNCS 6449, pp 1–25. SpringerGoogle Scholar
  44. 44.
    Young JW (1974) A first order approximation to the optimum checkpoint interval. Commun ACM 17(9):530–531CrossRefzbMATHGoogle Scholar
  45. 45.
    Young JW (1974) A first order approximation to the optimum checkpoint interval. Commun ACM 17(9):530–531 SeptCrossRefGoogle Scholar
  46. 46.
    Zheng G, Ni X, Kalé LV (2012) A scalable double in-memory checkpoint and restart scheme towards exascale. In: Dependable systems and networks workshops (DSN-W)Google Scholar
  47. 47.
    Zheng G, Shi L, Kalé LV (2004) FTC-Charm++: an in-memory checkpoint-based fault tolerant runtime for Charm++ and MPI. In: Proceedings of the IEEE international conference on cluster computing. IEEE Computer SocietyGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Guillaume Aupy
    • 1
  • Anne Benoit
    • 1
    Email author
  • Mohammed El Mehdi Diouri
    • 2
  • Olivier Glück
    • 3
  • Laurent Lefèvre
    • 4
  1. 1.Ecole Normale Supérieure de LyonLyon Cedex 07France
  2. 2.IGA CasablancaCasablancaMorocco
  3. 3.Université Claude Bernard de LyonVilleurbanneFrance
  4. 4.INRIA & Ecole Normale Supérieure de LyonLyon Cedex 07France

Personalised recommendations