Advertisement

Lessons Learned from a Decade of Providing Interactive, On-Demand High Performance Computing to Scientists and Engineers

  • Julia Mullen
  • Albert ReutherEmail author
  • William Arcand
  • Bill Bergeron
  • David Bestor
  • Chansup Byun
  • Vijay Gadepally
  • Michael Houle
  • Matthew Hubbell
  • Michael Jones
  • Anna Klein
  • Peter Michaleas
  • Lauren Milechin
  • Andrew Prout
  • Antonio Rosa
  • Siddharth Samsi
  • Charles Yee
  • Jeremy Kepner
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11203)

Abstract

For decades, the use of HPC systems was limited to those in the physical sciences who had mastered their domain in conjunction with a deep understanding of HPC architectures and algorithms. During these same decades, consumer computing device advances produced tablets and smartphones that allow millions of children to interactively develop and share code projects across the globe. As the HPC community faces the challenges associated with guiding researchers from disciplines using high productivity interactive tools to effective use of HPC systems, it seems appropriate to revisit the assumptions surrounding the necessary skills required for access to large computational systems. For over a decade, MIT Lincoln Laboratory has been supporting interactive, on-demand high performance computing by seamlessly integrating familiar high productivity tools to provide users with an increased number of design turns, rapid prototyping capability, and faster time to insight. In this paper, we discuss the lessons learned while supporting interactive, on-demand high performance computing from the perspectives of the users and the team supporting the users and the system. Building on these lessons, we present an overview of current needs and the technical solutions we are building to lower the barrier to entry for new users from the humanities, social, and biological sciences.

Keywords

HPC abstractions Interactive On-demand HPC 

References

  1. 1.
    Bliss, N., Bond, R., Kepner, J., Kim, H., Reuther, A.: Interactive grid computing at Lincoln laboratory. MIT Lincoln Lab. J. 16, 165–216 (2006)Google Scholar
  2. 2.
    Bliss, N.T., Kepner, J., Kim, H., Reuther, A.: pMATLAB: parallel MATLAB library for signal processing applications. In: 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP 2007, vol. 4, pp. IV-1189–IV-1192, April 2007.  https://doi.org/10.1109/ICASSP.2007.367288
  3. 3.
    Byun, C., et al.: Driving big data with big compute. In: 2012 IEEE Conference on High Performance Extreme Computing HPEC, pp. 1–6, September 2012.  https://doi.org/10.1109/HPEC.2012.6408678
  4. 4.
    Denning, P.J., Lewis, T.G.: Exponential laws of computing growth. Commun. ACM 60(1), 54–65 (2016).  https://doi.org/10.1145/2976758CrossRefGoogle Scholar
  5. 5.
    Edelman, A.: The Star-P high performance computing platform. In: 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP 2007, vol. 4, pp. IV-1197-IV-1200, April 2007.  https://doi.org/10.1109/ICASSP.2007.367290
  6. 6.
    Gadepally, V., et al.: D4M: bringing associative arrays to database engines. In: 2015 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, September 2015.  https://doi.org/10.1109/HPEC.2015.7322472
  7. 7.
    Hubbell, M., et al.: Big data strategies for data center infrastructure management using a 3D gaming platform. In: 2015 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, September 2015.  https://doi.org/10.1109/HPEC.2015.7322471
  8. 8.
    Jones, M., et al.: Scalability of VM provisioning systems. In: 2016 IEEE High Performance Extreme Computing Conference, HPEC 2016 (2016).  https://doi.org/10.1109/HPEC.2016.7761629
  9. 9.
    Kepner, J.: MatlabMPI. J. Parallel Distrib. Comput. 64(8), 997–1005 (2004).  https://doi.org/10.1016/j.jpdc.2004.03.018CrossRefzbMATHGoogle Scholar
  10. 10.
    Kepner, J.: Parallel MATLAB for Multicore and Multinode Computers. Society for Industrial and Applied Mathematics, New York (2009).  https://doi.org/10.1137/1.9780898718126
  11. 11.
    Kepner, J., et al.: D4M 2.0 schema: a general purpose high performance schema for the Accumulo database. In: 2013 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, September 2013.  https://doi.org/10.1109/HPEC.2013.6670318
  12. 12.
    Kepner, J., et al.: Dynamic distributed dimensional data model (D4M) database and computation system. In: 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5349–5352, March 2012.  https://doi.org/10.1109/ICASSP.2012.6289129
  13. 13.
    Kepner, J., et al.: Achieving 100,000,000 database inserts per second using Accumulo and D4M. In: 2014 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, September 2014.  https://doi.org/10.1109/HPEC.2014.7040945
  14. 14.
    Kepner, J., et al.: Computing on masked data: a high performance method for improving big data veracity. In: 2014 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, September 2014.  https://doi.org/10.1109/HPEC.2014.7040946
  15. 15.
    Lebak, J., Kepner, J., Hoffmann, H., Rutledge, E.: Parallel VSIPL++: an open standard software library for high-performance parallel signal processing. Proc. IEEE 93(2), 313–330 (2005).  https://doi.org/10.1109/JPROC.2004.840303CrossRefGoogle Scholar
  16. 16.
    Moler, C.: Why there isn’t a parallel MATLAB. MATLAB News and Notes, p. 12 (1995, Spring). https://www.mathworks.com/company/newsletters/articles/why-there-isnt-a-parallel-matlab.html
  17. 17.
    Mullen, J., Byun, C., Gadepally, V., Samsi, S., Reuther, A., Kepner, J.: Learning by doing, high performance computing education in the MOOC era. J. Parallel Distrib. Comput. 105, 105–115 (2017).  https://doi.org/10.1016/j.jpdc.2017.01.015CrossRefGoogle Scholar
  18. 18.
    Prout, A., et al.: MIT SuperCloud portal workspace: enabling HPC web application deployment. In: 2017 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, September 2017.  https://doi.org/10.1109/HPEC.2017.8091097
  19. 19.
    Prout, A., et al.: Enabling on-demand database computing with MIT SuperCloud database management system. In: 2015 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, September 2015.  https://doi.org/10.1109/HPEC.2015.7322482
  20. 20.
    Reuther, A., et al.: Technology requirements for supporting on-demand interactive grid computing. In: Proceedings of the Department of Defense High Performance Computing Modernization Office (HPCMO) Users Group Conference (UGC) 2005. IEEE, Nashville, June 2005.  https://doi.org/10.1109/DODUGC.2005.65
  21. 21.
    Reuther, A., et al.: LLSuperCloud: sharing HPC systems for diverse rapid prototyping. In: 2013 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6, September 2013.  https://doi.org/10.1109/HPEC.2013.6670329
  22. 22.
    Reuther, A., Kepner, J., MCcabe, A., Mullen, J., Bliss, N.T., Kim, H.: Technical challenges of supporting interactive HPC. In: Department of Defense - Proceedings of the HPCMP Users Group Conference 2007; High Performance Computing Modernization Program: A Bridge to Future Defense, DoD HPCMP UGC, pp. 403–409 (2007).  https://doi.org/10.1109/HPCMP-UGC.2007.72
  23. 23.
    Reuther, A., Michaleas, P., Prout, A., Kepner, J.: HPC-VMs: virtual machines in high performance computing systems. In: 2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012 (2012).  https://doi.org/10.1109/HPEC.2012.6408668
  24. 24.
  25. 25.
  26. 26.
    Tichenor, S., Reuther, A.: Making the business case for high performance computing: a benefit-cost analysis methodology. CTWatch Q. 2(4A), 2–8 (2006)Google Scholar
  27. 27.
    Wilson, G.: Software carpentry: getting scientists to write better code by making them more productive. Comput. Sci. Eng. 8(6), 66–69 (2006).  https://doi.org/10.1109/MCSE.2006.122CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.MIT Lincoln LaboratoryLexingtonUSA
  2. 2.Massachusetts Institute of TechnologyCambridgeUSA

Personalised recommendations