Advertisement

Portals for Interactive Steering of HPC Workflows

  • Robert SettlageEmail author
  • Srijith Rajamohan
  • Kevin Lahmers
  • Alan Chalker
  • Eric Franz
  • Steve Gallo
  • David Hudak
Conference paper
  • 16 Downloads
Part of the Communications in Computer and Information Science book series (CCIS, volume 1190)

Abstract

High performance computing workloads often benefit from human in the loop interactions. Steps in complex pipelines ranging from quality control to parameter adjustments are critical to the successful and efficient completion of modern problems. We give several example workflows in bioinformatics and deep learning where computing decisions are made throughout the processing pipelines ultimately changing the course of the compute. We also show how users can interact with the pipeline using Open OnDemand plus XDMoD or Plot.ly.

Keywords

HPC OnDemand XDMoD Steering Workflow Deep learning Bioinformatics 

References

  1. 1.
    Reinsel, D., Grantz, J., Rydning, J.: The Digitization of the World - From Edge to Core. IDC White Paper - #US44413318 (2018)Google Scholar
  2. 2.
    Hudak, D., et al.: Open OnDemand: a web-based client portal for HPC centers. J. Open Source Softw. 3(25), 622 (2018)CrossRefGoogle Scholar
  3. 3.
    Palmer, J.T., et al.: Open XDMoD: a tool for the comprehensive management of high-performance computing resources. Comput. Sci. Eng. 17(4), 52–62 (2015)CrossRefGoogle Scholar
  4. 4.
    Zhou, Z.H.: A brief introduction to weakly supervised learning. Nat. Sci. Rev. 5(1), 44–53 (2017)CrossRefGoogle Scholar
  5. 5.
    Apache Airflow Documentation: Apache Airflow Documentation - Airflow Documentation. https://airflow.apache.org/. Accessed 13 Sept 2019
  6. 6.
    Kim, D., Song, L., Breitwieser, F.P., Salzberg, S.L.: Centrifuge: rapid and sensitive classification of metagenomic sequences. Genome Res. 26(12), 1721–1729 (2016). Epub 2016 Oct 17. PubMed PMID: 27852649; PubMed Central PMCID: PMC5131823CrossRefGoogle Scholar
  7. 7.
    Breitwieser, F.P., Salzberg, S.L.: Pavian: interactive analysis of metagenomics data for microbiome studies and pathogen identification. Bioinformatics (2019).  https://doi.org/10.1093/bioinformatics/btz715. pii: btz715. [Epub ahead of print] PubMed PMID: 31553437
  8. 8.
    Rajamohan, S., Romanella, A., Ramesh, A.: A weakly-supervised attention-based visualization tool for assessing political affiliation. arXiv:1908.02282 [cs.CL] (2019)
  9. 9.
    Pontiki, M., et al.: SemEval-2016 task 5: aspect based sentiment analysis. In: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016) (2016)Google Scholar
  10. 10.
    Python-rq.org: RQ: Simple job queues for Python (2019). http://python-rq.org/. Accessed 13 Sept 2019
  11. 11.
    Tweepy: tweepy/tweepy, GitHub, 04 September 2019. https://github.com/tweepy/tweepy. Accessed 13 Sept 2019
  12. 12.
    The most popular database for modern apps, MongoDB. https://www.mongodb.com/. Accessed 13 Sept 2019
  13. 13.
    Metabase is the easy, open source way for everyone in your company to ask questions and learn from data. Metabase. https://metabase.com/. Accessed 13 Sept 2019
  14. 14.
    Explosion: explosion/spaCy, GitHub. https://github.com/explosion/spaCy. Accessed 13 Sept 2019
  15. 15.
    Modern Analytic Apps for the Enterprise, Plotly. https://plot.ly/. Accessed 13 Sept 2019
  16. 16.
    Comet.ml: Comet.ml - Supercharging Machine Learning. https://www.comet.ml/. Accessed 13 Sept 2019
  17. 17.
    Peters, M.E., et al.: Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018)

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Advanced Research ComputingVirginia TechBlacksburgUSA
  2. 2.Virginia Maryland College of Veterinary MedicineBlacksburgUSA
  3. 3.Ohio Supercomputer CenterColumbusUSA
  4. 4.University of BuffaloBuffaloUSA

Personalised recommendations