Abstract
We describe an in situ system for solving iterative problems. We specifically target inverse problems, where expensive simulations are approximated using a surrogate model. The model explores the parameter space of the simulation through iterative trials, each of which becomes a job managed by a parallel scheduler. Our work extends Henson [1], a cooperative multi-tasking system for in situ execution of loosely coupled codes.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
References
Morozov, D., Lukić, Z.: Master of puppets: cooperative multitasking for in situ processing. In: Proceedings of High-Performance Parallel and Distributed Computing, pp. 285–288 (2016)
Liu, Q., Logan, J., Tian, Y., Abbasi, H., Podhorszki, N., Choi, J.Y., Klasky, S., Tchoua, R., Lofstead, J., Oldfield, R., Parashar, M., Samatova, N., Schwan, K., Shoshani, A., Wolf, M., Wu, K., Yu, W.: Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks. Concurr. Comput. Pract. Exp. 26(7), 1453–1473 (2014)
Sun, Q., Jin, T., Romanus, M., Bui, H., Zhang, F., Yu, H., Kolla, H., Klasky, S., Chen, J., Parashar, M.: Adaptive data placement for staging-based coupled scientific workflows. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015, pp. 65:1–65:12. ACM, New York (2015)
Vishwanath, V., Hereld, M., Morozov, V., Papka, M.E.: Topology-aware data movement and staging for I/O acceleration on Blue Gene/P supercomputing systems. In: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2011, pp. 19:1–19:11. ACM, New York (2011)
Dorier, M., Sisneros, R., Peterka, T., Antoniu, G., Semeraro, D.: Damaris/Viz: a nonintrusive, adaptable and user-friendly in situ visualization framework. In: 2013 IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV), pp. 67–75, October 2013
Bauer, A.C., Geveci, B., Schroeder, W.: The ParaView Catalyst User’s Guide v2.0. Kitware Inc., New York (2015)
Whitlock, B., Favre, J.M., Meredith, J.S.: Parallel in situ coupling of simulation with a fully featured visualization system. In: Proceedings of the 11th Eurographics Conference on Parallel Graphics and Visualization, pp. 101–109 (2011)
Dorier, M., Dreher, M., Peterka, T., Antoniu, G., Raffin, B., Wozniak, J.M.: Lessons learned from building in situ coupling frameworks. In: First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, Austin, United States, November 2015
Ayachit, U., et al.: Performance analysis, design considerations, and applications of extreme-scale in situ infrastructures. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC) (2016)
Viel, M., Becker, G.D., Bolton, J.S., Haehnelt, M.G.: Warm dark matter as a solution to the small scale crisis: new constraints from high redshift Lyman-\(\alpha \) forest data. Phys. Rev. D 88(4), 043502 (2013)
Wozniak, J.M., Armstrong, T.G., Wilde, M., Katz, D.S., Lusk, E., Foster, I.T.: Swift/T: large-scale application composition via distributed-memory dataflow processing. In: IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pp. 95–102 (2013)
Booker, A.J., Dennis Jr., J.E., Frank, P.D., Serafini, D.B., Torczon, V., Trosset, M.W.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Multi. Optim. 17, 1–13 (1999)
Gutmann, H.-M.: A radial basis function method for global optimization. J. Global Optim. 19, 201–227 (2001)
Regis, R.G., Shoemaker, C.A.: A stochastic radial basis function method for the global optimization of expensive functions. INFORMS J. Comput. 19, 497–509 (2007)
Müller, J., Shoemaker, C.A.: Influence of ensemble surrogate models and sampling strategy on the solution quality of algorithms for computationally expensive black-box global optimization problems. J. Global Optim. 60, 123–144 (2014)
Wang, G.G., Shan, S.: Review of metamodeling techniques in support of engineering design optimization. J. Mech. Des. 129, 370–380 (2007)
Dinan, J., Krishnamoorthy, S., Balaji, P., Hammond, J.R., Krishnan, M., Tipparaju, V., Vishnu, A.: Noncollective communicator creation in MPI. In: Cotronis, Y., Danalis, A., Nikolopoulos, D.S., Dongarra, J. (eds.) EuroMPI 2011. LNCS, vol. 6960, pp. 282–291. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24449-0_32
Lukić, Z., Stark, C.W., Nugent, P., White, M., Meiksin, A.A., Almgren, A.: The Lyman \(\alpha \) forest in optically thin hydrodynamical simulations. Mon. Not. R. Astron. Soc. 446, 3697–3724 (2015)
Almgren, A.S., Bell, J.B., Lijewski, M.J., Lukić, Z., Van Andel, E.: Nyx: a massively parallel AMR code for computational cosmology. Astrophys. J. 765, 39 (2013)
Acknowledgements
We are grateful to Jack Deslippe for providing us the raw data on Edison queue times. This work was supported by Advanced Scientific Computing Research, Office of Science, U.S. Department of Energy, under Contract DE-AC02-05CH11231, and by the use of resources of the National Energy Research Scientific Computing Center (NERSC).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Lohrmann, E., Lukić, Z., Morozov, D., Müller, J. (2018). Programmable In Situ System for Iterative Workflows. In: Klusáček, D., Cirne, W., Desai, N. (eds) Job Scheduling Strategies for Parallel Processing. JSSPP 2017. Lecture Notes in Computer Science(), vol 10773. Springer, Cham. https://doi.org/10.1007/978-3-319-77398-8_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-77398-8_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-77397-1
Online ISBN: 978-3-319-77398-8
eBook Packages: Computer ScienceComputer Science (R0)