Abstract
Capturing provenance data for runtime analysis has several challenges in high performance computational science engineering applications. The main issues are avoiding significant overhead in data capture, loading and runtime query support; and coupling provenance capture mechanisms with applications built with highly efficient numerical libraries, and visualization frameworks targeted to high performance environments. This work presents DfA-prov, an approach to capture provenance data and domain data aiming at high performance applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Rüde, U., Willcox, K., McInnes, L.C., Sterck, H.D., Biros, G., et al.: Research and Education in Computational Science and Engineering. CoRR. abs/1610.02608 (2016)
IDEAS productivity. https://ideas-productivity.org
Bernholdt, D., Dubey, A., Heroux, M., Klinvex, A., McInnes, L.C.: Improving reproducibility through better software practices. In: SIAM Conference on CSE, Atlanta, GA (2017)
Alnæs, M., Blechta, J., Hake, J., Johansson, A., Kehlet, B., et al.: Archive of Numerical Software: The FEniCS Project Version 1.5. University Library Heidelberg (2015)
Stamatogiannakis, M., et al.: Trade-offs in automatic provenance capture. In: Mattoso, M., Glavic, B. (eds.) IPAW 2016. LNCS, vol. 9672, pp. 29–41. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40593-3_3
Moreau, L., Batlajery, B.V., Huynh, T.D., Michaelides, D., Packer, H.: A templating system to generate provenance. IEEE Trans. Softw. Eng. 44, 103–121 (2018)
Pimentel, J.F., Murta, L., Braganholo, V., Freire, J.: noWorkflow: a tool for collecting, analyzing, and managing provenance from python scripts. PVLDB 10, 1841–1844 (2017)
Miles, S., Groth, P., Munroe, S., Moreau, L.: PrIMe: a methodology for developing provenance-aware applications. ACM Trans. Softw. Eng. Methodol. 20, 1–42 (2011)
Silva, V., De Oliveira, D., Valduriez, P., Mattoso, M.: DfAnalyzer: runtime dataflow analysis of scientific applications using provenance. In: PVLDB, Rio de Janeiro, Brazil (2018)
Camata, J.J., Silva, V., Valduriez, P., Mattoso, M., Coutinho, A.L.G.A.: In situ visualization and data analysis for turbidity currents simulation. Comput. Geosci. 110, 23–31 (2018)
DfAnalyzer tool demonstration. https://github.com/vssousa/dfanalyzer-spark
Acknowledgments
We thank Vinícius Campos for his help in DfA-prov development. The research has received funding from CAPES, CNPq, FAPERJ and Inria (SciDISC projects), the European Commission (HPC4E H2020 project), and the Brazilian Ministry of Science, Technology, 290 Innovation and Communications. It has been performed (for P. Valduriez) in the context of the Computational Biology Institute.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Silva, V. et al. (2018). Capturing Provenance for Runtime Data Analysis in Computational Science and Engineering Applications. In: Belhajjame, K., Gehani, A., Alper, P. (eds) Provenance and Annotation of Data and Processes. IPAW 2018. Lecture Notes in Computer Science(), vol 11017. Springer, Cham. https://doi.org/10.1007/978-3-319-98379-0_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-98379-0_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-98378-3
Online ISBN: 978-3-319-98379-0
eBook Packages: Computer ScienceComputer Science (R0)