Distributed Cosmic Ray Detection Using Cloud Computing

  • Germán SchnyderEmail author
  • Sergio Nesmachnow
  • Gonzalo Tancredi
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 796)


This article presents a distributed computing approach to detect cosmic rays in images taken by the Hubble Space Telescope (HST). A cloud computing implementation is developed to improve the overall processing time for the available images dataset (15 TB), containing dark images from several HST instruments. A specific architecture is presented where images are stored in a replicated and highly available storage system. Image processing is performed on virtual machines from the Azure Batch framework using a developed Python application. The experimental evaluation shows that the architecture accomplished the purpose of processing the complete dataset based on scaling computing resources in terms of processing nodes. Speedup improved in a factor of \(6.57{\times }\) over a previous implementation using Apache Mesos. The overall computation took 10 days to complete and results are stored on a non-relational database available to astronomers and researchers.


Cosmic rays HST Azure Python 



This work has been partly supported by CSIC, ANII, and PEDECIBA (Uruguay).

The developed architecture contributes to project “Geophysics using Hubble Space Telescope” [11], to exploit the HST capabilities as a cosmic ray detector for analyzing the magnetosphere current strength. Using our results the researches will combine HST results with measurements of solar activity, cosmic ray flux on Earth’s surface, and geomagnetic data to understand external field variations.

Computing and storage resources were provided by Cluster FING and the “Microsoft Azure Sponsorship” program intended for researchers. This sponsorship included a cost-based usage of all the services provided by Microsoft Azure.


  1. 1.
    Astroplan: What are the IERS tables and how do I update them? Accessed July 2017
  2. 2.
    Christian, E.: Cosmic rays. Accessed July 2017
  3. 3.
    Mewaldt, R., Cummings, A., Stone, E.: Anomalous cosmic rays: interstellar interlopers in the heliosphere and magnetosphere. Eos Trans. Am. Geophys. Union 75(16), 185–193 (1994)CrossRefGoogle Scholar
  4. 4.
    Myers, T.: Designing a scalable partitioning strategy for Azure table storage. Accessed July 2017
  5. 5.
    Myers, T., Macy, M., Squillace, R.: Get started with the Azure Batch Python client. Accessed July 2017
  6. 6.
    Nesmachnow, S.: Computación científica de alto desempeño en la Facultad de Ingeniería, Universidad de la República. Revista de la Asociación de Ingenieros del Uruguay 61(1), 12–15 (2010)Google Scholar
  7. 7.
    Pence, W., Chiappetti, L., Page, C., Shaw, R., Stobie, E.: Definition of the Flexible Image Transport System (FITS), version 3.0. Astron. Astrophys. 524, A42 (2010)CrossRefGoogle Scholar
  8. 8.
    Roth, J., FitzMacken, T., Lian, J., Kemnetz, J., Squillace, R.: Azure subscription and service limits, quotas, and constraints. Accessed July 2017
  9. 9.
    Schnyder, G., Nesmachnow, S.: Improving the performance of cosmic ray detection using Apache Mesos. In: International Supercomputing Conference in México, pp. 1–15 (2016)Google Scholar
  10. 10.
    Smith, A., McDonald, R., Hurley, D., Holland, S., Groom, D., Berkeley, L., Brown, W., Gilmore, D., Stover, R., Wei, M.: Radiation events in astronomical CCD images, vol. 183, pp. 172–183 (2002)Google Scholar
  11. 11.
    Tancredi, G., Cromwell, G., Deustua, S., Gonzalez, G., Nesmachnow, S., Schnyder, G.: Geophysics using Hubble Space Telescope, Hubble Space Telescope Cycle 24 approved proposal (2016)Google Scholar
  12. 12.
    The Astropy Collaboration, Robitaille, T., Tollerud, E., Greenfield, P., Droettboom, M., Bray, E., Aldcroft, T., Davis, M., Ginsburg, A., Price, A., Kerzendorf, W., Conley, A., Crighton, N., Barbary, K., Muna, D., Ferguson, H., Grollier, F., Parikh, M., Nair, P., Gnther, H., Deil, C., Woillez, J., Conseil, S., Kramer, R., Turner, J., Singer, L., Fox, R., Weaver, B., Zabalza, V., Edwards, Z., Azalee, K., Burke, D., Casey, A., Crawford, S., Dencheva, N., Ely, J., Jenness, T., Labrie, K., Lim, P., Pierfederici, F., Pontzen, A., Ptak, A., Refsdal, B., Servillat, M., Streicher, O.: Astropy: a community python package for astronomy. Astron. Astrophys. 558, A33 (2013)Google Scholar
  13. 13.
    Tody, D.: The IRAF Data Reduction and Analysis System, pp. 1–20 (1986)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Germán Schnyder
    • 1
    Email author
  • Sergio Nesmachnow
    • 1
  • Gonzalo Tancredi
    • 1
  1. 1.Universidad de la RepúblicaMontevideoUruguay

Personalised recommendations