Skip to main content
SpringerLink
Log in

Generic Distributed In Situ Aggregation for Earth Remote Sensing Imagery

  • Conference paper
  • First Online:
Book cover Analysis of Images, Social Networks and Texts (AIST 2018)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11179))

Abstract

Earth remote sensing imagery come from satellites, unmanned aerial vehicles, airplanes, and other sources. National agencies, commercial companies, and individuals across the globe collect enormous amounts of such imagery daily. Array DBMS are one of the prominent tools to manage and process large volumes of geospatial imagery. The core data model of an array DBMS is an N-dimensional array. Recently we presented a geospatial array DBMS – ChronosDB – which outperforms SciDB by up to \(75\times \) on average. We are about to launch a Cloud service running our DBMS. SciDB is the only freely available distributed array DBMS to date. Remote sensing imagery are traditionally stored in files of sophisticated formats, not in databases. Unlike SciDB, ChronosDB does not require importing files into an internal DBMS format and works with imagery “in situ”: directly in their native file formats. This is one of the many virtues of ChronosDB. It has now certain aggregation capabilities, but this paper focuses on more advanced aggregation queries which still constitute a large portion of a typical workload applied to remote sensing imagery. We integrate the aggregation types into the data model, present the respective algorithms to perform aggregations in a distributed fashion, and thoroughly compare the performance of our technique with SciDB. We carried out experiments on real-world data on 8- and 16-node clusters in Microsoft Azure Cloud.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. ArcGIS for server—Image Extension. http://www.esri.com/software/arcgis/arcgisserver/extensions/image-extension

  2. Baumann, P., Dumitru, A.M., Merticariu, V.: The array database that is not a database: file based array query answering in RasDaMan. In: Nascimento, M.A., et al. (eds.) SSTD 2013. LNCS, vol. 8098, pp. 478–483. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40235-7_32

    Chapter  Google Scholar 

  3. Baumann, P., Holsten, S.: A comparative analysis of array models for databases. Int. J. Database Theory Appl. 5(1), 89–120 (2012)

    Google Scholar 

  4. Cudre-Mauroux, P., et al.: A demonstration of SciDB: a science-oriented DBMS. PVLDB 2(2), 1534–1537 (2009)

    Google Scholar 

  5. Earth on AWS. https://aws.amazon.com/earth/

  6. Hadoop streaming. https://wiki.apache.org/hadoop/HadoopStreaming

  7. Landsat apps. https://aws.amazon.com/blogs/aws/start-using-landsat-on-aws/

  8. Nativi, S., Caron, J., Domenico, B., Bigagli, L.: Unidata’s common data model mapping to the ISO 19123 data model. Earth Sci. Inform. 1, 59–78 (2008)

    Article  Google Scholar 

  9. Newberry, R.G., Lupo, A.R., Jensen, A.D., Rodriges Zalipynis, R.A.: An analysis of the spring-to-summer transition in the West Central Plains for application to long range forecasting. Atmos. Clim. Sci. 6(3), 375–393 (2016)

    Google Scholar 

  10. Oracle spatial and graph. http://www.oracle.com/technetwork/database/options/spatialandgraph/overview/index.html

  11. Papadopoulos, S., et al.: The TileDB array data storage manager. PVLDB 10(4), 349–360 (2016)

    Google Scholar 

  12. Planet Labs. https://www.planet.com/company/

  13. PostGIS raster data management. http://postgis.net/docs/manual-2.2/using_raster_dataman.html

  14. RasDaMan features. http://www.rasdaman.org/wiki/Features

  15. Rodriges Zalipynis, R.A.: ChronosServer: real-time access to “native” multi-terabyte retrospective data warehouse by thousands of concurrent clients. Inform. Cybern. Comput. Eng. 14(188), 151–161 (2011)

    Google Scholar 

  16. Rodriges Zalipynis, R.A.: Efficient isolines construction method for visualization of gridded georeferenced data. Probl. Model. Des. Autom. 10(197), 111–123 (2011)

    Google Scholar 

  17. Rodriges Zalipynis, R.A.: Representing Earth remote sensing data as time series. Syst. Anal. Environ. Soc. Sci. 2(3), 135–145 (2012)

    Google Scholar 

  18. Rodriges Zalipynis, R.A.: Ecologic assessment of air pollution by nitrogen dioxide over the territory of Europe using Earth remote sensing data. Inform. Cybern. Comput. Eng. 1(19), 126–130 (2014)

    Google Scholar 

  19. Rodriges Zalipynis, R.A.: ChronosServer: fast in situ processing of large multidimensional arrays with command line tools. In: Voevodin, V., Sobolev, S. (eds.) RuSCDays 2016. CCIS, vol. 687, pp. 27–40. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-55669-7_3

    Chapter  Google Scholar 

  20. Rodriges Zalipynis, R.A.: Array DBMS in environmental science: satellite sea surface height data in the cloud. In: 9th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS 2017, Bucharest, Romania, 21–23 September 2017, pp. 1062–1065. IEEE (2017). https://doi.org/10.1109/IDAACS.2017.8095248

  21. Rodriges Zalipynis, R.A.: ChronosDB: distributed, file based, geospatial array DBMS. PVLDB 11(10), 1247–1261 (2018). http://www.vldb.org/pvldb/vol11/p1247-zalipynis.pdf

    Google Scholar 

  22. Rodriges Zalipynis, R.A.: Distributed in situ processing of big raster data in the cloud. In: Petrenko, A.K., Voronkov, A. (eds.) PSI 2017. LNCS, vol. 10742, pp. 337–351. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-74313-4_24

    Chapter  Google Scholar 

  23. Rodriges Zalipynis, R.A., et al.: The Wikience: community data science. Concept and implementation. In: Informatics and Computer Technologies, pp. 113–117. DNTU (2011)

    Google Scholar 

  24. Rodriges Zalipynis, R.A., et al.: Retrospective satellite data in the cloud: an array DBMS approach. In: Voevodin, V., Sobolev, S. (eds.) RuSCDays 2017. Communications in Computer and Information Science, vol. 793, pp. 351–362. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-71255-0_28

    Chapter  Google Scholar 

  25. Rodriges Zalipynis, R.A., Pozdeev, E., Bryukhov, A.: Array DBMS and satellite imagery: towards big raster data in the cloud. In: van der Aalst, W.M.P., et al. (eds.) AIST 2017. LNCS, vol. 10716, pp. 267–279. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73013-4_25

    Chapter  Google Scholar 

  26. SciDB streaming. https://github.com/Paradigm4/streaming

  27. TileDB. http://istc-bigdata.org/tiledb/index.html

  28. Zhang, Y., et al.: SciQL: bridging the gap between science and relational DBMS. In: IDEAS (2011)

    Google Scholar 

Download references

Acknowledgments

This work was partially supported by Russian Science Foundation (grant №17-11-01052).

Author information

Authors and Affiliations

  1. National Research University Higher School of Economics, Moscow, Russia

    Ramon Antonio Rodriges Zalipynis

Authors
  1. Ramon Antonio Rodriges Zalipynis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ramon Antonio Rodriges Zalipynis .

Editor information

Editors and Affiliations

  1. RWTH Aachen University, Aachen, Germany

    Wil M. P. van der Aalst

  2. University of Ljubljana, Ljubljana, Slovenia

    Vladimir Batagelj

  3. University of Mannheim, Mannheim, Germany

    Goran Glavaš

  4. National Research University Higher School of Economics, Moscow, Russia

    Dmitry I. Ignatov

  5. Institute of Mathematics and Mechanics, Yekaterinburg, Russia

    Michael Khachay

  6. National Research University Higher School of Economics, Moscow, Russia

    Sergei O. Kuznetsov

  7. National Research University Higher School of Economics , Saint Petersburg, Russia

    Olessia Koltsova

  8. National Research University Higher School of Economics, Moscow, Russia

    Irina A. Lomazova

  9. Moscow State University, Moscow, Russia

    Natalia Loukachevitch

  10. Loria, Vandoeuvre lès Nancy, France

    Amedeo Napoli

  11. University of Hamburg, Hamburg, Germany

    Alexander Panchenko

  12. University of Florida, Gainesville, FL, USA

    Panos M. Pardalos

  13. Ca Foscari University of Venice, Venice, Italy

    Marcello Pelillo

  14. National Research University Higher School of Economics, Nizhny Novgorod, Russia

    Andrey V. Savchenko

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rodriges Zalipynis, R.A. (2018). Generic Distributed In Situ Aggregation for Earth Remote Sensing Imagery. In: van der Aalst, W., et al. Analysis of Images, Social Networks and Texts. AIST 2018. Lecture Notes in Computer Science(), vol 11179. Springer, Cham. https://doi.org/10.1007/978-3-030-11027-7_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-11027-7_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-11026-0

  • Online ISBN: 978-3-030-11027-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation