Abstract
Satellite imagery have always been “big” data. Array DBMS is one of the tools to streamline raster data processing. However, raster data are usually stored in files, not in databases. Respective command line tools have long been developed to process these files. Most of the tools are feature-rich and free but optimized for a single machine. The approach of partially delegating in situ raster data processing to such tools has been recently proposed. The approach includes a new formal N-d array data model to abstract from the files and the tools as well as new formal distributed algorithms based on the model. ChronosServer is a distributed array DBMS under development into which the approach is being integrated. This paper extends the approach with a new algorithm for the reshaping (tiling) of arbitrary N-d arrays onto a set of overlapping N-d arrays with a fixed shape. Cutting arrays with an overlap enables to perform a broad range of large imagery processing operations in a distributed shared-nothing fashion. Currently ChronosServer provides a rich collection of raster operations at scale and outperforms SciDB up to 80\(\times \) on Landsat data. SciDB is the only freely available distributed array DBMS to date. Experiments were carried out on 8- and 16-node clusters in Microsoft Azure Cloud.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
ArcGIS for server — Image Extension. http://www.esri.com/software/arcgis/arcgisserver/extensions/image-extension
Baumann, P., Dumitru, A.M., Merticariu, V.: The array database that is not a database: file based array query answering in rasdaman. In: Nascimento, M.A., Sellis, T., Cheng, R., Sander, J., Zheng, Y., Kriegel, H.-P., Renz, M., Sengstock, C. (eds.) SSTD 2013. LNCS, vol. 8098, pp. 478–483. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40235-7_32
Baumann, P., Holsten, S.: A comparative analysis of array models for databases. Int. J. Database Theory Appl. 5(1), 89–120 (2012)
Blanas, S., Wu, K., Byna, S., Dong, B., Shoshani, A.: Parallel data analysis directly on scientific file formats. In: ACM SIGMOD 2014, pp. 385–396 (2014)
Coverity scan: GDAL. https://scan.coverity.com/projects/gdal
Cudre-Mauroux, P., et al.: A demonstration of SciDB: a science-oriented DBMS. Proc. VLDB Endowment 2(2), 1534–1537 (2009)
Earth on AWS. https://aws.amazon.com/earth/
GeoTIFF. http://trac.osgeo.org/geotiff/
Hadoop streaming. wiki.apache.org/hadoop/HadoopStreaming
ImageMagic: History. http://imagemagick.org/script/history.php
Interpolation - SciDB forum. http://forum.paradigm4.com/t/interpolation/1283
Landsat apps. https://aws.amazon.com/blogs/aws/start-using-landsat-on-aws/
Landsat project statistics. https://landsat.usgs.gov/landsat-project-statistics
Nativi, S., Caron, J., Domenico, B., Bigagli, L.: Unidatas common data model mapping to the ISO 19123 data model. Earth Sci. Inf. 1, 59–78 (2008)
Oracle spatial and graph. http://www.oracle.com/technetwork/database/options/spatialandgraph/overview/index.html
Papadopoulos, S., et al.: The TileDB array data storage manager. Proc. VLDB Endowment 10, 349–360 (2016)
PostGIS raster data management. http://postgis.net/docs/manual-2.2/using_raster_dataman.html
RasDaMan features. http://www.rasdaman.org/wiki/Features
Richards, J.A.: Remote Sensing Digital Image Analysis: An Introduction, 5th edn. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-30062-2
Rodriges Zalipynis, R.A.: Chronosserver: real-time access to “native” multi-terabyte retrospective data warehouse by thousands of concurrent clients. Inf. Cybern. Comput. Eng. 14(188), 151–161 (2011)
Rodriges Zalipynis, R.A.: ChronosServer: fast in situ processing of large multidimensional arrays with command line tools. In: Voevodin, V., Sobolev, S. (eds.) RuSCDays 2016. CCIS, vol. 687, pp. 27–40. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-55669-7_3
Rodriges Zalipynis, R.A.: Distributed in situ processing of big raster data in the Cloud. In: Perspectives of System Informatics - 11th International Andrei Ershov Informatics Conference, PSI 2017, Moscow, Russia, June 27–29, 2017, Revised Selected Papers. LNCS. Springer (2017, in press)
SciDB output chunk distribution. http://forum.paradigm4.com/t/does-store-redistributes-chunks-among-cluster-nodes/1919
SciDB window functions. http://forum.paradigm4.com/t/user-defined-window-functions/1790
SciDB streaming. https://github.com/Paradigm4/streaming
Zhang, Y., et al.: SciQL: bridging the gap between science and relational DBMS. In: IDEAS (2011)
Acknowledgments
This work was partially supported by Russian Foundation for Basic Research (grant №16-37-00416).
Author information
Authors and Affiliations
Contributions
Rodriges: all text, figures, design and implementation of algorithms and ChronosServer, ChronosServer data model, Azure management code, SciDB import code, experimental setup. Pozdeev: SciDB cluster deployment. Bryukhov: adapted SciDB import code to Landsat data. All authors: experiments.
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Rodriges Zalipynis, R.A., Pozdeev, E., Bryukhov, A. (2018). Array DBMS and Satellite Imagery: Towards Big Raster Data in the Cloud. In: van der Aalst, W., et al. Analysis of Images, Social Networks and Texts. AIST 2017. Lecture Notes in Computer Science(), vol 10716. Springer, Cham. https://doi.org/10.1007/978-3-319-73013-4_25
Download citation
DOI: https://doi.org/10.1007/978-3-319-73013-4_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-73012-7
Online ISBN: 978-3-319-73013-4
eBook Packages: Computer ScienceComputer Science (R0)