Abstract
The paper submits the method how to solve big projects in the sphere of geographic information systems (GIS). Our aim is to answer the question whether we can or cannot solve similar projects on commonly used hardware and software. This method is based on making use of parallelism and optimization of individual processes. The whole GIS project is divided according to the territory principle into the individual projects which can be processed concurrently. In the frame of sub-projects data optimization of main theme is performed. After the finishing of the particular phases of the project a manual check of partial results follows. The final step consists in completing the separate results into common database. The project was solved for the GasNet, Ltd. Company which is a part of a RWE group in the Czech Republic. Input data were datasets of orthophoto with a resolution of 25 cm/pixel, layers of communications of ZABAGED CR and vector sets of the route of line of underground engineering networks. Due to the territorial coverage of the CR with the area of 64,350 km2, these were massive tasks with total data volume more than 500 GB. The data analysis was carried out in the special created application in Python language with the support of ESRI libraries and also in ArcGIS 10.0 environment.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bartoněk D, Bureš J, Opatřilová I (2014a) Technology of processing of enormous amounts of geographical data. In: 14th international multidisciplinary scientific geoconference and EXPO, SGEM 2014; Albena; Bulgaria; 17 June 2014 through 26 June 2014; Code 109739, 3(2):917–924
Bartoněk D, Bureš J, Opatřilová I (2014b) Optimization of pre-processing of extensive projects in geographic information systems. Adv Sci Lett 20(10–12):2026–2029. American Scientific Publishers. doi:10.1166/asl.2014.5664. ISSN:19366612
Bica M, Bacu V, Mihon D, Gorgan D (2014) Architectural solution for virtualized processing of big earth data. In: IEEE international conference on intelligent computer communication and processing (ICCP) 2014, pp 399–404. doi:10.1109/ICCP.2014.6937027. Print ISBN: 978-1-4799-6568-7
Boton C, Halin G, Kubicki S (2015) Challenges of big data in the age of building information modeling: a high-level conceptual pipeline. In: Cooperative design, visualization, and Engineering, pp 205–216
Dahlstrom S, Harms A (1997) Successfully integrating multiple utilities and supporting technologies. In: AM-FM-international conference XX-entering the mainstream. Nashville, TN, pp 691–701
Frye R, McKenney M (2015) Big data storage techniques for spatial databases: implications of big data architecture on spatial query processing. In: Information granularity, big data, and computing. Springer Science, pp 560–586
Idrizi B, Zhaku S, Izeiroski S (2015) Defining methodology for selecting most appropriate GIS software. Surv Rev 46(338):383–389
Kraemer M, Senner I (2015) A modular software architecture for processing of big geospatial data in the cloud. Comput Graph UK 49:69–81
Lu F, Zhang H (2014) Big data and generalized GIS. Geomat Inf Sci Wuhan Univ 39(6):645–654
Lu L, Xin L, Jianyuan X (2008) Numerical analysis of electric field for 1100 kV disconnector in GIS based on two kinds of structures. In: International conference on high voltage engineering and application, Chongqing, Peoples Republic of China, pp 527–530
Miller HJ, Goodchild MF (2014) Data-driven geography. GeoJournal 80(4):449–461
Mondai K, Dutta P (2015) Big data parallelism: challenges in different computational paradigms. In: Proceedings of the 2015 3rd international conference on computer, communication, control and information technology, C3IT 2015, 12 March 2015, Article number 7060186
Pawlak Z (1981) Information system theoretical foundations. Inf Syst 6:205–218
Schoier G, Borruso G (2015) On the problem of clustering spatial big data. In: Osvaldo G, Beniamino M, Sanjay M, Gavrilova ML, Ana Maria Alves Coutinho R, Carmelo T, David T, Apduhan BO (eds) Computational science and its applications. Proceedings, Part I: 15th international conference, Banff, AB, Canada, 22–25 June 2015, pp 325–338. ISBN 978-3-319-21403-0 (Print) 978-3-319-21404-7 (Online)
Shahrokni H, Van Der Heijde B, Lazarevic D, Brandt N (2014) Big data GIS analytics towards efficient waste management in Stockholm. In: 2nd international conference on ICT for sustainability, ICT4S 2014; Stockholm; Sweden; 24 August 2014 through 27 August 2014; Pages 140–147, Code 111677
Wang Y, Liu Z, Liao H, Li C, (2015) Improving the performance of GIS polygon overlay computation with MapReduce for spatial big data processing. In: Salim H (ed) Cluster computing, Springer Science + Business Media, New York, pp 506–516. ISSN: 1386-7857 (Print version), ISSN: 1573-7543 (Electronic version)
Yue P, Jiang L (2014) BigGIS: How big data can shape next-generation GIS. In: The 3rd international conference on agro-geoinformatics, agro-geoinformatics 2014, Beijing, China, 11 August 2014 through 14 August 2014. Category number CFP1448T-ART, Article number 6910649, Code 114697
Zhang H, Chen G, Ooi BC, Tan K-L, Zhang M (2015) In-memory big data management and processing: a survey. IEEE Trans Knowl Data Eng 27(7): 1920–1948. ISSN:1041-4347
Zhu XY, Zhu Q, Zhang YH (2003) Applications of remote sensing and GIS technologies in the planning and development information system for Beijing virescence separator. In: 2nd Conference on remote sensing for environmental monitoring, GIS applications and geology. Book series: proceedings of the society of photo-optical instrumentation engineers (SPIE), vol 4886, pp 107–114
Acknowledgments
The work was solved within the project marked FAST-S-15-2723 and within research project of MŠMT (Ministry of Education of the Czech Republic) AdMaS ED2.1.00/03.0097 Nr. HS12357021212200 “Data analysis of surfaces above RWE gas lines in the Czech republic”.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Bartoněk, D. (2017). The Possibilities of Big GIS Data Processing on the Desktop Computers. In: Ivan, I., Singleton, A., Horák, J., Inspektor, T. (eds) The Rise of Big Spatial Data. Lecture Notes in Geoinformation and Cartography. Springer, Cham. https://doi.org/10.1007/978-3-319-45123-7_20
Download citation
DOI: https://doi.org/10.1007/978-3-319-45123-7_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-45122-0
Online ISBN: 978-3-319-45123-7
eBook Packages: Earth and Environmental ScienceEarth and Environmental Science (R0)