Abstract
Significant increase in the number of Earth remote sensing devices requires improvement of ground-based means of automatic stream data processing, which would allow the implementation of a mass remote sensing service in real time. The experience of choosing a freely distributed parallel file system working on the Linux operating system for solving problems of operational storage and processing of remote sensing data (RSD) is presented. RSD processing is organized according to the technology which excludes multiple copying of data between the processing steps and the delivery of the program code to the data. The processed data array is a collection of both target data files with the length of up to tens of gigabytes, and a set of short files associated with them with service data of tens of kilobytes in length. Technology Hadoop is used to implement the complex. To improve data processing performance, it was decided to replace the standard HDFS file system with a more efficient one. As a result of analysis and testing, the parallel OrangeFS file system was chosen. Instead of the previously used AVRO format, HDF5 is used as the internal technological format of data exchange. The issues of optimizing the settings of the data access mechanisms stack are considered, including the mode with dynamic selection of the I/O scheduler. The use of a more advanced parallel file system made it possible to increase the processing speed up to 40% relative to the standard Hadoop file system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anikeyeva, I.A.: Comparative analysis of space imagery materials obtained from the Kanopus-V and AIST-2D spacecraft. In: Proceedings of the 10th International Scientific and Practical Conference “Geodesy, Mine Surveying, Aerial Photography” [Materialy X mezhdunarodnoj nauchno-prakticheskoj konferencii “Geodezija, Markshejderija, Ajeros’emka”], 14–15 February 2019, Moscow, Russia (2019). (in Russian)
Federal Space Program 2016–2025 key points, Osnovnye polozhenija Federal’noj kosmicheskoj programmy 2016–2025. https://www.roscosmos.ru/22347/. (in Russian)
The new satellite system “Sphere” will launch in 2022, Kommersant newspaper, 19 July 2018. http://rusletter.com/articles/the_new_satellite_system_sphere_will_launch_in_2022. (in Russian)
List of file systems. https://en.wikipedia.org/wiki/List_of_file_systems#Distributed_file_systems
Chandra, A., Ghosh, S.: Remote Sensing and Geographical Information System. Narosa Publishing House, New Delhi (2006)
Malyshevsky, A.A.: Using the Hadoop ecosystem in the earth remote sensing data processing. In: Proceedings of the National Supercomputer Forum (NSCF 2015) [Sbornik dokladov Nacional’nogo superkomp’yuternogo foruma (NSKF 2015)], 24–27 November 2015, Pereslavl-Zalessky, Russia (2015). (in Russian)
Apache Hadoop. Apache software foundation. http://hadoop.apache.org/
Moroz, V., Belov, S., Nikonov, O., Ermakov, V.: Automated complex of the meteor-M-2 satellite payload data quality analysis. In: Proceedings of the 18th International Scientific and Technical Conference “FROM IMAGERY TO DIGITAL REALITY: ERS & Photogrammetry”, 24–27 September 2018, Crete, Greece (2018). http://conf.racurs.ru/conf2018/eng/program/Conference_Proceedings.pdf
Belov, S.A., Smirnov, V.N., Kleyev, A.V., Mikhailyukova, P.G.: Distributed computing environment for solving problems of the earth remote sensing (RSD), Collection of presentations of reports and articles NSKF 2018. [Sbornik prezentacij i statej dokladov NSKF 2018], 27–30 November 2018, Pereslavl-Zalessky, Russia (2018). http://2018.nscf.ru/prezentacii/. (in Russian)
Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D.E., Maltzahn, C.: Ceph: a scalable, high-performance distributed file system. In: Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI), pp. 307–320 (2006)
Davies, A., Orsaria, A.: Scale out with GlusterFS. Linux J. 235, 1 (2013)
Sun Microsystems, Inc., Santa Clara, CA, USA, Lustre file system – High performance storage architecture and scalable cluster file system (2007)
Konopelko, P.: MooseFS 3.0 User’s Manual, 7 January 2017. https://moosefs.com/Content/Downloads/moosefs-3-0-users-manual.pdf
BeeGFS Technology Overveiw Brochure. ThinkParQ. Trippstadter Str. 110, Kaiserslautern, Germany. https://www.beegfs.io/docs/BeeGFS_Flyer.pdf
OrangeFS Overview. Omnibond Systems LLC (2018). http://orangefs.com/#services
Blomer, J.: Experience on File Systems. Which is the best file system for you? CHEP, Okinawa, Japan (2015)
General Parallel File System (GPFS) product documentation, IBM Corporation 1990–2017. https://www.ibm.com/support/knowledgecenter/SSFKCN/gpfs_content.html
Introduction to HDF5, The HDF Group, 1–15 May 2019. https://portal.hdfgroup.org/display/HDF5/Introduction+to+HDF5+–+PDF
Seelam, S., Romero, R., Teller, P.: Enhancements to Linux I/O scheduling. In: Proceedings of the Linux Symposium, Ottawa Linux Symposium, vol. 2, pp. 175–192, July 2005
Acknowledgments
Authors are grateful to D.N. Golubev for the help in preparing the material. The publication was prepared with the support of the state program AAAA-A19-119020690042-2 “Research and development of data mining methods”.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Vinogradov, A.N., Kurshev, E.P., Belov, S. (2020). Open Source File System Selection for Remote Sensing Data Operational Storage and Processing. In: Arseniev, D., Overmeyer, L., Kälviäinen, H., Katalinić, B. (eds) Cyber-Physical Systems and Control. CPS&C 2019. Lecture Notes in Networks and Systems, vol 95. Springer, Cham. https://doi.org/10.1007/978-3-030-34983-7_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-34983-7_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-34982-0
Online ISBN: 978-3-030-34983-7
eBook Packages: EngineeringEngineering (R0)