Advertisement

Open Source File System Selection for Remote Sensing Data Operational Storage and Processing

  • Andrei N. VinogradovEmail author
  • Evgeny P. Kurshev
  • Sergey Belov
Conference paper
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 95)

Abstract

Significant increase in the number of Earth remote sensing devices requires improvement of ground-based means of automatic stream data processing, which would allow the implementation of a mass remote sensing service in real time. The experience of choosing a freely distributed parallel file system working on the Linux operating system for solving problems of operational storage and processing of remote sensing data (RSD) is presented. RSD processing is organized according to the technology which excludes multiple copying of data between the processing steps and the delivery of the program code to the data. The processed data array is a collection of both target data files with the length of up to tens of gigabytes, and a set of short files associated with them with service data of tens of kilobytes in length. Technology Hadoop is used to implement the complex. To improve data processing performance, it was decided to replace the standard HDFS file system with a more efficient one. As a result of analysis and testing, the parallel OrangeFS file system was chosen. Instead of the previously used AVRO format, HDF5 is used as the internal technological format of data exchange. The issues of optimizing the settings of the data access mechanisms stack are considered, including the mode with dynamic selection of the I/O scheduler. The use of a more advanced parallel file system made it possible to increase the processing speed up to 40% relative to the standard Hadoop file system.

Keywords

Earth remote sensing ERS Computer cluster Distributed file system Parallel file system Remote direct memory access RDM Asynchronous input-output AIO I/O scheduler 

Notes

Acknowledgments

Authors are grateful to D.N. Golubev for the help in preparing the material. The publication was prepared with the support of the state program AAAA-A19-119020690042-2 “Research and development of data mining methods”.

References

  1. 1.
    Anikeyeva, I.A.: Comparative analysis of space imagery materials obtained from the Kanopus-V and AIST-2D spacecraft. In: Proceedings of the 10th International Scientific and Practical Conference “Geodesy, Mine Surveying, Aerial Photography” [Materialy X mezhdunarodnoj nauchno-prakticheskoj konferencii “Geodezija, Markshejderija, Ajeros’emka”], 14–15 February 2019, Moscow, Russia (2019). (in Russian)Google Scholar
  2. 2.
    Federal Space Program 2016–2025 key points, Osnovnye polozhenija Federal’noj kosmicheskoj programmy 2016–2025. https://www.roscosmos.ru/22347/. (in Russian)
  3. 3.
    The new satellite system “Sphere” will launch in 2022, Kommersant newspaper, 19 July 2018. http://rusletter.com/articles/the_new_satellite_system_sphere_will_launch_in_2022. (in Russian)
  4. 4.
  5. 5.
    Chandra, A., Ghosh, S.: Remote Sensing and Geographical Information System. Narosa Publishing House, New Delhi (2006)Google Scholar
  6. 6.
    Malyshevsky, A.A.: Using the Hadoop ecosystem in the earth remote sensing data processing. In: Proceedings of the National Supercomputer Forum (NSCF 2015) [Sbornik dokladov Nacional’nogo superkomp’yuternogo foruma (NSKF 2015)], 24–27 November 2015, Pereslavl-Zalessky, Russia (2015). (in Russian)Google Scholar
  7. 7.
    Apache Hadoop. Apache software foundation. http://hadoop.apache.org/
  8. 8.
    Moroz, V., Belov, S., Nikonov, O., Ermakov, V.: Automated complex of the meteor-M-2 satellite payload data quality analysis. In: Proceedings of the 18th International Scientific and Technical Conference “FROM IMAGERY TO DIGITAL REALITY: ERS & Photogrammetry”, 24–27 September 2018, Crete, Greece (2018). http://conf.racurs.ru/conf2018/eng/program/Conference_Proceedings.pdf
  9. 9.
    Belov, S.A., Smirnov, V.N., Kleyev, A.V., Mikhailyukova, P.G.: Distributed computing environment for solving problems of the earth remote sensing (RSD), Collection of presentations of reports and articles NSKF 2018. [Sbornik prezentacij i statej dokladov NSKF 2018], 27–30 November 2018, Pereslavl-Zalessky, Russia (2018). http://2018.nscf.ru/prezentacii/. (in Russian)
  10. 10.
    Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D.E., Maltzahn, C.: Ceph: a scalable, high-performance distributed file system. In: Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI), pp. 307–320 (2006)Google Scholar
  11. 11.
    Davies, A., Orsaria, A.: Scale out with GlusterFS. Linux J. 235, 1 (2013)Google Scholar
  12. 12.
    Sun Microsystems, Inc., Santa Clara, CA, USA, Lustre file system – High performance storage architecture and scalable cluster file system (2007)Google Scholar
  13. 13.
    Konopelko, P.: MooseFS 3.0 User’s Manual, 7 January 2017. https://moosefs.com/Content/Downloads/moosefs-3-0-users-manual.pdf
  14. 14.
    BeeGFS Technology Overveiw Brochure. ThinkParQ. Trippstadter Str. 110, Kaiserslautern, Germany. https://www.beegfs.io/docs/BeeGFS_Flyer.pdf
  15. 15.
    OrangeFS Overview. Omnibond Systems LLC (2018). http://orangefs.com/#services
  16. 16.
    Blomer, J.: Experience on File Systems. Which is the best file system for you? CHEP, Okinawa, Japan (2015)Google Scholar
  17. 17.
    General Parallel File System (GPFS) product documentation, IBM Corporation 1990–2017. https://www.ibm.com/support/knowledgecenter/SSFKCN/gpfs_content.html
  18. 18.
    Introduction to HDF5, The HDF Group, 1–15 May 2019. https://portal.hdfgroup.org/display/HDF5/Introduction+to+HDF5+–+PDF
  19. 19.
    Seelam, S., Romero, R., Teller, P.: Enhancements to Linux I/O scheduling. In: Proceedings of the Linux Symposium, Ottawa Linux Symposium, vol. 2, pp. 175–192, July 2005Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Andrei N. Vinogradov
    • 1
    Email author
  • Evgeny P. Kurshev
    • 2
  • Sergey Belov
    • 3
  1. 1.Department of Information TechnologiesPeoples’ Friendship University of Russia (RUDN University)MoscowRussia
  2. 2.Ailamazyan Program Systems Institute of RAS (PSI RAS)Pereslavl DistrictRussia
  3. 3.Creation and Transfer of Technologies JSCMoscowRussia

Personalised recommendations