Data Aware Distributed Storage (DAS) for Performance Improvement Across a Hadoop Commodity Cluster

  • R. Phani BhushanEmail author
  • D. V. L. N. Somayajulu
  • S. Venkatraman
  • R. B. V. Subramanyam
Conference paper
Part of the Learning and Analytics in Intelligent Systems book series (LAIS, volume 3)


Big Data is the order of the day and has found in-roads into many areas of working other than just the internet, which has been the breeding ground for this technology. The Remote Sensing domain has also seen growth in volumes and velocity of spatial data and thus the term Spatial Big Data has been coined to refer to this type of data. Processing the spatial data for applications such as urban mapping, object detection, change detection have undergone changes for the sake of computational efficiency from being single monolithic centralized processing to distributed processing and from single core CPUs to Multicore CPUs and further to GPUs and specific hardware in terms of architecture. The two major problems faced in this regard is the size of the data to be processed per unit of memory/time and the storage and retrieval of data for efficient processing. In this paper, we discuss a method of distributing data across a HDFS cluster, which aids in fast retrieval and faster processing per unit of available memory in the Image Processing domain. We evaluate our technique and compare the same with the traditional approach on a 4-node HDFS cluster. Significant improvement is found while performing edge detection on large spatial data, which has been tabulated in the results section.


Cluster HDFS Big Data Image processing Remote sensing Spatial Big Data 


  1. 1.
    Lee CA, Gasster SD, Plaza A, Chang C-I, Huang B (2011) Recent developments in high performance computing for remote sensing: a review. IEEE J Sel Top Appl Earth Obs Remote Sens 4(3):508–527CrossRefGoogle Scholar
  2. 2.
    Lv Z, Hu Y, Zhong H, Wu J, Li B, Zhao H (2010) Parallel k-means clustering of remote sensing images based on mapreduce. In: Proceedings of the 2010 international conference on web information systems and mining, ser. WISM 2010. Springer-Verlag, Berlin, Heidelberg, pp 162–170Google Scholar
  3. 3.
    Li Y, Crandall DJ, Huttenlocher DP (2009) Landmark classification in large-scale image collections. In: ICCV, 1957–1964Google Scholar
  4. 4.
    Bajcsy P, Vandecreme A, Amelot J, Nguyen P, Chalfoun J, Brady M (2013) Terabyte sized image computations on hadoop cluster platforms. In: Big Data, 2013 IEEE international conference, October 2013, pp 729–737Google Scholar
  5. 5.
    Zhao JY, Li Q, Zhou HW (2011) A cloud-based system for spatial analysis service. In: 2011 international conference on remote sensing, environment and transportation engineering (RSETE), Nanjing, 24–26 June 2011, pp 1–4Google Scholar
  6. 6.
    Yang C-T, Chen L-T, Chou W-L, Wang K-C (2010) Implementation of a medical image file accessing system on cloud computing. In: 2010 IEEE 13th international conference on computational science and engineering (CSE), Hong Kong, 11–13 December 2010, pp 321–326.
  7. 7.
    Shelly, Raghava NS (2011) Iris recognition on hadoop: a biometrics system implementation on cloud computing. In: 2011 IEEE international conference on cloud computing and intelligence systems (CCIS), Beijing, 15–17 September 2011, pp 482–485.
  8. 8.
    Alonso-Calvo R, Crespo J, Maojo V, Muñoz A, Garcia-Remesal M, Perez-Rey D (2011) Cloud computing service for managing large medical Image data-sets using balanced collaborative agents. Adv Intell Soft Comput 88:265–270. Scholar
  9. 9.
    Phani Bhushan R, Somayajulu DVLN, Venkatraman S et al (2018) A raster data framework based on distributed heterogeneous cluster. J Indian Soc Remote Sens.
  10. 10.

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • R. Phani Bhushan
    • 1
    Email author
  • D. V. L. N. Somayajulu
    • 2
  • S. Venkatraman
    • 1
  • R. B. V. Subramanyam
    • 2
  1. 1.Department of SpaceADRINHyderabadIndia
  2. 2.NITWarangalIndia

Personalised recommendations