A Raster Data Framework Based on Distributed Heterogeneous Cluster

Abstract

Advancements in satellite imaging and sensor technologies result in capturing of large amount of spatial data. Many parallel processing techniques based on data or control parallelism have been attempted during the past 2 decades to provide performance improvement in image processing applications such as urban sprawl, weather prediction and crop estimation. These techniques have used block-based distributed file processing or the more modern MapReduce-based programming for implementation which still have gaps between optimal and best processing in terms of resource scheduling, data distribution and ease of programming. In this paper, we present a layered framework for parallel data processing to improve storage, retrieval and processing performance of spatial data on an underlying distributed file system. The paper presents a data placement strategy across a distributed HDFS cluster in a way to optimize spatial data retrieval and processing. The presence of neighborhood pixels local to the processing node in a distributed environment reduces network latencies and improves the efficiency of applications such as object recognition, change detection and site selection. We evaluate the data placement strategy on a four-node HDFS cluster and show that it can deliver good performance benefits by way of reading blocks of data at almost 10–12 times the default, which contributes to the improvement in efficiency of the various applications that use region growing methods.