Abstract
According to the great hunger in performance capability and scalability for remote sensing analysis models, it is important to exploit scalable parallelism for remote sensing data analysis models. In this paper, a method named data transformation graph (shortly DTG) is introduced, which describes an analysis model by transformations among data items. DTG can be used to study the solvability and performance of analysis models. Taking global drought detection as an example, its execution and optimization are studied carefully by DTG, and some methods are proposed for accelerating remote sensing data analysis models. At last, a distributed data-intensive computing test system is built based on Robinia, and global drought detection application is implemented for performance evaluation. The test result shows that DTG based parallelization and optimization improves the performance with high efficiency evidently, and DTG is valuable to study and optimize remote sensing data analysis models for higher performance in distributed and parallel computing environments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Loveman, D.B.: High performance fortran. IEEE Parallel Distrib. Technol.: Syst. Appl. 1(1), 25–42 (1993)
Dagum, L., Menon, R.: OpenMP: an industry standard API for shared-memory programming. IEEE Comput. Sci. Eng. 5(1), 46–55 (1998)
Geist, A. (ed.): PVM: Parallel Virtual Machine: A Users’ Guide and Tutorial for Networked Parallel Computing. The MIT Press, Cambridge (1994)
Gropp, W., Lusk, E., Skjellum, A.: Using MPI: Portable Parallel Programming with the Message Passing Interface, vol. 1. MIT Press, Cambridge (1999)
Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the condor experience. Concurr. Comput.: Prac. Exp. 17(2–4), 323–356 (2005)
Cossu, R., Bally, P., Colin, O., Fusco, L.: ESA grid processing on demand for fast access to earth observation data and rapid mapping of flood events. European Geosciences Union General Assembly (2008)
Sekiguchi, S., Tanaka, Y., Kojima, I., Yamamoto, N., Yokoyama, S., Tanimura, Y., et al.: Design principles and IT overviews of the GEO Grid. IEEE Syst. J. 2(3), 374–389 (2008)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI), pp. 137–150, December 2004
White, T.: Hadoop: The Definitive Guide. O’Reilly Media Inc., Sebastopol (2012)
Lee, C.A., Gasster, S.D., Plaza, A., Chang, C.I., Huang, B.: Recent developments in high performance computing for remote sensing: a review. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 4(3), 508–527 (2011)
Olston, C., Reed, B., Srivastava, U., Kumar, R., Tomkins, A.: Pig latin: a not-so-foreign language for data. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, pp. 1099–1110. ACM (2008)
Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. ACM SIGOPS Oper. Syst. Rev. 41(3), 59–72 (2007)
Chaiken, R., Jenkins, B., Larson, P.Å., Ramsey, B., Shakib, D., Weaver, S., Zhou, J.: SCOPE: easy and efficient parallel processing of massive data sets. Proc. VLDB Endow. 1(2), 1265–1276 (2008)
Ghemawat, S., Gobioff, H., Leung, S.T.: The google file system. ACM SIGOPS Oper. Syst. Rev. 37(5), 29–43 (2003)
Chang, F., et al.: Bigtable: a distributed storage system for structured data. In: OSDI 2006, pp. 205–218 (2006)
Melnik, S., Gubarev, A., Long, J.J., Romer, G., Shivakumar, S., Tolton, M., Vassilakis, T.: Dremel: interactive analysis of web-scale datasets. Proc. VLDB Endow. 3(1–2), 330–339 (2010)
Corbett, J.C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J.J., et al.: Spanner: google’s globally-distributed database. In: Proceedings of the 10th USENIX Symposium on Operating System Design and Implementation (OSDI 2012), pp. 251–264 (2012)
Mandl, D.: Matsu: an elastic cloud connected to a sensorweb for disaster response. In: Workshop on Cloud Computing for Spacecraft Operations, Ground System Architectures Workshop (GSAW), 2 March 2011
Guan, X., Wu, H., Li, L.: A parallel framework for processing massive spatial data with a SplitCandCMerge paradigm. Trans. GIS 16(6), 829–843 (2012)
Thies, W., Karczmarek, M., Amarasinghe, S.: Streamit: a language for streaming applications. In: Nigel Horspool, R. (ed.) CC 2002. LNCS, vol. 2304, pp. 179–196. Springer, Heidelberg (2002)
Halbwachs, N., Caspi, P., Raymond, P., Pilaud, D.: The synchronous data flow programming language LUSTRE. Proc. IEEE 79(9), 1305–1320 (1991)
Gao, B.C.: NDWI-a normalized difference water index for remote sensing of vegetation liquid water from space. Remote Sens. Environ. 58(3), 257–266 (1996)
Wang, Q.S., Zhao, D., Huang, Z.C.: Research on the performance of virtualization-based remote sensing data processing platform. In: 2012 International Conference on Systems and Informatics (ICSAI 2012), Yantai, China, 19–21 May 2012
McGrath, R.E., Xinjian, L., Folk, M.: Java (TM) applications using NCSA HDF files. Concurr. Prac. Exp. 9(11), 1113–1125 (1997)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Huang, Z., Li, G. (2015). Exploiting Scalable Parallelism for Remote Sensing Analysis Models by Data Transformation Graph. In: Wang, G., Zomaya, A., Martinez, G., Li, K. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2015. Lecture Notes in Computer Science(), vol 9530. Springer, Cham. https://doi.org/10.1007/978-3-319-27137-8_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-27137-8_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-27136-1
Online ISBN: 978-3-319-27137-8
eBook Packages: Computer ScienceComputer Science (R0)