Abstract
The study of clouds, i.e., where they occur and what are their characteristics, plays a key role in the understanding of climate change. Clustering is a common machine learning technique used in atmospheric science to classify cloud types. Many parallelism techniques e.g., MPI, OpenMP and Spark, could achieve efficient and scalable clustering of large-scale satellite observation data. In order to understand their differences, this paper studies and compares three different approaches on parallel clustering of satellite observation data. Benchmarking experiments with k-means clustering are conducted with three parallelism techniques, namely OpenMP, OpenMP+MPI, and Spark, on a HPC cluster using up to 16 nodes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Apache Software Foundation: Apache spark - unified analytics engine for big data. https://spark.apache.org/. Accessed 15 June 2018
Apache Software Foundation: MLlib \(|\) Apache Spark. https://spark.apache.org/mllib/. Accessed 15 June 2018
Apache Software Foundation: Spark MLlib Python API docs. https://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark/.ml.clustering.KMeans. Accessed 15 June 2018
Apache Software Foundation: Spark SQL, dataframes and datasets guide. https://spark.apache.org/docs/2.3.0/sql-programming-guide.html. Accessed 15 June 2018
Arora, K., Barajas, C., Gobbert, M.K.: Parallel performance studies for an elliptic test problem on the Stampede2 cluster and comparison of networks. Technical report HPCF-2018-10, UMBC High Performance Computing Facility, University of Maryland, Baltimore County (2018)
Barajas, C., Guo, P., Mukherjee, L., Daeho, J.: https://github.com/big-data-lab-umbc/cybertraining/tree/master/year-1-projects/team-2. Source Code
Behnel, S., Bradshaw, R., Citro, C., Dalcin, L., Seljebotn, D.S., Smith, K.: Cython: the best of both worlds. Comput. Sci. Eng. 13(2), 31–39 (2011)
Che, S., et al.: Rodinia: a benchmark suite for heterogeneous computing. In: 2009 IEEE International Symposium on Workload Characterization (IISWC), pp. 44–54, October 2009
Fauld, J.: Unsupervised learning: association rule learning and clustering, March 2018
Graham, S.: https://earthobservatory.nasa.gov/Features/Clouds/?src=share. March 1999
Guo, P., Wang, J., Chen, Z.: A comparison of big data application programming approaches: a travel companion case study. In: 2017 IEEE International Conference on Big Data (Big Data), pp. 2869–2878 (2017)
Huang, S., Huang, J., Dai, J., Xie, T., Huang, B.: The HiBench benchmark suite: characterization of the MapReduce-based data analysis. In: 2010 IEEE 26th International Conference on Data Engineering Workshops (ICDEW 2010), pp. 41–51, March 2010
Jin, D., Oreopoulos, L., Lee, D.: Regime-based evaluation of cloudiness in CMIP5 models. Clim. Dyn. 48(1), 89–112 (2017)
Macqueen, J.: Some methods for classification and analysis of multivariate observations. In: 5-th Berkeley Symposium on Mathematical Statistics and Probability, pp. 281–297 (1967)
Polytechnic University of Milan: A tutorial on clustering algorithms: K-means clustering. https://home.deib.polimi.it/matteucc/Clustering/tutorial_html/kmeans.html. Accessed 15 June 2018
Popuri, S.K., Gobbert, M.K.: A comparative evaluation of Matlab, Octave, R, and Julia on Maya. Technical report HPCF-2017-3, UMBC High Performance Computing Facility, University of Maryland, Baltimore County (2017)
Rossow, W.B., Schiffer, R.A.: Advances in understanding clouds from ISCCP. Bull. Am. Meteorol. Soc. 80(11), 2261–2288 (1999)
Wallace, J.M.: Atmospheric Science: An Introductory Survey. Academic Press, New York (1977)
Acknowledgment
This work is supported by NSF grant with number OAC–1730250 and NASA grant 80NSSC17K0366. The hardware used the UMBC High Performance Computing Facility, which is supported by NSF grants (CNS–0821258, CNS–1228778, and OAC–1726023) and the SCREMS program (DMS–0821311), with additional substantial support from UMBC.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Barajas, C. et al. (2019). Benchmarking Parallel K-Means Cloud Type Clustering from Satellite Data. In: Zheng, C., Zhan, J. (eds) Benchmarking, Measuring, and Optimizing. Bench 2018. Lecture Notes in Computer Science(), vol 11459. Springer, Cham. https://doi.org/10.1007/978-3-030-32813-9_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-32813-9_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-32812-2
Online ISBN: 978-3-030-32813-9
eBook Packages: Computer ScienceComputer Science (R0)