Abstract
Meteorology is a branch of science which can be leveraged to gain useful insight into many phenomenon that have significant impacts on our daily lives such as weather precipitation, cyclones, thunderstorms, climate change. It is a highly data-driven field that involves large datasets of images captured from both radar and satellite, thus requiring efficient technologies for storing, processing and data mining to find hidden patterns in these datasets. Different big data tools and ecosystems, most of them integrating Hadoop and Spark, have been designed to address big data issues. However, despite its importance, only few works have been done on the application of these tools and ecosystems for solving meteorology issues. This paper proposes and evaluate the performance of a precipitation data processing system that builds upon the Cloudera ecosystem to analyse large datasets of images as a classification problem. The system can be used as a replacement to machine learning techniques when the classification problem consists of finding zones of high, moderate and low precipitations in satellite images.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
GmbH, J.: Joint Aviation Authorities Airline Transport Pilot’s Licence Theoretical Knowledge Manual. Oxford Aviation Training (2001)
Ahrens, C.D.: Meteorology Today: An Introduction to Weather, Climate, and the Environment. Cengage Learning, Boston (2012)
Swails, B., Berlinger, J.: Tropical cyclone kenneth death toll rises to 38 in mozambique, officials say (2019)
Shi, E., Li, Q., Gu, D., Zhao, Z.: A method of weather radar echo extrapolation based on convolutional neural networks. In: Schoeffmann, K., et al. (eds.) MMM 2018. LNCS, vol. 10704, pp. 16–28. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73603-7_2
Kamilaris, A., Prenafeta-Boldú, F.X.: Deep learning in agriculture: a survey. Comput. Electron. Agric. 147, 70–90 (2018)
Al-Jarrah, O.Y., Yoo, P.D., Muhaidat, S., Karagiannidis, G.K., Taha, K.: Efficient machine learning for big data: a review. Big Data Res. 2(3), 87–93 (2015)
Dagade, V., Lagali, M., Avadhani, S., Kalekar, P.: Big data weather analytics using hadoop. Int. J. Emerg. Technol. Comput. Sci. Electron. (IJETCSE) ISSN, 0976–1353 (2015)
Chen, C.P., Zhang, C.-Y.: Data-intensive applications, challenges, techniques and technologies: a survey on big data. Inf. Sci. 275, 314–347 (2014)
Ibrahim, G., et al.: Big data techniques: hadoop and mapreduce for weather forecasting. Int. J. Latest Trends Eng. Technol. 194–199 (2016)
Pandey, A., Agrawal, C., Agrawal, M.: A hadoop based weather prediction model for classification of weather data. In: 2017 Second International Conference on Electrical, Computer and Communication Technologies (ICECCT), pp. 1–5. IEEE (2017)
Riyaz, P., Varghese, S.M.: Leveraging map reduce with hadoop for weather data analytics. J. Comput. Eng. 17(3), 6–12 (2015)
Oury, D.T.M., Singh, A.: Data analysis of weather data using hadoop technology. In: Satapathy, S.C., Bhateja, V., Das, S. (eds.) Smart Computing and Informatics. SIST, vol. 77, pp. 723–730. Springer, Singapore (2018). https://doi.org/10.1007/978-981-10-5544-7_71
Jayanthi, D., Sumathi, G.: Weather data analysis using spark-an in-memory computing framework. In: 2017 Innovations in Power and Advanced Computing Technologies (i-PACT), pp. 1–5. IEEE (2017)
White, T.: Hadoop: The Definitive Guide. O’Reilly Media, Inc., Newton (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Hussein, E., Sadiki, R., Jafta, Y., Sungay, M.M., Ajayi, O., Bagula, A. (2020). Big Data Processing Using Hadoop and Spark: The Case of Meteorology Data. In: Zitouni, R., Agueh, M., Houngue, P., Soude, H. (eds) e-Infrastructure and e-Services for Developing Countries. AFRICOMM 2019. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 311. Springer, Cham. https://doi.org/10.1007/978-3-030-41593-8_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-41593-8_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-41592-1
Online ISBN: 978-3-030-41593-8
eBook Packages: Computer ScienceComputer Science (R0)