Advertisement

MapReduce Hadoop Models for Distributed Neural Network Processing of Big Data Using Cloud Services

  • Natalia Axak
  • Mykola KorablyovEmail author
  • Dmytro Rosinskiy
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1080)

Abstract

The paper proposes a formalization process of Big Data distributed intelligent processing using Cloud-Fog-Dew architecture. This process provides specialized services, based on continuous support of experts in areas of concern, and advisory support for their actions in diagnostically complex cases. The method of Big Data processing based on neural networks is considered, which is distinguished by dynamic redistribution of work between computers, that allows to uniformly load the computing cluster with different data topologies. Proposed method is less by one order of computational complexity and less time spent. The MapReduce Hadoop models for distributed neural network processing of Big Data were proposed, characterized by the adaptation of data topology to the corresponding architectural computer cluster. This reduces the amount of information transmitted between nodes to increase productivity in solving complex tasks and effectively balancing the load of computing resources with different data topologies. An experimental Hadoop cluster was created to evaluate the performance of developed models for Big Data distributed processing. It allows for the implementation of parallel learning procedures for multilayer neural networks based on “star” and “fully connected graph” data topology with different amounts of input data.

Keywords

Cloud-Fog-Dew architecture Distributed processing MapReduce Hadoop Model Neural network Service-oriented system 

References

  1. 1.
    Marjani, M.: Big IoT data analytics: architecture, opportunities, and open research chasllenges. IEEE Access 5, 5247–5261 (2017)CrossRefGoogle Scholar
  2. 2.
    Nikolaychuk, Ya.M.: Specialized computer technologies in informatics: monography. In: Nikolaychuk, Ya.M. (ed.) Beskydy, Ternopil, 919 p. (2017). (in ukrainian)Google Scholar
  3. 3.
    Haykin, S.: Neural Networks and Learning Machines, 3rd edn., 937 p. Pearson Education (2009)Google Scholar
  4. 4.
    Bogdanova, V.G.: Multiagent approach to controlling distributed computing in a cluster grid system. J. Comput. Syst. Sci. Int. 53, 713–722 (2014)CrossRefGoogle Scholar
  5. 5.
    Lützenberger, M.: Multi-agent system in practice: when research meets reality. In: Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems. International Foundation for Autonomous Agents and Multiagent Systems, pp. 796–805 (2016)Google Scholar
  6. 6.
    Fernandes, L.M., O’Connor, M., Weaver, V.: Big data, bigger outcomes. J. AHIMA 83(10), 38–43 (2012)Google Scholar
  7. 7.
    Patibandla, R.S.M.L., Veeranjaneyulu, N.: Survey on clustering algorithms for unstructured data. In: Intelligent Engineering Informatics, pp. 421–429. Springer, Singapore (2018)Google Scholar
  8. 8.
    Shirkhorshidi, A.S.: Big data clustering: a review. In: International Conference on Computational Science and Its Applications, pp. 707–720. Springer, Cham (2014)Google Scholar
  9. 9.
    Chen, H.: Smart health and wellbeing [Trends & Controversies]. IEEE Intell. Syst. 26(5), 78–90 (2011)CrossRefGoogle Scholar
  10. 10.
    Axak, N., Rosinskiy, D., Barkovska O., Novoseltsev, I.: Cloud-fog-dew architecture for personalized service-oriented systems. In: The 9th IEEE International Conference on Dependable Systems, Services and Technologies, DESSERT 2018, Kyiv, pp. 80–84 (2018)Google Scholar
  11. 11.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  12. 12.
    Ghemawat, S., Gobioff, H., Leung, ST.: The Googlefile system. SIGOPS Oper. Syst. Rev. 37, 29–43 (2003)Google Scholar
  13. 13.
    Axak, N.G., Lebyodkina, A.Yu.: Method of uniform distribution of parallel operatrions for the multy-layer neural network accellerated learning based on different data topologies. Control. Navig. Commun. Syst. 2(18), 66–73 (2011). (in Russian)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Kharkiv National University of Radio ElectronicsKharkivUkraine

Personalised recommendations