Real-Time Big Data Stream Processing Using GPU with Spark Over Hadoop Ecosystem

  • M. Mazhar Rathore
  • Hojae Son
  • Awais Ahmad
  • Anand Paul
  • Gwanggil Jeon
Part of the following topical collections:
  1. Special Issue on Programming Models and Algorithms for Data Analysis in HPC Systems


In this technological era, every person, authorities, entrepreneurs, businesses, and many things around us are connected to the internet, forming Internet of thing (IoT). This generates a massive amount of diverse data with very high-speed, termed as big data. However, this data is very useful that can be used as an asset for the businesses, organizations, and authorities to predict future in various aspects. However, efficiently processing Big Data while making real-time decisions is a quite challenging task. Some of the tools like Hadoop are used for Big Datasets processing. On the other hand, these tools could not perform well in the case of real-time high-speed stream processing. Therefore, in this paper, we proposed an efficient and real-time Big Data stream processing approach while mapping Hadoop MapReduce equivalent mechanism on graphics processing units (GPUs). We integrated a parallel and distributed environment of Hadoop ecosystem and a real-time streaming processing tool, i.e., Spark with GPU to make the system more powerful in order to handle the overwhelming amount of high-speed streaming. We designed a MapReduce equivalent algorithm for GPUs for a statistical parameter calculation by dividing overall Big Data files into fixed-size blocks. Finally, the system is evaluated while considering the efficiency aspect (processing time and throughput) using (1) large-size city traffic video data captured by static as well as moving vehicles’ cameras while identifying vehicles and (2) large text-based files, like twitter data files, structural data, etc. Results show that the proposed system working with Spark on top and GPUs under the parallel and distributed environment of Hadoop ecosystem is more efficient and real-time as compared to existing standalone CPU-based MapReduce implementation.


Big Data Hadoop Spark GPU MapReduce 



This study was supported by the Brain Korea 21 Plus project (SW Human Resource Development Program for Supporting Smart Life) funded by Ministry of Education, School of Computer Science and Engineering, Kyungpook National University, Korea (21A20131600005).


  1. 1.
    Cohen, J., Dolan, B., Dunlap, M., Hellerstein, J.M., Welton, C.: Mad skills: new analysis practices for Big Data. Proc. VLDB Endow. 2(2), 1481–1492 (2009)CrossRefGoogle Scholar
  2. 2.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  3. 3.
    IBM, Armonk, NY, USA.: Four Vendor Views on Big Data and Big Data Analytics. IBM [Online]. (2012)
  4. 4.
    CISCO.: The Internet of Things, Infographic. (2015)
  5. 5.
    Sivaraman, S., Trivedi, M.M.: Integrated lane and vehicle detection, localization, and tracking: a synergistic approach. IEEE Trans. Intell. Transp. Syst. 14(2), 906–917 (2013)CrossRefGoogle Scholar
  6. 6.
    Rathore, M.M., Ahmad, A., Paul, A., Jeon, G.: Efficient graph-oriented smart transportation using internet of things generated Big Data. In: 2015 11th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), pp. 512–519 (2015)Google Scholar
  7. 7.
    Ahmad, A., Paul, A., Rathore, M.M., Chang, H.: Smart cyber society: integration of capillary devices with high usability based on cyber-physical system. Future Gen. Comput. Syst. 56, 493–503 (2016)CrossRefGoogle Scholar
  8. 8.
    Rathore, M.M., Ahmad, A., Paul, A., Wan, J., Daqiang, Z.: Real-time medical emergency response system: exploiting IoT and Big Data for public health. J. Med. Syst. 40(12), 283 (2016)CrossRefGoogle Scholar
  9. 9.
    Rathore, M.M., Ahmad, A., Paul, A., Rho, S.: Urban planning and building smart cities based on the internet of things using Big Data analytics. Comput. Netw. 101, 63–80 (2016)CrossRefGoogle Scholar
  10. 10.
    Ahmad, A., Paul, A., Rathore, M.M.: An efficient divide-and-conquer approach for Big Data analytics in machine-to-machine communication. Neurocomputing 174, 439–453 (2016)CrossRefGoogle Scholar
  11. 11.
    Jin, J., Gubbi, J., Marusic, S., Palaniswami, M.: An information framework for creating a smart city through internet of things. IEEE Internet Things J. 1(2), 112–121 (2014)CrossRefGoogle Scholar
  12. 12.
    Apache Hadoop.: Welcome to Apache™ Hadoop®!. (2016). Accessed 1 Nov 2016
  13. 13.
    Apache SPARK.: Apache Spark™. (2016). Accessed 1 Nov 2016
  14. 14.
    Ailamaki, A., Govindaraju, N.K., Harizopoulos, S., Manocha, D.: Query co-processing on commodity processors. VLDB 6, 1267–1267 (2006)Google Scholar
  15. 15.
    Hadoop.: (2010). Accessed 1 Nov 2016
  16. 16.
    Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating mapreduce for multi-core and multiprocessor systems. In: IEEE 13th International Symposium on High Performance Computer Architecture 2007. HPCA 2007, pp. 13–24 (2007)Google Scholar
  17. 17.
    Cerotti, D., et al.: Modeling and analysis of performances for concurrent multithread applications on multicore and graphics processing unit systems. Concurr. Comput. Pract. Exp. 28(2), 438–452 (2016)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Qureshi, M.K., Patt, Y.N.: Utility-based cache partitioning: a low-overhead, high-performance, runtime mechanism to partition shared caches. In: Microarchitecture. 2006. MICRO-39. 39th Annual IEEE/ACM International Symposium on IEEE (2006)Google Scholar
  19. 19.
    Kavadias, S.G. et al.: On-chip communication and synchronization mechanisms with cache-integrated network interfaces. In: Proceedings of the 7th ACM International Conference on Computing Frontiers. ACM (2010)Google Scholar
  20. 20.
    Liu, F., Xiaowei J., Solihin, Y.: Understanding how off-chip memory bandwidth partitioning in chip multiprocessors affects system performance. In: High Performance Computer Architecture (HPCA). 2010 IEEE 16th International Symposium on IEEE (2010)Google Scholar
  21. 21.
    D’Amore, L., et al.: HPC computation issues of the incremental 3D variational data assimilation scheme in OceanVar software. J. Numer. Anal. Ind. Appl. Math. 7(3–4), 91–105 (2012)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Che, S., et al.: A performance study of general-purpose applications on graphics processors using CUDA. J. Parallel Distrib. Comput. 68(10), 1370–1380 (2008)CrossRefGoogle Scholar
  23. 23.
    Owens, J.D., et al.: GPU computing. Proc. IEEE 96(5), 879–899 (2008)CrossRefGoogle Scholar
  24. 24.
    Gregg, C., Hazelwood K.: Where is the data? Why you cannot debate CPU versus GPU performance without the answer. In: Performance Analysis of Systems and Software (ISPASS), 2011 IEEE International Symposium on IEEE (2011)Google Scholar
  25. 25.
    Shi, L., et al.: vCUDA: GPU-accelerated high-performance computing in virtual machines. IEEE Trans. Comput. 61(6), 804–816 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Aldinucci, M., et al.: Parallel visual data restoration on multi-GPGPUs using stencil-reduce pattern. Int. J. High Perform. Comput. Appl. 29(4), 461–472 (2015)CrossRefGoogle Scholar
  27. 27.
    Wu, W., et al.: Hierarchical dag scheduling for hybrid distributed systems. In: Parallel and Distributed Processing Symposium (IPDPS), 2015 International IEEE (2015)Google Scholar
  28. 28.
    Song, F., Dongarra, J.: A scalable approach to solving dense linear algebra problems on hybrid CPU-GPU systems. Concurr. Comput. Pract. Exp. 27(14), 3702–3723 (2015)CrossRefGoogle Scholar
  29. 29.
    Du, P., et al.: Soft error resilient QR factorization for hybrid system with GPGPU. J. Comput. Sci. 4(6), 457–464 (2013)CrossRefGoogle Scholar
  30. 30.
    Dongarra, J., et al.: Hpc programming on intel many-integrated-core hardware with magma port to xeon phi. Sci. Program. 2015, 9 (2015)Google Scholar
  31. 31.
    Braun, T.D., et al.: A comparison of eleven static heuristics for mapping a class of independent tasks onto heterogeneous distributed computing systems. J. Parallel Distrib. Comput. 61(6), 810–837 (2001)CrossRefzbMATHGoogle Scholar
  32. 32.
    Anderson, E., et al.: LAPACK Users’ guide. In: Society for Industrial and Applied Mathematics (1999)Google Scholar
  33. 33.
    Anderson, E., Bai, Z., Bischof, C., Blackford, S., Demmel, J., Dongarra, J., Du Croz, J., Greenbaum, A., Hammarling, S., McKenney, A., Sorensen, D.: LAPACK Users’ Guide, 3rd edn. SIAM, Philadelphia (1999)CrossRefzbMATHGoogle Scholar
  34. 34.
    Agullo, E., Dongarra, J., Hadri, B., Kurzak, J., Langou, J., Langou, J., Ltaief, H., Luszczek, P., YarKhan, A.: Plasma Users’ Guide, Technical report. In: ICL, UTK (2014)Google Scholar
  35. 35.
    Blackford, L.S., Choi, J., Cleary, A., D’Azeuedo, E., Demmel, J., Dhillon, I., Hammarling, S., Henry, G., Petitet, A., Stanley, K., Walker, D., Whaley, R.C.: ScaLAPACK User’s Guide. In: Society for Industrial and Applied Mathematics, Philadelphia (1997)Google Scholar
  36. 36.
    Song, F., YarKhan, A., Dongarra, J.: Dynamic task scheduling for linear algebra algorithms on distributed-memory multicore systems. In: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis. pp. 1–11 (2009)Google Scholar
  37. 37.
    Ahmad, A., et al.: Multilevel data processing using parallel algorithms for analyzing Big Data in high-performance computing. Int. J. Parallel Program. doi: 10.1007/s10766-017-0498-x (2017)
  38. 38.
    Rathore, M.M., et al.: Exploiting encrypted and tunneled multimedia calls in high-speed Big Data environment. Multimed. Tools Appl. doi: 10.1007/s11042-017-4393-7 (2017)
  39. 39.
    NVIDIA ACCELERATED COMPUTING.: CUDA Toolkit 8.0. (2016). Accessed 1 Nov 2016
  40. 40.
    Dean, J., Ghemawat, S.: Mapreduce: Simplified data processing on large clusters. In: Proceedings of Sixth Conference Symposium on Opearting Systems Design and Implementation (OSDI) (2004)Google Scholar
  41. 41. Live traffic cameras. (2016). Accessed 1 Nov 2016
  42. 42.
    43Earth Cam.: LIVE Webcam Network. (2016). Accessed 1 Nov 2016

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  • M. Mazhar Rathore
    • 1
  • Hojae Son
    • 1
  • Awais Ahmad
    • 2
  • Anand Paul
    • 1
  • Gwanggil Jeon
    • 3
  1. 1.School of Computer Science and EngineeringKyungpook National UniversityDaeguKorea
  2. 2.Department of Information and Communication EngineeringYeungnam UniversityGyeongbukKorea
  3. 3.Department of Embedded Systems EngineeringIncheon National UniversityIncheonKorea

Personalised recommendations