Skip to main content

Big Data and HPC Convergence for Smart Infrastructures: A Review and Proposed Architecture

  • Chapter
  • First Online:
Smart Infrastructure and Applications

Abstract

The world has seen exponential data growth due to social media, mobility, E-commerce, and other factors. The issues related to avalanche of data being produced are immense and cover variety of challenges that need a careful consideration. The use of HPDA (High Performance Data Analytics) is increasing at brisk speed in many industries and has resulted in expansion of HPC market in many new territories. HPC (High Performance Computing) and big data are different systems, not only at the technical level, but also have different ecosystems. HPC systems are mainly developed for computationally intensive applications but recently data intensive applications are also among the major workload in HPC environment. Big data analytics have grown in different perspectives and have separate developer communities. As we head towards the exascale and smart infrastructure era, the necessary integration of big data and HPC is currently a hot topic of research but still at very infant stages. Both systems have different architecture and their integration brings many challenges. The aim of this work is to identify the driving forces, challenges, current and future trends associated with the integration of HPC and big data. This paper is an extension of our earlier work. We have reviewed programming models and frameworks of big data and HPC. The big data and HPC challenges in the exascale-computing era are discussed. Additional elaborations are provided on HPC and big data convergence research efforts and future directions are provided. The HPC-big data convergence architecture proposed in our earlier paper has been enhanced.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Singh, K., Kaur, R.: Hadoop: addressing challenges of big data. In: 2014 IEEE International Advance Computing Conference (IACC), pp. 686–689. IEEE (2014)

    Google Scholar 

  2. Mehmood, R., Faisal, M.A., Altowaijri, S.: Future networked healthcare systems: a review and case study. In: Boucadair, M., Jacquenet, C. (eds.) Handbook of research on redesigning the future of internet architectures, pp. 531–558. IGI Global, Hershey (2015)

    Chapter  Google Scholar 

  3. Charl, S.: IBM—HPC and HPDA for the cognitive journey with OpenPOWER. https://www-03.ibm.com/systems/power/solutions/bigdata-analytics/smartpaper/high-value-insights.html

  4. Alzahrani, S., Ikbal, M.R., Mehmood, R., Fayez, M., Katib, I.: Performance evaluation of Jacobi iterative solution for sparse linear equation system on multicore and manycore architectures. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 296–305. Springer, Cham (2018)

    Google Scholar 

  5. Alyahya, H., Mehmood, R., Katib, I.: Parallel sparse matrix vector multiplication on Intel MIC: performance analysis. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 306–322. Springer, Cham (2018)

    Google Scholar 

  6. Kwiatkowska, M., Mehmood, R.: Out-of-core solution of large linear systems of equations arising from stochastic modelling. In: Hermanns, H., S.R. (eds.) Process Algebra and Probabilistic Methods: Performance Modeling and Verification. PAPM-PROBMIV, pp. 135–151. Springer, Berlin (2002)

    Chapter  Google Scholar 

  7. Mehmood, R.: Disk-based techniques for efficient solution of large Markov chains. http://www.academia.edu/download/3361709/rashidsthesis.pdf (2004)

  8. Mehmood, R., Crowcroft, J.: Parallel iterative solution method for large sparse linear equation systems. Technical Report Number UCAM-CL-TR-650, Computer Laboratory, University of Cambridge, Cambridge, UK (2005)

    Google Scholar 

  9. Kwiatkowska, M., Mehmood, R., Norman, G., Parker, D.: A symbolic out-of-core solution method for Markov Models. Electron. Notes Theor. Comput. Sci. 68, 589–604 (2002)

    Article  Google Scholar 

  10. Mehmood, R., Lu, J.A.: Computational Markovian analysis of large systems. J. Manuf. Technol. Manag. 22, 804–817 (2011)

    Article  Google Scholar 

  11. Kwiatkowska, M., Parker, D., Yi Zhang, Y., Mehmood, R.: Dual-processor parallelisation of symbolic probabilistic model checking. In: The IEEE Computer Society’s 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, 2004 (MASCOTS 2004). Proceedings, pp. 123–130. IEEE (2004)

    Google Scholar 

  12. Mehmood, R.: Serial disk-based analysis of large stochastic models. In: Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.P., S.M. (eds.) Validation of Stochastic Systems, pp. 230–255. Springer, Berlin (2004)

    Chapter  Google Scholar 

  13. Mehmood, R., Crowcroft, J., Elmirghani, J.M.H.: A parallel implicit method for the steady-state solution of CTMCs. In: 14th IEEE International Symposium on Modeling, Analysis, and Simulation, pp. 293–302. IEEE (2006)

    Google Scholar 

  14. Keable, C: The convergence of high performance computing and big data—ascent. https://ascent.atos.net/convergence-high-performance-computing-big-data/

  15. Usman, S., Mehmood, R., Katib, I.: Big data and HPC convergence: The cutting edge and outlook. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 11–26. Springer, Cham (2018)

    Google Scholar 

  16. Padhy, R.P.: Big data processing with Hadoop-MapReduce in cloud systems. IJ-CLOSER Int. J. Cloud Comput. Serv. Sci. 2, 233–245 (2012)

    Google Scholar 

  17. Hess, K.: Hadoop vs. spark: the new age of big data. http://www.datamation.com/data-center/hadoop-vs.-spark-the-new-age-of-big-data.html

  18. Muhammad, J.: Is Apache Spark going to replace Hadoop? http://aptuz.com/blog/is-apache-spark-going-to-replace-hadoop/

  19. OLCF staff writer: OLCF group to offer spark on-demand data analysis. https://www.olcf.ornl.gov/2016/03/29/olcf-group-to-offer-spark-on-demand-data-analysis/

  20. Jost, G., Jin, H.-Q., anMey, D., Hatay, F.F.: Comparing the OpenMP, MPI, and hybrid programming paradigm on an SMP cluster. In: European Workshop on OpenMP and Applications. Aachen Germany (2003)

    Google Scholar 

  21. De Wael, M., Marr, S., De Fraine, B., Van Cutsem, T., De Meuter, W.: Partitioned global address space languages. ACM Comput. Surv. 47, 1–27 (2015)

    Article  Google Scholar 

  22. Calin, G., Derevenetc, E., Majumdar, R., Meyer, R.: A theory of partitioned global address spaces (2013)

    Google Scholar 

  23. Joseph, E., Sorensen, B.: IDC update on how big data is redefining high performance computing. https://www.tacc.utexas.edu/documents/1084364/1136739/IDC+HPDA+Briefing+slides+10.21.2014_2.pdf

  24. Geist, A., Lucas, R.: Whitepaper on the Major Computer Science Challenges at Exascale (2009)

    Article  Google Scholar 

  25. Krishnan, S., Tatineni, M., Baru, C.: myHadoop-Hadoop-on-demand on traditional HPC resources (2011)

    Google Scholar 

  26. Xuan, P., Denton, J., Ge, R., Srimani, P.K., Luo, F.: Big data analytics on traditional HPC infrastructure using two-level storage (2015)

    Google Scholar 

  27. Is Hadoop the New HPC. http://www.admin-magazine.com/HPC/Articles/Is-Hadoop-the-New-HPC

  28. Katal, A., Wazid, M., Goudar, R.H.: Big data: issues, challenges, tools and good practices. In: 2013 Sixth International Conference on Contemporary Computing (IC3), pp. 404–409. IEEE (2013)

    Google Scholar 

  29. Philip Chen, C.L., Zhang, C.Y.: Data-intensive applications, challenges, techniques and technologies: a survey on big data. Inf. Sci. (Ny). 275, 314–347 (2014)

    Article  Google Scholar 

  30. Chen, M., Mao, S., Liu, Y.: Big data: a survey. Mob. Netw. Appl. 19, 171–209 (2014)

    Article  Google Scholar 

  31. Xu, Z., Shi, Y.: Exploring big data analysis: fundamental scientific problems. Ann. Data Sci. 2(4), 363–372 (2015)

    Article  Google Scholar 

  32. Hashem, I.A.T., Yaqoob, I., Badrul Anuar, N., Mokhtar, S., Gani, A., Ullah Khan, S.: The rise of “Big Data” on cloud computing: review and open research issues. Inf. Syst. 47, 98–115 (2014)

    Article  Google Scholar 

  33. Reed, D.A., Dongarra, J.: Exascale computing and big data. Commun. ACM. 58, 56–68 (2015)

    Article  Google Scholar 

  34. Asch, M., Moore, T., Badia, R., Beck, M., Beckman, P., Bidot, T., Bodin, F., Cappello, F., Choudhary, A., de Supinski, B., Deelman, E., Dongarra, J., Dubey, A., Fox, G., Fu, H., Girona, S., Gropp, W., Heroux, M., Ishikawa, Y., Keahey, K., Keyes, D., Kramer, W., Lavignon, J.-F., Lu, Y., Matsuoka, S., Mohr, B., Reed, D., Requena, S., Saltz, J., Schulthess, T., Stevens, R., Swany, M., Szalay, A., Tang, W., Varoquaux, G., Vilotte, J.-P., Wisniewski, R., Xu, Z., Zacharov, I.: Big data and extreme-scale computing. Int. J. High Perform. Comput. Appl. 32, 435–479 (2018)

    Article  Google Scholar 

  35. Islam, N.S., Lu, X., Wasi-ur-Rahman, M., Shankar, D., Panda, D.K.: Triple-H: a hybrid approach to accelerate HDFS on HPC clusters with heterogeneous storage architecture. In: 2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 101–110. IEEE (2015)

    Google Scholar 

  36. Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating MapReduce for multi-core and multiprocessor systems. In: 2007 IEEE 13th International Symposium on High Performance Computer Architecture, pp. 13–24. IEEE (2007)

    Google Scholar 

  37. Schwan, P., Schwan, P.: Lustre: building a file system for 1000-node clusters. In: Proceedings of 2003 LINUX Symposium (2003)

    Google Scholar 

  38. Ghemawat, S., Gobioff, H., Leung, S.-T., Ghemawat, S., Gobioff, H., Leung, S.-T.: The Google file system. In: Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles—SOSP’03, p. 29. ACM Press, New York (2003)

    Google Scholar 

  39. Owre, S., Shankar, N., Rushby, J.M., Stringer-Calvert, D.W.J.: PVS System Guide (2001)

    Google Scholar 

  40. Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The Hadoop Distributed File System. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–10. IEEE (2010)

    Google Scholar 

  41. Apache Hadoop 2.9.0—C API libhdfs. https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/LibHdfs.html

  42. Martinec, J., Rango, A., Major, E.: The Snowmelt-Runoff Model (SRM) user’s manual (1983)

    Google Scholar 

  43. Rajasekar, A., Moore, R., Hou, C.-Y., Lee, C.A., Marciano, R., de Torcy, A., Wan, M., Schroeder, W., Chen, S.-Y., Gilbert, L., Tooby, P., Zhu, B.: iRODS primer: integrated rule-oriented data system. In: Synth. Lect. Inf. Concepts, Retrieval, Serv. 2, 1–143 (2010)

    Article  Google Scholar 

  44. Plimpton, S.J., Devine, K.D.: MapReduce in MPI for large-scale graph algorithms. Parallel Comput. 37, 610–632 (2011)

    Article  Google Scholar 

  45. Mantha, P.K., Luckow, A., Jha, S.: Pilot-MapReduce. In: Proceedings of Third International Workshop on MapReduce and Its Applications Date—MapReduce’12, p. 17. ACM Press, New York (2012)

    Google Scholar 

  46. Tiwari, N., Sarkar, S., Bellur, U., Indrawan, M.: An empirical study of Hadoop’s energy efficiency on a HPC cluster. Procedia Comput. Sci. 29, 62–72 (2014)

    Article  Google Scholar 

  47. Woodie, A: Does InfiniBand have a future on Hadoop? http://www.datanami.com/2015/08/04/does-infiniband-have-a-future-on-hadoop/

  48. Veiga, J., Exp, R.R., Taboada, G.L., Touri, J.: Analysis and evaluation of big data computing solutions in an HPC environment (2015)

    Google Scholar 

  49. Wang, Y., Jiao, Y., Xu, C., Li, X., Wang, T., Que, X., Cira, C., Wang, B., Liu, Z., Bailey, B., Yu, W.: Assessing the performance impact of high-speed interconnects on MapReduce. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) LNCS, vol. 8163, pp. 148–163 (2014)

    Chapter  Google Scholar 

  50. Islam, N.S., Lu, X., Wasi-ur-Rahman, M., Panda, D.K.: Can parallel replication benefit Hadoop distributed file system for high performance interconnects? In: 2013 IEEE 21st Annual Symposium on High-Performance Interconnects, pp. 75–78. IEEE (2013)

    Google Scholar 

  51. Moore, J., Chase, J., Ranganathan, P., Sharma, R.: Making scheduling cool: temperature-aware workload placement in data centers (2005)

    Google Scholar 

  52. Rajovic, N., Puzovic, N., Vilanova, L., Villavieja, C., Ramirez, A.: The low-power architecture approach towards exascale computing. In: Proceedings of the Second Workshop on Scalable Algorithms for Large-Scale Systems—ScalA’11, p. 1. ACM Press, New York (2011)

    Google Scholar 

  53. Cappello, F.: Fault tolerance in petascale/exascale systems: current knowledge, challenges and research opportunities. Int. J. High Perform. Comput. Appl. 23, 212–226 (2009)

    Article  Google Scholar 

  54. Gutierrez, D: The convergence of big data and HPC—inside BIGDATA. https://insidebigdata.com/2016/10/25/the-convergence-of-big-data-and-hpc/

  55. High performance data analytics (HPDA) market-forecast 2022. https://www.marketresearchfuture.com/reports/high-performance-data-analytics-hpda-market

  56. Willard, C.G., Snell, A., Segervall, L.: Top six predictions for HPC in 2015 (2015)

    Google Scholar 

  57. Egham: Gartner says 8.4 billion connected “things” will be in use in 2017, up 31 percent from 2016. http://www.gartner.com/newsroom/id/3598917

  58. Ahmed, W., Khan, M., Khan, A.A., Mehmood, R., Algarni, A., Albeshri, A., Katib, I.: A framework for faster porting of scientific applications between heterogeneous clouds. In: Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST, pp. 27–43. Springer, Cham (2018)

    Google Scholar 

  59. Alotaibi, S., Mehmood, R.: Big data enabled healthcare supply chain management: opportunities and challenges. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 207–215. Springer, Cham (2018)

    Google Scholar 

  60. Alamoudi, E., Mehmood, R., Albeshri, A., Gojobori, T.: DNA profiling methods and tools: a review. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 216–231. Springer, Cham (2018)

    Google Scholar 

  61. Al Shehri, W., Mehmood, R., Alayyaf, H.: A smart pain management system using big data computing. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 232–246. Springer, Cham (2018)

    Google Scholar 

  62. Khanum, A., Alvi, A., Mehmood, R.: Towards a semantically enriched computational intelligence (SECI) framework for smart farming. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 247–257. Springer, Cham (2018)

    Google Scholar 

  63. Suma, S., Mehmood, R., Albeshri, A.: Automatic event detection in smart cities using big data analytics. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 111–122. Springer, Cham (2018)

    Google Scholar 

  64. Aqib, M., Mehmood, R., Albeshri, A., Alzahrani, A.: Disaster management in smart cities by forecasting traffic plan using deep learning and GPUs. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 139–154. Springer, Cham (2018)

    Google Scholar 

  65. Alam, F., Mehmood, R., Katib, I.: D2TFRS: an object recognition method for autonomous vehicles based on RGB and spatial values of pixels. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 155–168. Springer, Cham (2018)

    Google Scholar 

  66. Muhammed, T., Mehmood, R., Albeshri, A.: Enabling reliable and resilient IoT based smart city applications. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 169–184. Springer, Cham (2018)

    Google Scholar 

  67. Al-Dhubhani, R., Mehmood, R., Katib, I., Algarni, A.: Location privacy in smart cities era. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 123–138. Springer, Cham (2018)

    Google Scholar 

  68. Alomari, E., Mehmood, R.: Analysis of Tweets in Arabic Language for detection of road traffic conditions. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 98–110. Springer, Cham (2018)

    Google Scholar 

  69. Arfat, Y., Aqib, M., Mehmood, R., Albeshri, A., Katib, I., Albogami, N., Alzahrani, A.: Enabling smarter societies through mobile big data fogs and clouds. Procedia Comput. Sci. 109, 1128–1133 (2017)

    Article  Google Scholar 

  70. Schlingensiepen, J., Nemtanu, F., Mehmood, R., McCluskey, L.: Autonomic transport management systems—enabler for smart cities, personalized medicine, participation and industry grid/industry 4.0. In: Intelligent Transportation Systems—Problems and Perspectives, pp. 3–35. Springer, Cham (2016)

    Google Scholar 

  71. Arfat, Y., Mehmood, R., Albeshri, A.: Parallel shortest path graph computations of United States Road Network Data on Apache Spark. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 323–336. Springer, Cham (2018)

    Google Scholar 

  72. El Baz, D.: IoT and the need for high performance computing. In: 2014 International Conference on Identification, Information and Knowledge in the Internet of Things, pp. 1–6. IEEE (2014)

    Google Scholar 

  73. Conway, S: High performance data analysis (HPDA): HPC—big data convergence—insideHPC (2017)

    Google Scholar 

  74. Keutzer, K., Tim, M.: Our pattern language_our pattern language, file:///Users/abdulmanan/Desktop/Our Pattern Language_Our Pattern Language.htm (2016)

    Google Scholar 

  75. Bodkin, R., Bodkin, R.: Big data patterns, pp. 1–23 (2017)

    Google Scholar 

  76. Mysore, D., Khupat, S., Jain, S.: Big data architecture and patterns, Part 1: introduction to big data classification and architecture. https://www.ibm.com/developerworks/library/bd-archpatterns1/index.html

Download references

Acknowledgments

The authors acknowledge with thanks the technical and financial support from the Deanship of Scientific Research (DSR) at the King Abdul-Aziz University (KAU), Jeddah, Saudi Arabia, under the grant number G-673-793-38. The work carried out in this paper is supported by the HPC Center at the King Abdul-Aziz University.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Sardar Usman .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Usman, S., Mehmood, R., Katib, I. (2020). Big Data and HPC Convergence for Smart Infrastructures: A Review and Proposed Architecture. In: Mehmood, R., See, S., Katib, I., Chlamtac, I. (eds) Smart Infrastructure and Applications. EAI/Springer Innovations in Communication and Computing. Springer, Cham. https://doi.org/10.1007/978-3-030-13705-2_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-13705-2_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-13704-5

  • Online ISBN: 978-3-030-13705-2

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics