The state of the art and taxonomy of big data analytics: view from new big data framework

  • Azlinah Mohamed
  • Maryam Khanian NajafabadiEmail author
  • Yap Bee Wah
  • Ezzatul Akmal Kamaru Zaman
  • Ruhaila Maskat


Big data has become a significant research area due to the birth of enormous data generated from various sources like social media, internet of things and multimedia applications. Big data has played critical role in many decision makings and forecasting domains such as recommendation systems, business analysis, healthcare, web display advertising, clinicians, transportation, fraud detection and tourism marketing. The rapid development of various big data tools such as Hadoop, Storm, Spark, Flink, Kafka and Pig in research and industrial communities has allowed the huge number of data to be distributed, communicated and processed. Big data applications use big data analytics techniques to efficiently analyze large amounts of data. However, choosing the suitable big data tools based on batch and stream data processing and analytics techniques for development a big data system are difficult due to the challenges in processing and applying big data. Practitioners and researchers who are developing big data systems have inadequate information about the current technology and requirement concerning the big data platform. Hence, the strengths and weaknesses of big data technologies and effective solutions for Big Data challenges are needed to be discussed. Hence, due to that, this paper presents a review of the literature that analyzes the use of big data tools and big data analytics techniques in areas like health and medical care, social networking and internet, government and public sector, natural resource management, economic and business sector. The goals of this paper are to (1) understand the trend of big data-related research and current frames of big data technologies; (2) identify trends in the use or research of big data tools based on batch and stream processing and big data analytics techniques; (3) assist and provide new researchers and practitioners to place new research activity in this domain appropriately. The findings of this study will provide insights and knowledge on the existing big data platforms and their application domains, the advantages and disadvantages of big data tools, big data analytics techniques and their use, and new research opportunities in future development of big data systems.


Parallel and distributed computing Big data tools Big data analytics techniques Domain area 



This work is supported under the university Research Entity Initiatives Grant (600-RMI/DANA 5/3/REI (16/2015)). We thank IRMI (Institute of Research, Management and Innovation), UiTM for their continuous support.


  1. Agerri R, Artola X, Beloki Z, Rigau G, Soroa A (2015) Big data for natural language processing: a streaming approach. Knowl-Based Syst 79:36–42CrossRefGoogle Scholar
  2. Ahmad A, Paul A, Rathore MM (2016) An efficient divide-and-conquer approach for big data analytics in machine-to-machine communication. Neurocomputing 174:439–453CrossRefGoogle Scholar
  3. Ahmad A, Khan M, Paul A, Din S, Rathore MM, Jeon G, Choi GS (2017) Toward modeling and optimization of features selection in big data based social internet of things. Future Gener Comput SystGoogle Scholar
  4. Ai W, Li K, Li K (2017) An effective hot topic detection method for microblog on spark. Appl Soft ComputGoogle Scholar
  5. Amato F, Moscato V, Picariello A, Piccialli F (2017) SOS: a multimedia recommender system for online social networks. Future Gener Comput SystGoogle Scholar
  6. Apiletti D, Baralis E, Cerquitelli T, Garza P, Pulvirenti F, Michiardi P (2017) A parallel MapReduce algorithm to efficiently support itemset mining on high dimensional data. Big Data Res 10:53–69CrossRefGoogle Scholar
  7. Arias J, Gamez JA, Puerta JM (2017) Learning distributed discrete Bayesian network classifiers under MapReduce with Apache spark. Knowl-Based Syst 117:16–26CrossRefGoogle Scholar
  8. Aufaure MA, Chiky R, Curé O, Khrouf H, Kepeklian G (2016) From business intelligence to semantic data stream management. Future Gener Comput Syst 63:100–107CrossRefGoogle Scholar
  9. Babar M, Arif F (2017) Smart urban planning using big data analytics to contend with the interoperability in Internet of Things. Future Gener Comput Syst 77:65–76CrossRefGoogle Scholar
  10. Barba-González C, García-Nieto J, Nebro AJ, Cordero JA, Durillo JJ, Navas-Delgado I, Aldana-Montes JF (2017) jMetalSP: a framework for dynamic multi-objective big data optimization. Appl Soft ComputGoogle Scholar
  11. Basanta-Val P, Fernández-García N, Wellings AJ, Audsley NC (2015) Improving the predictability of distributed stream processors. Future Gener Comput Syst 52:22–36CrossRefGoogle Scholar
  12. Basanta-Val, P., Fernández-García, N., & Sánchez-Fernández, L. (2017). Predictable remote invocations for distributed stream processing. Future Gener Comput SystGoogle Scholar
  13. Batarseh FA, Latif EA (2016) Assessing the quality of service using big data analytics: with application to healthcare. Big Data Res 4:13–24CrossRefGoogle Scholar
  14. Bechini A, Marcelloni F, Segatori A (2016) A MapReduce solution for associative classification of big data. Inf Sci 332:33–55CrossRefGoogle Scholar
  15. Bei Z, Yu Z, Luo N, Jiang C, Xu C, Feng S (2018) Configuring in-memory cluster computing using random forest. Future Gener Comput Syst 79:1–15CrossRefGoogle Scholar
  16. Bharti SK, Vachha B, Pradhan RK, Babu KS, Jena SK (2016) Sarcastic sentiment detection in tweets streamed in real time: a big data approach. Digital Commun Netw 2(3):108–121CrossRefGoogle Scholar
  17. Carcillo F, Dal Pozzolo A, Le Borgne YA, Caelen O, Mazzer Y, Bontempi G (2018) Scarff: a scalable framework for streaming credit card fraud detection with spark. Inf Fusion 41:182–194CrossRefGoogle Scholar
  18. Castiglione A, Colace F, Moscato V, Palmieri F (2017) CHIS: a big data infrastructure to manage digital cultural items. Future Gener Comput SystGoogle Scholar
  19. Chen CP, Zhang CY (2014) Data-intensive applications, challenges, techniques and technologies: a survey on big data. Inf Sci 275:314–347CrossRefGoogle Scholar
  20. Chen G, Wu S, Wang Y (2015) The evolvement of big data systems: from the perspective of an information security application, 65–73Google Scholar
  21. Chen H, Li T, Cai Y, Luo C, Fujita H (2016) Parallel attribute reduction in dominance-based neighborhood rough set. Inf Sci 373:351–368CrossRefGoogle Scholar
  22. Chen Y, Crespi N, Ortiz AM, Shu L (2017) Reality mining: a prediction algorithm for disease dynamics based on mobile big data. Inf Sci 379:82–93CrossRefGoogle Scholar
  23. De Maio C, Fenza G, Loia V, Orciuoli F (2017) Distributed online temporal fuzzy concept analysis for stream processing in smart cities. J Parallel Distrib Comput 110:31–41CrossRefzbMATHGoogle Scholar
  24. Del Río S, López V, Benítez JM, Herrera F (2014) On the use of MapReduce for imbalanced big data using random forest. Inf Sci 285:112–137CrossRefGoogle Scholar
  25. Ding L, Liu Y, Han B, Zhang S, Song B (2017) HB-file: an efficient and effective high-dimensional big data storage structure based on US-ELM. Neurocomputing 261:184–192CrossRefGoogle Scholar
  26. Eiras-Franco C, Bolón-Canedo V, Ramos S, González-Domínguez J, Alonso-Betanzos A, Touriño J (2016) Multithreaded and spark parallelization of feature selection filters. J Comput Sci 17:609–619CrossRefGoogle Scholar
  27. Elkano M, Galar M, Sanz J, Bustince H (2017) CHI-BD: a fuzzy rule-based classification system for big data classification problems. Fuzzy Sets SystGoogle Scholar
  28. Elsebakhi E, Lee F, Schendel E, Haque A, Kathireason N, Pathare T, Al-Ali R (2015) Large-scale machine learning based on functional networks for biomedical big data with high performance computing platforms. J Comput Sci 11:69–81MathSciNetCrossRefGoogle Scholar
  29. Fernández-Rodríguez JY, Álvarez-García JA, Fisteus JA, Luaces MR, Magaña VC (2017) Benchmarking real-time vehicle data streaming models for a Smart City. Inf Syst 72:62–76CrossRefGoogle Scholar
  30. Ferranti A, Marcelloni F, Segatori A, Antonelli M, Ducange P (2017) A distributed approach to multi-objective evolutionary generation of fuzzy rule-based classifiers from big data. Inf Sci 415:319–340CrossRefGoogle Scholar
  31. Fonseca A, Cabral B (2017) Prototyping a GPGPU neural network for deep-learning big data analysis. Big Data Res 8:50–56CrossRefGoogle Scholar
  32. Gadiraju KK, Verma M, Davis KC, Talaga PG (2016) Benchmarking performance for migrating a relational application to a parallel implementation. Future Gener Comput Syst 63:148–156CrossRefGoogle Scholar
  33. Genuer R, Poggi JM, Tuleau-Malot C, Villa-Vialaneix N (2017) Random forests for big data. Big Data Res 9:28–46CrossRefGoogle Scholar
  34. Ghadiri N, Ghaffari M, Nikbakht MA (2016) BigFCM: fast, precise and scalable FCM on Hadoop. arXiv preprint arXiv:1605.03047
  35. Guo J, Song B, Yu FR, Yan Z, Yang LT (2017) Object detection among multimedia big data in the compressive measurement domain under mobile distributed architecture. Future Gener Comput Syst 76:519–527CrossRefGoogle Scholar
  36. He W, Wu H, Yan G, Akula V, Shen J (2015) A novel social media competitive analytics framework with sentiment benchmarks. Inf Manag 52(7):801–812CrossRefGoogle Scholar
  37. Hernández ÁB, Perez MS, Gupta S, Muntés-Mulero V (2017) Using machine learning to optimize parallelism in big data applications. Future Gener Comput SystGoogle Scholar
  38. Hidalgo N, Wladdimiro D, Rosas E (2017) Self-adaptive processing graph with operator fission for elastic stream processing. J Syst Softw 127:205–216CrossRefGoogle Scholar
  39. Higashino WA, Capretz MA, Bittencourt LF (2016) CEPSim: modelling and simulation of complex event processing systems in cloud environments. Future Gener Comput Syst 65:122–139CrossRefGoogle Scholar
  40. Huang S, Wang B, Qiu J, Yao J, Wang G, Yu G (2016) Parallel ensemble of online sequential extreme learning machine based on map reduce. Neurocomputing 174:352–367CrossRefGoogle Scholar
  41. Huang CS, Tsai MF, Huang PH, Su LD, Lee KS (2017) Distributed asteroid discovery system for large astronomical data. J Netw Comput Appl 93:27–37CrossRefGoogle Scholar
  42. Iqbal R, Doctor F, More B, Mahmud S, Yousuf U (2017) Big data analytics and computational intelligence for cyber–physical systems: recent trends and state of the art applications. Future Gener Comput SystGoogle Scholar
  43. Jayasena KPN, Li L, Xie Q (2017) Multi-modal multimedia big data analyzing architecture and resource allocation on cloud platform. Neurocomputing 253:135–143CrossRefGoogle Scholar
  44. Jiang R, Lu R, Choo KKR (2018) Achieving high performance and privacy-preserving query over encrypted multidimensional big metering data. Future Gener Comput Syst 78:392–401CrossRefGoogle Scholar
  45. Karunaratne P, Karunasekera S, Harwood A (2017) Distributed stream clustering using micro-clusters on Apache Storm. J Parallel Distrib Comput 108:74–84CrossRefGoogle Scholar
  46. Kousiouris G, Akbar A, Sancho J, Ta-shma P, Psychas A, Kyriazis D, Varvarigou T (2018) An integrated information lifecycle management framework for exploiting social network data to identify dynamic large crowd concentration events in smart cities applications. Future Gener Comput Syst 78:516–530CrossRefGoogle Scholar
  47. Kovalchuk SV, Krotov E, Smirnov PA, Nasonov DA, Yakovlev AN (2018) Distributed data-driven platform for urgent decision making in cardiological ambulance control. Future Gener Comput Syst 79:144–154CrossRefGoogle Scholar
  48. Kranjc J, Orač R, Podpečan V, Lavrač N, Robnik-Šikonja M (2017) ClowdFlows: online workflows for distributed big data mining. Future Gener Comput Syst 68:38–58CrossRefGoogle Scholar
  49. Kumar M, Rath SK (2015) Classification of microarray using MapReduce based proximal support vector machine classifier. Knowl-Based Syst 89:584–602CrossRefGoogle Scholar
  50. Liang Y, Wu D, Liu G, Li Y, Gao C, Ma ZJ, Wu W (2016) Big data-enabled multiscale serviceability analysis for aging bridges. Digit Commun Netw 2(3):97–107CrossRefGoogle Scholar
  51. Lin W, Dou W, Zhou Z, Liu C (2015) A cloud-based framework for home-diagnosis service over big medical data. J Syst Softw 102:192–206CrossRefGoogle Scholar
  52. Maillo J, Ramírez S, Triguero I, Herrera F (2017) kNN-IS: an iterative spark-based design of the k-nearest neighbors classifier for big data. Knowl-Based Syst 117:3–15CrossRefGoogle Scholar
  53. Manco G, Ritacco E, Rullo P, Gallucci L, Astill W, Kimber D, Antonelli M (2017) Fault detection and explanation through big data analysis on sensor streams. Expert Syst Appl 87:141–156CrossRefGoogle Scholar
  54. Manogaran G, Varatharajan R, Lopez D, Kumar PM, Sundarasekar R, Thota C (2017) A new architecture of internet of things and big data ecosystem for secured smart healthcare monitoring and alerting system. Future Gener Comput SystGoogle Scholar
  55. Maté A, Peral J, Ferrández A, Gil D, Trujillo J (2016) A hybrid integrated architecture for energy consumption prediction. Future Gener Comput Syst 63:131–147CrossRefGoogle Scholar
  56. Mavridis I, Karatza H (2017) Performance evaluation of cloud-based log file analysis with Apache Hadoop and Apache Spark. J Syst Softw 125:133–151CrossRefGoogle Scholar
  57. Mestre DG, Pires CES, Nascimento DC (2017) Towards the efficient parallelization of multi-pass adaptive blocking for entity matching. J Parallel Distrib Comput 101:27–40CrossRefGoogle Scholar
  58. Mohapatra SK, Sahoo PK, Wu SL (2016a) Big data analytic architecture for intruder detection in heterogeneous wireless sensor networks. J Netw Comput Appl 66:236–249CrossRefGoogle Scholar
  59. Mohapatra SK, Sahoo PK, Wu SL (2016b) Big data analytic architecture for intruder detection in heterogeneous wireless sensor networks. J Netw Comput Appl 66:236–249CrossRefGoogle Scholar
  60. Nair LR, Shetty SD, Shetty SD (2017) Applying spark based machine learning model on streaming big data for health status prediction. Comput Electr EngGoogle Scholar
  61. Najafabadi MK, Mahrin MNR (2016) A systematic literature review on the state of research and practice of collaborative filtering technique and implicit feedback. Artif Intell Rev 45(2):167–201CrossRefGoogle Scholar
  62. Najafabadi MK, Mohamed AH, Mahrin MNR (2017) A survey on data mining techniques in recommender systems. Soft Comput, 1–28Google Scholar
  63. Najafabadi MK, Mahrin MNR, Chuprat S, Sarkan HM (2017b) Improving the accuracy of collaborative filtering recommendations using clustering and association rules mining on implicit data. Comput Hum Behav 67:113–128CrossRefGoogle Scholar
  64. Nghiem PP, Figueira SM (2016) Towards efficient resource provisioning in MapReduce. J Parallel Distrib Comput 95:29–41CrossRefGoogle Scholar
  65. Nguyen T, Larsen ME, O’Dea B, Nguyen DT, Yearwood J, Phung D, Christensen H (2017) Kernel-based features for predicting population health indices from geocoded social media data. Decis Support Syst 102:22–31CrossRefGoogle Scholar
  66. Oneto L, Fumeo E, Clerico G, Canepa R, Papa F, Dambra C, Anguita D (2017) Train delay prediction systems: a big data analytics perspective. Big Data ResGoogle Scholar
  67. Pedersen E, Bongo LA (2017) Large-scale biological meta-database management. Future Gener Comput Syst 67:481–489CrossRefGoogle Scholar
  68. Peralta D, García S, Benitez JM, Herrera F (2017) Minutiae-based fingerprint matching decomposition: methodology for big data frameworks. Inf Sci 408:198–212CrossRefGoogle Scholar
  69. Plimpton SJ, Shead T (2014) Streaming data analytics via message passing with application to graph algorithms. J Parallel Distrib Comput 74(8):2687–2698CrossRefGoogle Scholar
  70. Prajapati DJ, Garg S, Chauhan NC (2017) MapReduce based multilevel consistent and inconsistent association rule detection from big data using interestingness measures. Big Data Res 9:18–27CrossRefGoogle Scholar
  71. Pulgar-Rubio F, Rivera-Rivas AJ, Pérez-Godoy MD, González P, Carmona CJ, del Jesus MJ (2017) MEFASD-BD: multi-objective evolutionary fuzzy algorithm for subgroup discovery in big data environments-A MapReduce solution. Knowl-Based Syst 117:70–78CrossRefGoogle Scholar
  72. Qian J, Lv P, Yue X, Liu C, Jing Z (2015) Hierarchical attribute reduction algorithms for big data using MapReduce. Knowl-Based Syst 73:18–31CrossRefGoogle Scholar
  73. Rahman MN, Esmailpour A, Zhao J (2016) Machine learning with big data an efficient electricity generation forecasting system. Big Data Res 5:9–15CrossRefGoogle Scholar
  74. Rathore MM, Ahmad A, Paul A, Rho S (2016) Urban planning and building smart cities based on the internet of things using big data analytics. Comput Netw 101:63–80CrossRefGoogle Scholar
  75. Rathore MM, Paul A, Ahmad A, Chilamkurthi N, Hong WH, Seo H (2017) Real-time secure communication for Smart City in high-speed big data environment. Future Gener Comput SystGoogle Scholar
  76. Ruan G, Zhang H (2017) Closed-loop big data analysis with visualization and scalable computing. Big Data Res 8:12–26CrossRefGoogle Scholar
  77. Sahal R, Khafagy MH, Omara FA (2017) Exploiting coarse-grained reused-based opportunities in Big Data multi-query optimization. J Comput SciGoogle Scholar
  78. Singh H, Bawa S (2017) A MapReduce-based scalable discovery and indexing of structured big data. Future Gener Comput Syst 73:32–43CrossRefGoogle Scholar
  79. Singh K, Guntuku SC, Thakur A, Hota C (2014) Big data analytics framework for peer-to-peer botnet detection using random forests. Inf Sci 278:488–497CrossRefGoogle Scholar
  80. Singh S, Garg R, Mishra PK (2017) Performance optimization of MapReduce-based Apriori algorithm on Hadoop cluster. Comput Electr EngGoogle Scholar
  81. Spivak A, Razumovskiy A, Nasonov D, Boukhanovsky A, Redice A (2018) Storage tier-aware replicative data reorganization with prioritization for efficient workload processing. Future Gener Comput Syst 79:618–629CrossRefGoogle Scholar
  82. Sun D, Zhang G, Yang S, Zheng W, Khan SU, Li K (2015) Re-stream: real-time and energy-efficient resource scheduling in big data stream computing environments. Inf Sci 319:92–112MathSciNetCrossRefGoogle Scholar
  83. Tennant M, Stahl F, Rana O, Gomes JB (2017) Scalable real-time classification of data streams with concept drift. Future Gener Comput Syst 75:187–199CrossRefGoogle Scholar
  84. Triguero I, Peralta D, Bacardit J, García S, Herrera F (2015) MRPR: a MapReduce solution for prototype reduction in big data classification. Neurocomputing 150:331–345CrossRefGoogle Scholar
  85. Tripathy BK, Mittal D (2016) Hadoop based uncertain possibilistic kernelized c-means algorithms for image segmentation and a comparative analysis. Appl Soft Comput 46:886–923CrossRefGoogle Scholar
  86. Tsai CW, Liu SJ, Wang YC (2017) A parallel metaheuristic data clustering framework for cloud. J Parallel Distrib ComputGoogle Scholar
  87. Um JH, Lee S, Kim TH, Jeong CH, Song SK, Jung H (2016) Semantic complex event processing model for reasoning research activities. Neurocomputing 209:39–45CrossRefGoogle Scholar
  88. Vennila V, Kannan AR (2016) Symmetric matrix-based predictive classifier for big data computation and information sharing in cloud. Comput Electr Eng 56:831–841CrossRefGoogle Scholar
  89. Wang H, Belhassena A (2017) Parallel trajectory search based on distributed index. Inf Sci 388:62–83CrossRefGoogle Scholar
  90. Wang B, Huang S, Qiu J, Liu Y, Wang G (2015) Parallel online sequential extreme learning machine based on MapReduce. Neurocomputing 149:224–232CrossRefGoogle Scholar
  91. Wang H, Xu Z, Fujita H, Liu S (2016) Towards felicitous decision making: an overview on challenges and trends of big data. Inf Sci 367:747–765CrossRefGoogle Scholar
  92. Wang J, He C, Liu Y, Tian G, Peng I, Xing J, Wang FL (2017a) Efficient alarm behavior analytics for telecom networks. Inf Sci 402:1–14CrossRefGoogle Scholar
  93. Wang, Y., Geng, S., & Gao, H. (2017). A proactive decision support method based on deep reinforcement learning and state partition. Knowl-Based SystGoogle Scholar
  94. Xia Y, Chen J, Lu X, Wang C, Xu C (2016) Big traffic data processing framework for intelligent monitoring and recording systems. Neurocomputing 181:139–146CrossRefGoogle Scholar
  95. Yuan J, Chen M, Jiang T, Li T (2017) Complete tolerance relation based parallel filling for incomplete energy big data. Knowl-Based Syst 132:215–225CrossRefGoogle Scholar
  96. Zhang F, Cao J, Khan SU, Li K, Hwang K (2015) A task-level adaptive MapReduce framework for real-time streaming data in healthcare applications. Future Gener Comput Syst 43:149–160CrossRefGoogle Scholar
  97. Zhang CY, Chen CP, Chen D, Ng KT (2016) MapReduce based distributed learning algorithm for restricted Boltzmann machine. Neurocomputing 198:4–11CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  • Azlinah Mohamed
    • 1
  • Maryam Khanian Najafabadi
    • 2
    Email author
  • Yap Bee Wah
    • 1
  • Ezzatul Akmal Kamaru Zaman
    • 1
  • Ruhaila Maskat
    • 1
  1. 1.Advanced Analytics Engineering Centre, Faculty Computer and Mathematical SciencesUniversiti Teknologi MARA (UiTM)Shah AlamMalaysia
  2. 2.Faculty of Information TechnologyINTI International University & CollegesNilaiMalaysia

Personalised recommendations