Skip to main content
SpringerLink
Log in

An Empirical Study of Big Data: Opportunities, Challenges and Technologies

  • Conference paper
  • First Online:
New Paradigm in Decision Science and Management

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 1005))

  • 675 Accesses

Abstract

Nowadays, Big Data is considered to be the emerging field of research. However, it has gained momentum a decade ago and is still in its infancy stage. Big Data is a huge volume of data that is produced from various sources and thus traditional database systems are incapable of processing such a voluminous data. This is the need of hour to use advanced tools and methods to get value from Big Data. However, the pace of data generation is greater than ever which leads to numerous challenges such as data inconsistency, security, timeliness and scalability. Here, in this paper, a brief introduction to the Big Data technology and its importance in various fields. This paper mainly focuses on characteristics of Big Data and gives insights of a brief clarification of challenges related to Big Data and some analytical methods such as Hadoop and MapReduce. The various tools and techniques that can be used in Big Data technology have been discussed further.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Jaseena, K.U., David, J.M.: Issues, challenges, and solutions: big data mining. Comput. Sci. Inf. Technol. (CS&IT) 131–140 (2014)

    Google Scholar 

  2. Lawal, Z.K., Zakari, R.Y., Shuaibu, M.Z., Bala, A.: A review: issues and challenges in big data from analytic and storage perspectives. Int. J. Eng. Comput. Sci. 5(3), 15947–15961 (2016)

    Google Scholar 

  3. Holzinger, A., Stocker, C., Ofner, B., Prohaska, G., Brabenetz, A., Hofmann-Wellenhof, R.: Combining HCI, natural language processing, and knowledge discovery-potential of IBM content analytics as an assistive technology in the biomedical field. In: Human-Computer Interaction and Knowledge Discovery in Complex, Unstructured, Big Data (pp. 13–24). Springer, Berlin (2013)

    Chapter  Google Scholar 

  4. Landset, S., Khoshgoftaar, T.M., Richter, A.N., Hasanin, T.: A survey of open source tools for machine learning with big data in the Hadoop ecosystem. J. Big Data 2(1), 24 (2015)

    Article  Google Scholar 

  5. Jin, X., Wah, B.W., Cheng, X., Wang, Y.: Significance and challenges of big data research. Big Data Res. 2(2), 59–64 (2015)

    Article  Google Scholar 

  6. Kaur, H., Alam, M.A., Jameel, R., Mourya, A., Chang, V.A.: Proposed solution and future direction for blockchain-based heterogeneous medicare data in cloud environment. J. Med. Sys. 42(8) (2018)

    Google Scholar 

  7. Anuradha, J.: A brief introduction on Big Data 5Vs characteristics and Hadoop technology. Procedia Comput. Sci. 48, 319–324 (2015)

    Article  Google Scholar 

  8. Kaur, H., Lechman, E., Marszk, A.: Catalyzing Development through ICT Adoption: The Developing World Experience, pp. 288. Springer Publishers, Switzerland  (2017)

    Google Scholar 

  9. Beyer, M.A., Laney, D.: The Importance of ‘Big Data’: A Definition, pp. 2014–2018. Gartner, Stamford (2012)

    Google Scholar 

  10. Kaisler, S., Armour, F., Espinosa, J.A., Money, W.: Big data: issues and challenges moving forward. In: 2013 46th Hawaii International Conference on System Sciences (HICSS) (pp. 995–1004). IEEE (2013)

    Google Scholar 

  11. Mervis, J. (2012). Agencies rally to tackle big data

    Article  Google Scholar 

  12. Hey, T., Tansley, S., Tolle, K.M.: The Fourth Paradigm: Data-Intensive Scientific Discovery, vol. 1. Microsoft Research, Redmond (2009)

    Google Scholar 

  13. O’Neil, C., Schutt, R.: Doing Data Science: Straight Talk from the Frontline. O’Reilly Media, Inc. (2013)

    Google Scholar 

  14. Chen, H., Chiang, R.H., Storey, V.C.: Business intelligence and analytics: from big data to big impact. MIS Q. 1165–1188 (2012)

    Article  Google Scholar 

  15. Ginsberg, J., Mohebbi, M.H., Patel, R.S., Brammer, L., Smolinski, M.S., Brilliant, L.: Detecting influenza epidemics using search engine query data. Nature 457(7232), 1012 (2009)

    Article  Google Scholar 

  16. Gantz, J., Reinsel, D.: Extracting value from chaos. IDC iview 1142(2011), 1–12 (2011)

    Google Scholar 

  17. Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The hadoop distributed file system. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) (pp. 1–10). Ieee (2010)

    Google Scholar 

  18. Ghemawat, S., Gobioff, H., Leung, S.T.: The Google File System, 37(5), 29–43. ACM (2003)

    Google Scholar 

  19. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)

    Article  Google Scholar 

  20. Ahrens, J., Hendrickson, B., Long, G., Miller, S., Ross, R., Williams, D.: Data-intensive science in the US DOE: case studies and future challenges. Comput. Sci. Eng. 13(6), 14–24 (2011)

    Article  Google Scholar 

  21. Sivarajah, U., Kamal, M.M., Irani, Z., Weerakkody, V.: Critical analysis of Big Data challenges and analytical methods. J. Bus. Res. 70, 263–286 (2017)

    Article  Google Scholar 

  22. Lu, R., Zhu, H., Liu, X., Liu, J.K., Shao, J.: Toward efficient and privacy-preserving computing in big data era. IEEE Netw. 28(4), 46–50 (2014)

    Article  Google Scholar 

  23. Khan, N., Yaqoob, I., Hashem, I.A.T., Inayat, Z., Ali, M., Kamaleldin, W., Alam, M., Shiraz, M., Gani, A.: Big data: survey, technologies, opportunities, and challenges. Sci. World J. (2014)

    Google Scholar 

  24. Zikopoulos, P., Eaton, C.: Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data. McGraw-Hill Osborne Media (2011)

    Google Scholar 

  25. Demchenko, Y., Grosso, P., De Laat, C., Membrey, P.: Addressing big data issues in scientific data infrastructure. In: 2013 International Conference on Collaboration Technologies and Systems (CTS) (pp. 48–55). IEEE (2013)

    Google Scholar 

  26. Yi, X., Liu, F., Liu, J., Jin, H.: Building a network highway for big data: architecture and challenges. IEEE Netw. 28(4), 5–13 (2014)

    Article  Google Scholar 

  27. Samuel, S.J., RVP, K., Sashidhar, K., Bharathi, C.R.: A survey on big data and its research challenges. ARPN J. Eng. Appl. Sci. 10(8), 3343–3347 (2015)

    Google Scholar 

  28. Bezdek, J.C.: Objective function clustering. In: Pattern Recognition with Fuzzy Objective Function Algorithms (pp. 43–93). Springer, Boston (1981)

    Chapter  Google Scholar 

  29. Arel, I., Rose, D.C., Karnowski, T.P.: Deep machine learning-a new frontier in artificial intelligence research. IEEE Comput. Intell. Mag. 5(4), 13–18 (2010)

    Article  Google Scholar 

  30. Mitra, P., Murthy, C.A., Pal, S.K.: A probabilistic active support vector learning algorithm. IEEE Trans. Pattern Anal. Mach. Intell. 26(3), 413–418 (2004)

    Article  Google Scholar 

  31. Mansour, Y., Chang, A.Y., Tamby, J., Vaahedi, E., Corns, B.R., El-Sharkawi, M.A.: Large scale dynamic security screening and ranking using neural networks. IEEE Trans. Power Syst. 12(2), 954–960 (1997)

    Article  Google Scholar 

  32. Fujimoto, Y., Fukuda, N., Akabane, T.: Massively parallel architectures for large scale neural network simulations. IEEE Trans. Neural Netw. 3(6), 876–888 (1992)

    Article  Google Scholar 

  33. Ahn, J.B.: Neuron machine: parallel and pipelined digital neurocomputing architecture. In: 2012 IEEE International Conference on Computational Intelligence and Cybernetics (CyberneticsCom) (pp. 143–147). IEEE (2012)

    Google Scholar 

  34. Ma, H., King, I., Lyu, M.R.: Mining web graphs for recommendations. IEEE Trans. Knowl. Data Eng. 24(6), 1051–1064 (2012)

    Article  Google Scholar 

  35. White, T.: Hadoop: the definitive guide: the definitive guide. O’Reilly Media, Inc. (2009)

    Google Scholar 

  36. Ingersoll, G.: Introducing Apache Mahout. Scalable, Commercial Friendly Machine Learning for Building Intelligent Applications. IBM (2009)

    Google Scholar 

  37. Rong, C.: November. Using Mahout for clustering Wikipedia’s latest articles: a comparison between k-means and fuzzy c-means in the cloud. In 2011 IEEE Third International Conference on Cloud Computing Technology and Science (CloudCom), pp. 565–569. IEEE (2011)

    Google Scholar 

  38. Chauhan, R., Kaur, H., Lechman, E., Marszk, A.: Big data analytics for ICT monitoring and development. In: Kaur et al. (eds.) Catalyzing Development through ICT Adoption: The Developing World Experience, pp. 25–36. Springer (2017)

    Google Scholar 

  39. Storm. http://storm-project.net (2012)

  40. Chauhan, J., Chowdhury, S.A., Makaroff, D.: Performance evaluation of Yahoo! S4: a first look. In: 2012 Seventh International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (pp. 58–65). IEEE (2012)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, School of Engineering Sciences and Technology, Jamia Hamdard, New Delhi, India

    Shafqat Ul Ahsaan, Harleen Kaur & Sameena Naaz

Authors
  1. Shafqat Ul Ahsaan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Harleen Kaur
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sameena Naaz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shafqat Ul Ahsaan .

Editor information

Editors and Affiliations

  1. Siksha ‘O’ Anusandhan University, Bhubaneswar, India

    Srikanta Patnaik

  2. Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University, Kowloon, Hong Kong

    Andrew W. H. Ip

  3. Department of Business Systems and Analytics, La Salle University, Philadelphia, PA, USA

    Madjid Tavana

  4. Victoria Business School, School of Management, Victoria University of Wellington, Wellington, New Zealand

    Vipul Jain

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ul Ahsaan, S., Kaur, H., Naaz, S. (2020). An Empirical Study of Big Data: Opportunities, Challenges and Technologies. In: Patnaik, S., Ip, A., Tavana, M., Jain, V. (eds) New Paradigm in Decision Science and Management. Advances in Intelligent Systems and Computing, vol 1005. Springer, Singapore. https://doi.org/10.1007/978-981-13-9330-3_6

Download citation

Publish with us

Policies and ethics

Search

Navigation