Skip to main content
SpringerLink
Log in

Comparative Performance Evaluation Using Hadoop Ecosystem –PIG and HIVE Through Rendering of Duplicates

  • Conference paper
  • First Online:
International Conference on Advanced Computing Networking and Informatics

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 870))

Abstract

Traditionally, for analysis and decision making, preprocessed data have been stored on data warehouse and various operations are performed on those stored data. With the rapid growth in cloud applications and IoT-based systems, data get generated with high velocity and increased volume. Thus, big data, which get generated by variety of structured and unstructured data sources, are heterogeneous. There is a need to integrate variety of data and analyze the large-scale data. Hadoop provides a solution for such processing needs. Inherently, it is designed for high-throughput batch processing jobs and for handling complex queries for streaming data. This paper presents the MapReduce model of Hadoop framework with two analytical ecosystems PIG and HIVE. Here, we also present performance evaluation for each category like processing time for some queries executed on Pig and Hive while combining two healthcare datasets, gathered from different data sources. Comparative analysis has also been done and is presented in this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. A. Luntovskyy, D. Guetter, M. Klymash, Up-to-date paradigms for distributed computing, in 2nd International Conference on Advanced Information and Communication Technologies (AICT). IEEE Conference Publications, Ukraine, 2017, pp. 113–119

    Google Scholar 

  2. H. Cai, B. Xu, L. Jiang, IoT-based big data storage systems in cloud computing: perspectives and challenges. IEEE Internet Things J. 4(1), 75–87 (2017)

    Google Scholar 

  3. Apache Hadoop, http://hadoop.apache.org/

  4. K.H. Lee, Y.J. Lee, H. Choi, Y.D. Chung, B. Moon, Parallel data processing with MapReduce: a surver, SIGMOD Rec. 40(4), Korea (2011)

    Google Scholar 

  5. S. Ryza, U. Laserson, S. Owen, J. Wills, Advanced Analytics with Spark: Patterns for Learning from Data at Scale (O’Reilly Media, 2015)

    Google Scholar 

  6. C. Cao, W. Wang, Y. Zhang, X. Ma, Leveraging column family to improve multidimensional query performance in HBase, in IEEE 10th International Conference on Cloud Computing (CLOUD) (IEEE Conference Publications, USA, 2017), pp. 106–113

    Google Scholar 

  7. A. Thusoo, J.S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, R. Murthy, Hive: a warehousing solution over a map-reduceframework. Proc. VLDB Endow 2(2), 1626–1629 (2009)

    Article  Google Scholar 

  8. Z. Shao, A. Thusoo, J.S. Sarma, N. Jain, Hive-a petabyte scale data warehouse using hadoop, in Data Engineering (ICDE) (2010)

    Google Scholar 

  9. A. Gates, Programming Pig. O’Reilly Media, 1st edn. (October 2011)

    Google Scholar 

  10. C. Olston, B. Reed, U. Srivastava, R. Kumar, A. Tomkins, Pig latin: a not-so-foreign language for data processing, in Proceedings of the ACM SIGMOD International Conference on Management of Data, ACM (2008), pp. 1099–1110

    Google Scholar 

  11. T. Liu, J. Liu, H. Liu, W. Li, A performance evaluation of Hive for scientific data management, in IEEE International Conference on Big Data (2013), pp. 39–46

    Google Scholar 

  12. K. Shvachko, H. Kuang, S. Radia, R. Chansler, The hadoop distributed file system, in Proceedings of the IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST ‘10) (IEEE Computer Society, USA, 2010), pp. 1–10

    Google Scholar 

  13. Data Science Central, https://www.datasciencecentral.com

  14. UCI Machine Learning Repository, http://archive.ics.uci.edu

  15. A. Holmes, Hadoop in Practice, 2nd edn. (September 2014)

    Google Scholar 

  16. A. Choudhary C.S. Satsangi, Query Execution Performance Analysis of Big Data Using Hive and Pig of Hadoop, vol. Su-9, no. 3, Sep. (2015), pp. 91–93

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Medicaps University, Indore, India

    Pragya Pandey & C. S. Satsangi

Authors
  1. Pragya Pandey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. C. S. Satsangi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pragya Pandey .

Editor information

Editors and Affiliations

  1. Medi-Caps University, Indore, Madhya Pradesh, India

    Raj Kamal

  2. Loughborough University, Loughborough, UK

    Michael Henshaw

  3. Medi-Caps Group of Institutions, Indore, Madhya Pradesh, India

    Pramod S. Nair

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pandey, P., Satsangi, C.S. (2019). Comparative Performance Evaluation Using Hadoop Ecosystem –PIG and HIVE Through Rendering of Duplicates. In: Kamal, R., Henshaw, M., Nair, P. (eds) International Conference on Advanced Computing Networking and Informatics. Advances in Intelligent Systems and Computing, vol 870. Springer, Singapore. https://doi.org/10.1007/978-981-13-2673-8_11

Download citation

  • DOI: https://doi.org/10.1007/978-981-13-2673-8_11

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-13-2672-1

  • Online ISBN: 978-981-13-2673-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation