Skip to main content
SpringerLink
Log in

Distributed Computing Technologies in Big Data Analytics

  • Chapter
  • First Online:
Distributed Computing in Big Data Analytics

Part of the book series: Scalable Computing and Communications ((SCC))

Abstract

One of the fundamental technology used in Big Data Analytics is the distributed computing. The traditional distributed computing technology has been adapted to create a new class of distributed computing platform and software components that make the big data analytics easier to implement. In this chapter, we discuss few of these technologies. First, we discuss the distributed database technology and how this technology has been adapted to develop no-SQL database technologies. Following this, we discuss the distributed file system (HDFS) and distributed computing technology such as map-reduce and spark. We discuss how the distributed storage and distributed computing has impacted the machine learning platforms for big data. Next, we discuss the distributed search platform and how such search platform can be used for data analytics on textual documents. We also describe the distributed communication platform such as message queue and message processing software. The data visualization technology is also changing with the big data. So lastly we introduce readers to few newer data visualization platforms targeted for big data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Adamic, L. A., & Huberman, B. A. (2000). Power-Law Distribution of the World Wide Web. Science, 287(5461).

    Google Scholar 

  2. Aerospike. (2017). Aerospike | High Performance NoSQL Database. Retrieved March 23, 2017, from http://www.aerospike.com/

  3. Amazon. (2017). AWS | Amazon SimpleDB – Simple Database Service. Retrieved March 23, 2017, from https://aws.amazon.com/simpledb/

  4. Apache. (2015). Solr. Retrieved from http://lucene.apache.org/solr/

  5. Apache. (2017a). Apache CouchDB. Retrieved March 23, 2017, from http://couchdb.apache.org/

  6. Apache. (2017b). Apache Mahout: Scalable machine learning and data mining. Retrieved March 23, 2017, from http://mahout.apache.org/

  7. Apache Flink. (2017). Apache Flink: Introduction to Apache Flink®. Retrieved March 23, 2017, from https://flink.apache.org/introduction.html

  8. Apache Kafka. (2017). Apache Kafka. Retrieved March 23, 2017, from https://kafka.apache.org/intro

  9. Apache Lucene. (2017). Apache Lucene – Welcome to Apache Lucene. Retrieved March 23, 2017, from https://lucene.apache.org/

  10. Apache Zeppelin. (2017). Zeppelin. Retrieved March 23, 2017, from https://zeppelin.apache.org/

  11. Basho. (2017). Riak – Distributed Databases. Retrieved March 23, 2017, from http://basho.com/products/

  12. Brewer, E., & Eric. (2010). A certain freedom. In Proceeding of the 29th ACM SIGACT-SIGOPS symposium on Principles of distributed computing – PODC ’10 (pp. 335–335). New York, New York, USA: ACM Press. http://doi.org/10.1145/1835698.1835701

  13. Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., … Gruber, R. E. (2006). Bigtable: A distributed storage system for structured data. In 7th Symposium on Operating Systems Design and Implementation (OSDI ’06), November 6–8, Seattle, WA, USA (pp. 205–218). USENIX Association. Retrieved from http://research.google.com/archive/bigtable-osdi06.pdf

  14. Cloudera. (2017). How-to: Build a Machine-Learning App Using Sparkling Water and Apache Spark – Cloudera Engineering Blog. Retrieved March 23, 2017, from http://blog.cloudera.com/blog/2015/10/how-to-build-a-machine-learning-app-using-sparkling-water-and-apache-spark/

  15. Datastax. (2017). Introduction to Cassandra Query Language. Retrieved March 23, 2017, from https://docs.datastax.com/en/cql/3.1/cql/cql_intro_c.html

  16. DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., … Vogels, W. (2007). Dynamo. In Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles – SOSP ’07 (p. 205). New York, New York, USA: ACM Press. http://doi.org/10.1145/1294261.1294281

  17. Elastic. (2017). Open Source Search Analytics · Elasticsearch. Retrieved March 23, 2017, from https://www.elastic.co/

  18. Erb, B. (2016). The Challenge of Distributed Database Systems. Retrieved March 23, 2017, from http://berb.github.io/diploma-thesis/community/061_challenge.html

  19. Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003). The Google file system. ACM SIGOPS Operating Systems Review.

    Google Scholar 

  20. Gilbert, S., & Lynch, N. (2002). Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. ACM SIGACT News, 33(2), 51. http://doi.org/10.1145/564585.564601

  21. Gino, I. (2017). Caching a MongoDB Database with Redis — SitePoint. Retrieved March 23, 2017, from https://www.sitepoint.com/caching-a-mongodb-database-with-redis/

  22. H2O. (2017). H2O.ai. Retrieved March 23, 2017, from https://www.h2o.ai/h2o/

  23. Hortonworks. (2017a). Apache Hadoop HDFS – Hortonworks. Retrieved March 23, 2017, from https://hortonworks.com/apache/hdfs/#section_2

  24. Hortonworks. (2017b). Introduction to Kafka – Hortonworks Data Platform. Retrieved March 23, 2017, from https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_kafka-user-guide/content/ch_using_kafka.html

  25. Hotcodeshare. (2017). How Elasticsearch index document? | Hot code share. Retrieved March 23, 2017, from http://www.hotcodeshare.com/content/how-elasticsearch-index-document

  26. IBM. (2017). IBM Data Science Experience. Retrieved March 23, 2017, from https://www.ibm.com/us-en/marketplace/data-science-experience/resources

  27. Jupyter. (2017). Project Jupyter. Retrieved March 23, 2017, from http://jupyter.org/

  28. kickstarthadoop. (2017). Kick Start Hadoop: Word Count – Hadoop Map Reduce Example. Retrieved March 23, 2017, from http://kickstarthadoop.blogspot.com/2011/04/word-count-hadoop-map-reduce-example.html

  29. Leavitt, N. (2010). Will NoSQL Databases Live Up to Their Promise? Computer, 43(2), 12–14. http://doi.org/10.1109/MC.2010.58

  30. Liip. (2017). On ElasticSearch performance – Liip Blog. Retrieved March 23, 2017, from https://blog.liip.ch/archive/2013/07/19/on-elasticsearch-performance.html

  31. Memcached. (2017). memcached – a distributed memory object caching system. Retrieved March 23, 2017, from https://memcached.org/

  32. MongoDB Inc. (2015). mongoDB. Retrieved from https://www.mongodb.org/

  33. Oracle. (2017). Berkeley DB Products. Retrieved March 23, 2017, from https://www.oracle.com/database/berkeley-db/index.html

  34. Ozsu, M. T., & Valduriez, P. (2011). Principles of Distributed Database Systems – M. Tamer Özsu, Patrick Valduriez – Google Books. Retrieved from https://books.google.com/books?hl=en&lr=&id=TOBaLQMuNV4C&oi=fnd&pg=PR3&dq=Distributed+Database&ots=LqFjgM_P-7&sig=mcmEnxerBLtixHY-0CrzS2hFojc#v=onepage&q=Distributed Database&f=false

  35. Pritchett, D., & Dan. (2008). BASE: AN ACID ALTERNATIVE. Queue, 6(3), 48–55. http://doi.org/10.1145/1394127.1394128

  36. RabbitMQ. (2017). RabbitMQ – Messaging that just works. Retrieved March 23, 2017, from https://www.rabbitmq.com/

  37. Redis. (2017). Redis. Retrieved March 23, 2017, from https://redis.io/

  38. Saphanatutorial. (2017). How YARN Overcomes MapReduce Limitations in Hadoop 2.0. Retrieved March 23, 2017, from http://saphanatutorial.com/how-yarn-overcomes-mapreduce-limitations-in-hadoop-2-0/

  39. Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010). The Hadoop Distributed File System. In 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) (pp. 1–10). IEEE. http://doi.org/10.1109/MSST.2010.5496972

  40. Sparkflows. (2017). SparkFlows.io | Big Data Application Development Made Easy. Retrieved March 23, 2017, from https://www.sparkflows.io/overview

  41. Tableau. (2017). Business Intelligence and Analytics – Tableau Software. Retrieved March 23, 2017, from https://www.tableau.com/

  42. Tzoumas, K., & Metzger, R. (2015). Kafka + Flink: A practical, how-to guide – data Artisans. Retrieved March 23, 2017, from https://data-artisans.com/kafka-flink-a-practical-how-to/

Download references

Author information

Authors and Affiliations

  1. University of South Florida, Tampa, FL, USA

    Kaushik Dutta

Authors
  1. Kaushik Dutta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kaushik Dutta .

Editor information

Editors and Affiliations

  1. IBM Analytics, San Ramon, California, USA

    Sourav Mazumder

  2. Discipline of Computer Science and Engineering, Indian Institute of Technology Indore, Indore, Madhya Pradesh, India

    Robin Singh Bhadoria

  3. Directorate General of Training, Ministry of Skill Development and Entrepreneurship, New Delhi, Delhi, India

    Ganesh Chandra Deka

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Dutta, K. (2017). Distributed Computing Technologies in Big Data Analytics. In: Mazumder, S., Singh Bhadoria, R., Deka, G. (eds) Distributed Computing in Big Data Analytics. Scalable Computing and Communications. Springer, Cham. https://doi.org/10.1007/978-3-319-59834-5_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-59834-5_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-59833-8

  • Online ISBN: 978-3-319-59834-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation