Pro Docker pp 219-228 | Cite as

Using Apache Spark

  • Deepak Vohra

Abstract

Apache Spark is a data processing engine for large data sets. Apache Spark is much faster (up to 100 times faster in memory) than Apache Hadoop MapReduce. In cluster mode, Spark applications run as independent processes coordinated by the SparkContext object in the driver program, which is the main program. The SparkContext may connect to several types of cluster managers to allocate resources to Spark applications. The supported cluster managers include the Standalone cluster manager, Mesos and YARN. Apache Spark is designed to access data from varied data sources including the HDFS, Apache HBase and NoSQL databases such as Apache Cassandra and MongoDB. In this chapter we shall use the same CDH Docker image that we used for several of the Apache Hadoop frameworks including Apache Hive and Apache HBase. We shall run an Apache Spark Master in cluster mode using the YARN cluster manager in a Docker container.

Keywords

Cluster Mode Cluster Manager NoSQL Database Application Report Detailed Output 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Copyright information

© Deepak Vohra 2016

Authors and Affiliations

  • Deepak Vohra
    • 1
  1. 1.BCCanada

Personalised recommendations