Abstract
Apache Spark is a data processing engine for large data sets. Apache Spark is much faster (up to 100 times faster in memory) than Apache Hadoop MapReduce. In cluster mode, Spark applications run as independent processes coordinated by the SparkContext object in the driver program, which is the main program. The SparkContext may connect to several types of cluster managers to allocate resources to Spark applications. The supported cluster managers include the Standalone cluster manager, Mesos and YARN. Apache Spark is designed to access data from varied data sources including the HDFS, Apache HBase and NoSQL databases such as Apache Cassandra and MongoDB. In this chapter we shall use the same CDH Docker image that we used for several of the Apache Hadoop frameworks including Apache Hive and Apache HBase. We shall run an Apache Spark Master in cluster mode using the YARN cluster manager in a Docker container.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2016 Deepak Vohra
About this chapter
Cite this chapter
Vohra, D. (2016). Using Apache Spark. In: Pro Docker. Apress, Berkeley, CA. https://doi.org/10.1007/978-1-4842-1830-3_14
Download citation
DOI: https://doi.org/10.1007/978-1-4842-1830-3_14
Publisher Name: Apress, Berkeley, CA
Print ISBN: 978-1-4842-1829-7
Online ISBN: 978-1-4842-1830-3
eBook Packages: Professional and Applied ComputingProfessional and Applied Computing (R0)Apress Access Books