Abstract
In the real world, deployments of Big Data systems fit into a larger ecosystem that consists of managed cloud instances, cluster managers, data stores and warehouses, and so on. Cloud deployments enable organizations to pass the buck of DevOps to the cloud service provider and thus free them to focus on application development and business operations. Along with economies of scale, this provides on-demand horizontal elasticity and simplified scheduling. Similar to other systems, Spark can work in the cloud with fully managed instances provided by a number of companies including Google (Dataproc), Databricks, and IBM (Bluemix). No book about Spark would be complete without a discussion of running it in the cloud. Other topics covered in this chapter include the Spark Python API, the lambda architecture, and graph processing.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Brad Plumer, “How Yelp Is Killing Chain Restaurants,” The Washington Post, October 3, 2011, https://www.washingtonpost.com/blogs/ezra-klein/post/how-yelp-is-killing-chain-restaurants/2011/10/03/gIQAokJvHL_blog.html .
- 2.
- 3.
At the time of writing, Google is offering $300 of free credit for 60 days for new accounts.
- 4.
Contained in yelp_academic_dataset_business.json.
- 5.
- 6.
Craig Smith, “By the Numbers: 50 Amazing Yelp Statistics,” DMR, April 1, 2016, http://expandedramblings.com/index.php/yelp-statistics/ .
- 7.
Nathan Marz, “How to Beat the CAP Theorom,” Thoughts from the Red Planet, October 13, 2011, http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html .
- 8.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2016 Zubair Nabi
About this chapter
Cite this chapter
Nabi, Z. (2016). Of Clouds, Lambdas, and Pythons. In: Pro Spark Streaming. Apress, Berkeley, CA. https://doi.org/10.1007/978-1-4842-1479-4_10
Download citation
DOI: https://doi.org/10.1007/978-1-4842-1479-4_10
Published:
Publisher Name: Apress, Berkeley, CA
Print ISBN: 978-1-4842-1480-0
Online ISBN: 978-1-4842-1479-4
eBook Packages: Professional and Applied ComputingApress Access BooksProfessional and Applied Computing (R0)