Spark Streaming (Advanced)

Abstract

The previous chapter introduced the core concepts of streaming processing, the features that the Spark Structured Streaming processing engine provides, and the basic steps of putting together a streaming application. Real-world streaming applications usually need to extract insights or patterns from the incoming real-time data at scale and feed that information into downstream applications to make business decisions or to save that information in some storage system for further analysis or visualization purposes. Another aspect of real-world streaming applications is that they are continuously running to process real-time data as it comes in. Therefore, they must be resilient against failures. The first half of this chapter covers event-time processing and stateful processing features in Structured Streaming and how they can help with extracting insights or patterns from incoming real-time data. The second half of this chapter explains the support Structured Streaming provides to help streaming applications to be fault tolerant against failures and to monitor the status and progress of streaming applications.