Exactly-Once Semantics with Real-Time Data Pipelines

Abstract

Real-time systems like IoT, recommendation systems, fraud detection systems often have a need of ensuring that the application processes the data only once. In real-time streaming applications there is often a possibility that a batch of data might be handed over to the application multiple times resulting in duplicate data being processed by the application. Any stream processing product cannot unilaterally guarantee exactly once processing semantics. It is true under certain assumptions or when the application and the stream processing framework collaborate in certain ways. In this paper, we present a design to address the problem of real-time streaming applications by achieving an end-to-end exactly once delivery. The main contribution of our work is to provide solution to the complex task of recovering the application state from application restarts, network crashes, etc., and detecting and filtering out of order duplicate data while maintaining a high throughput.