Enabling Deep Analytics in Stream Processing Systems

Abstract

Real-time applications often analyze data coming from sensor networks using relational and domain-specific operations such as signal processing and machine learning algorithms. To support such increasingly important scenarios, many data management systems integrate with numerical frameworks like R. Such solutions, however, incur significant performance penalties as relational engines and numerical tools operate on fundamentally different data models with expensive inter-communication mechanisms. In addition, none of these solutions supports efficient real-time and incremental analysis. In this work, we advocate a deep integration of domain-specific operations into general-purpose query processors with the goal of providing unified query and data models for both online and offline processing. Our proof-of-concept system tightly integrates relational and digital signal processing operations and achieves orders of magnitude better performance than existing loosely-coupled data management systems.