Integrating a Stream Processing Engine and Databases for Persistent Streaming Data Management

  • Yousuke Watanabe
  • Shinichi Yamada
  • Hiroyuki Kitagawa
  • Toshiyuki Amagasa
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4653)


Because of increased stream data, managing stream data has become quite important. This paper describes our data stream management system, which employs an architecture combining a stream processing engine and DBMS. Based on the architecture, the system processes both continuous queries and traditional one-shot queries. Our proposed query language supports not only filtering, join, and projection over data streams, but also continuous persistence requirements for stream data. Users can also specify continuous queries that integrate streaming data and historical data stored in DBMS. Another contribution of this paper is feasibility validation of queries. Processing queries on streams with frequent inputs may cause the system to overflow its capacity. Specifically, the maximum writing rate to DBMS is a significant bottleneck when we try to store stream data into DBMS. Our system detects infeasible queries in advance.


Stream Data Query Language Input Rate Stream Processing Query Plan 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abadi, D.J., et al.: Aurora: a New Model and Architecture for Data Stream Management. VLDB Journal 12(2), 120–139 (2003)CrossRefGoogle Scholar
  2. 2.
    Abadi, D.J., et al.: The Design of the Borealis Stream Processing Engine. In: Proc. CIDR, pp. 277–289 (2005)Google Scholar
  3. 3.
    Arasu, A., et al.: The CQL Continuous Query Language: Semantic Foundations and Query Execution. VLDB Journal 15(2) (2006)Google Scholar
  4. 4.
    Ayad, A.M., et al.: Static Optimization of Conjunctive Queries with Sliding Windows Over Infinite Streams. In: Proc. ACM SIGMOD, pp. 419–430 (2004)Google Scholar
  5. 5.
    Babcock, B., et al.: Load Shedding for Aggregation Queries over Data Streams. In: Proc. ICDE, pp. 350–361 (2004)Google Scholar
  6. 6.
    Chandrasekaran, S., et al.: TelegraphCQ: Continuous Dataflow Processing for an Uncertain World. In: Proc. CIDR (2003)Google Scholar
  7. 7.
    Chen, J., et al.: NiagaraCQ: A Scalable Continuous Query System for Internet Databases. In: Proc. ACM SIGMOD, pp. 379–390 (2000)Google Scholar
  8. 8.
    Motwani, R., et al.: Query Processing, Resource Management, and Approximation in a Data Stream Management System. In: Proc. CIDR (2003)Google Scholar
  9. 9.
    Tatbul, N., et al.: Load Shedding in a Data Stream Manager. In: Proc. VLDB, pp. 309–320 (2003)Google Scholar
  10. 10.
    Viglas, S.D., et al.: Rate-based Query Optimization for Streaming Information Sources. In: Proc. ACM SIGMOD, pp.37–48 (2002)Google Scholar
  11. 11.
    Wang, S., et al.: State-Slice: New Paradigm of Multi-query Optimization of Window-based Stream Queries. In: Proc. VLDB, pp. 619–630 (2006)Google Scholar
  12. 12.

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Yousuke Watanabe
    • 1
  • Shinichi Yamada
    • 2
  • Hiroyuki Kitagawa
    • 2
    • 3
  • Toshiyuki Amagasa
    • 2
    • 3
  1. 1.Japan Science and Technology Agency 
  2. 2.Graduate School of Systems and Information Engineering, University of Tsukuba 
  3. 3.Center of Computational Sciences, University of Tsukuba 

Personalised recommendations