Encyclopedia of Database Systems

2018 Edition
| Editors: Ling Liu, M. Tamer Özsu

Data Stream Management Architectures and Prototypes

  • Yanif Ahmad
  • Ugur Cetintemel
Reference work entry
DOI: https://doi.org/10.1007/978-1-4614-8265-9_24

Definition

Data stream processing architectures perform database-style query processing on a set of continuously arriving input streams. The core query executor in this type of architecture is designed to process continuous queries, rather than ad hoc queries, by pushing inputs through a series of operators functioning in a pipelined and potentially non-blocking manner. Stream processing applications perform explicit read and write operations to access storage via asynchronous disk I/O operations. Other architectural components that differ significantly from standard database designs include the stream processor’s scheduler, storage manager, and queue manager.

Historical Background

Support database-style query processing for long-running applications that operate in high (data) volume environments and impose high throughput and low latency requirements on the system. There have been several efforts from both the academic and industrial communities at developing functional prototypes of...

This is a preview of subscription content, log in to check access.

Recommended Reading

  1. 1.
    Abadi DJ, Carney D, Çetintemel U, Cherniack M, Convey C, Lee S, Stonebraker M, Tatbul N, Zdonik S. Aurora: a new model and architecture for data stream management. VLDB J. 2003; 12(2):120–39.Google Scholar
  2. 2.
    Babcock B, Babu S, Datar M, Motwani R, Widom J. Models and issues in data stream systems. In: Proceedings of the 21st ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems; 2002.Google Scholar
  3. 3.
    Chandrasekaran S, Deshpande A, Franklin M, Hellerstein J. TelegraphCQ: continuous dataflow processing for an uncertain world. In: Proceedings of the 1st Biennial Conference on Innovative Data Systems Research; 2003.Google Scholar
  4. 4.
    Arasu A, Babcock B, Babu S, McAlister J, Widom J. Characterizing memory requirements for queries over continuous data streams. ACM Trans Database Syst. 2004;29(1):162–94.CrossRefGoogle Scholar
  5. 5.
    Carney D, Çetintemel U, Rasin A, Zdonik SB, Cherniack M, Stonebraker M. Operator scheduling in a data stream manager. In: Proceedings of the 29th International Conference on Very Large Data Bases; 2003. p. 838–49.CrossRefGoogle Scholar
  6. 6.
    Tatbul N, Çetintemel U, Zdonik SB, Cherniack M, Stonebraker M. Load shedding in a data stream manager. In: Proceedings of the 29th International Conference on Very Large Data Bases; 2003. p. 309–20.CrossRefGoogle Scholar
  7. 7.
    Babu S, Motwani R, Munagala K, Nishizawa I, Widom J Adaptive ordering of pipelined stream filters. In: Proceedings of the ACM SIGMOD International Conference on Management of Data; 2004. p. 407–18.Google Scholar
  8. 8.
    Abadi D, Ahmad Y, Balazinska M, Çetintemel U, Cherniack M, Hwang J.-H, Lindner W, Maskey AS, Rasin A, Ryvkina E, Tatbul N, Xing Y, Zdonik S. The design of the Borealis stream processing engine. In: Proceedings of the 2nd Biennial Conference on Innovative Data Systems Research; 2005.Google Scholar
  9. 9.
    Cranor CD, Johnson T, Spatscheck O, Shkapenyuk V. Gigascope: a stream database for network applications. In: Proceedings of the ACM SIGMOD International Conference on Management of Data; 2003. p. 647–51.Google Scholar
  10. 10.
    Hammad MA, Mokbel MF, Ali MH, Aref WG, Catlin AC, Elmagarmid AK, Eltabakh MY, Elfeky MG, Ghanem TM, Gwadera R, Ilyas IF, Marzouk MS, Xiong X. Nile: a query processing engine for data streams. In: Proceedings of the 20th International Conference on Data Engineering; 2004. p. 851.Google Scholar
  11. 11.
    Gedik B, Andrade H, Wu K-L, Yu PS, Doo M. SPADE: the systems S declarative stream processing engine. In: Proceedings of the ACM SIGMOD International Conference on Management of Data; 2008.Google Scholar
  12. 12.
    Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters. Matei Zaharia, Tathagata Das, Haoyuan Li, Scott Shenker, Ion Stoica. HotCloud 2012. June 2012.Google Scholar
  13. 13.
    Das T, Dave A, Ma J, McCauley M, Franklin MJ, Shenker S, Stoica I. Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. Matei Zaharia, Mosharaf Chowdhury. NSDI 2012. April 2012.Google Scholar
  14. 14.
    Cetintemel U, Du J, Kraska T, Madden S, Maier D, Meehan J, Pavlo A, Stonebraker M, Sutherland E, Tatbul N, Tufte K, Wang H, Zdonik S. S-Store: a streaming NewSQL system for big velocity applications (demonstration). In: Proceedings of the 40th International Conference on Very Large Data Bases; 2014.Google Scholar
  15. 15.
    Kallman R, Kimura H, Natkins J, Pavlo A, Rasin A, Zdonik S, Jones EPC, Madden S, Stonebraker M, Zhang Y, Hugg J, Abadi DJ. H-Store: a high-performance, distributed main memory transaction processing system. Proc VLDB Endow. 2008;1(2):1496–9.CrossRefGoogle Scholar
  16. 16.
    Babcock B, Babu S, Datar M, Motwani R, Thomas D. Operator scheduling in data stream systems. VLDB J. 2004;13(4):333–53.CrossRefGoogle Scholar
  17. 17.
    Chen J, DeWitt DJ, Tian F, Wang Y. NiagaraCQ: a scalable continuous query system for internet databases. In: Proceedings of the ACM SIGMOD International Conference on Management of Data; 2000. p. 379–90.Google Scholar
  18. 18.
    Gehrke J. Data stream processing. IEEE Data Eng Bull. 2003;26(1).Google Scholar
  19. 19.
    Golab L, Özsu MT. Issues in data stream management. ACM SIGMOD Rec. 2003;32(2):5–14.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer ScienceBrown UniversityProvidenceUSA

Section editors and affiliations

  • Ugur Cetintemel
    • 1
  1. 1.Department of Computer ScienceBrown UniversityProvidenceUSA