Abstract
Network administrators are faced with the increasingly challenging task of monitoring their network’s health in real time, drawing upon diverse and voluminous measurement data feeds and extensively mining them. The role of database systems in network monitoring has traditionally been that of data repositories; even if an application uses a database, the application logic is implemented using external programs. While such programs are flexible, they tend to be ad-hoc, opaque, inefficient and hard to maintain over time. In this paper, we propose a new way of implementing network monitoring applications: directly within a database as continually updated tables defined using a declarative query language (SQL). We also address a crucial technical issue with realizing this approach: SQL was designed for set-oriented data transformations, but network monitoring involves sequence-oriented analysis. To solve this problem, we propose an extension to SQL that makes sequence-oriented analysis easier to express and faster to evaluate. Using a prototype implementation in a large-scale production data warehouse, we demonstrate how the declarative sequence-oriented query language simplifies application development and how the associated system optimizations improve application performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Agrawal, J., et al.: Efficient pattern matching over event streams. In: SIGMOD 2008, pp. 147–160 (2008)
Ahuja, M., et al.: Peta-scale data warehousing at Yahoo! In: SIGMOD 2009, pp. 855–862 (2009)
Balazinska, M., et al.: Moirae: History-enhanced monitoring. In: CIDR 2007, pp. 375–386 (2007)
Cranor, C., et al.: A stream database for network applications. In: SIGMOD 2003, pp. 647–651 (2003)
Deri, L., Lorenzetti, V., Mortimer, S.: Collection and Exploration of Large Data Monitoring Sets Using Bitmap Databases. In: Ricciato, F., Mellia, M., Biersack, E. (eds.) TMA 2010. LNCS, vol. 6003, pp. 73–86. Springer, Heidelberg (2010)
Desnoyers, P., Shenoy, P.J.: Hyperion: High volume stream archival for retrospective querying. In: USENIX Annual Technical Conference, pp. 45–58 (2007)
Eriksson, B., et al.: Basisdetect: a model-based network event detection framework. In: IMC 2010, pp. 451–464 (2010)
Golab, L., et al.: Stream warehousing with DataDepot. In: SIGMOD 2009, pp. 847–854 (2009)
Golab, L., Johnson, T., Shkapenyuk, V.: Scheduling updates in a real-time stream warehouse. In: ICDE 2009, pp. 1207–1210 (2009)
Jain, N., et al.: Towards a streaming SQL standard. Proc. of the VLDB Endowment 1(2), 1379–1390 (2008)
Kalmanek, C., et al.: Darkstar: Using exploratory data mining to raise the bar on network reliability and performance. In: DRCN 2009 (2009)
Li, X., et al.: Advanced indexing techniques for wide-area network monitoring. In: NetDB 2005 (2005)
Maier, G., et al.: Enriching network security analysis with time travel. SIGCOMM Comput. Commun. Rev. 38, 183–194 (2008)
Markopoulou, A., et al.: Characterization of failures in an operational ip backbone network. IEEE/ACM Trans. Netw. 16(4), 749–762 (2008)
Papadogiannakis, A., Polychronakis, M., Markatos, E.P.: RRDtrace: Long-term raw network traffic recording using fixed-size storage. In: MASCOTS 2010, pp. 101–110 (2010)
Qiu, T., et al.: What happened in my network: mining network events from router syslogs. In: IMC 2010, pp. 472–484 (2010)
Quass, D., Widom, J.: On-line warehouse view maintenance. In: SIGMOD 1997, pp. 393–404 (1997)
Reiss, F., et al.: Enabling real-time querying of live and historical stream data. In: SSDBM 2007, p. 28 (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Golab, L., Johnson, T., Sen, S., Yates, J. (2012). A Sequence-Oriented Stream Warehouse Paradigm for Network Monitoring Applications. In: Taft, N., Ricciato, F. (eds) Passive and Active Measurement. PAM 2012. Lecture Notes in Computer Science, vol 7192. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-28537-0_6
Download citation
DOI: https://doi.org/10.1007/978-3-642-28537-0_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-28536-3
Online ISBN: 978-3-642-28537-0
eBook Packages: Computer ScienceComputer Science (R0)