Abstract
Modern communication networks generate large amounts of operational data, including traffic and utilization statistics and alarm/fault data at various levels of detail. These massive collections of network-management data can grow in the order of several Terabytes per year, and typically hide “knowledge” that is crucial to some of the key tasks involved in effectively managing a communication network (e.g., capacity planning and traffic engineering). In this short paper, we provide an overview of some of our recent and ongoing work in the context of the \( \mathcal{N}\mathcal{E}\mathcal{M}\mathcal{E}\mathcal{S}\mathcal{I}\mathcal{S} \) project at Bell Laboratories that aims to develop novel data warehousing and mining technology for the effective storage, exploration, and analysis of massive network-management data sets. We first give some highlights of our work on Model-Based Semantic Compression (MBSC), a novel data-compression framework that takes advantage of attribute semantics and data-mining models to perform lossy compression of massive network-data tables. We discuss the architecture and some of the key algorithms underlying \( \mathcal{S}\mathcal{P}\mathcal{A}\mathcal{R}\mathcal{T}\mathcal{A}\mathcal{N} \) , a model-based semantic compression system that exploits predictive data correlations and prescribed error tolerances for individual attributes to construct concise and accurate Classification and Regression Tree (CaRT) models for entire columns of a table. We also summarize some of our ongoing work on warehousing and analyzing network-fault data and discuss our vision of how data-mining techniques can be employed to help automate and improve fault-management in modern communication networks. More specifically, we describe the two key components of modern fault-management architectures, namely the event-correlation and the root-cause analysis engines, and propose the use of mining ideas for the automated inference and maintenance of the models that lie at the core of these components based on warehoused network data.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
“NetFlow Services and Applications”. Cisco Systems White Paper, 1999.
S. Babu, M. Garofalakis, and R. Rastogi. “SPARTAN: A Model-Based Semantic Compression System for Massive Data Tables”. In Proc. of the 2001 ACM SIGMOD Intl. Conf. on Management of Data, May 2001.
L. Breiman, J.H. Friedman, R.A. Olshen, and C.J. Stone. “Classification and Regression Trees”. Chapman & Hall, 1984.
A.L. Buchsbaum, D.F. Caldwell, K. Church, G.S. Fowler, and S. Muthukrishnan. “Engineering the Compression of Massive Tables: An Experimental Approach”. In Proc. of the 11th Annual ACM-SIAM Symp. on Discrete Algorithms, January 2000.
K. Chakrabarti, M. Garofalakis, R. Rastogi, and K. Shim. “Approximate Query Processing Using Wavelets”. In Proc. of the 26th Intl. Conf. on Very Large Data Bases, September 2000.
A. Deshpande, M. Garofalakis, and R. Rastogi. “Independence is Good: Dependency-Based Histogram Synopses for High-Dimensional Data”. In Proc. of the 2001 ACM SIGMOD Intl. Conf. on Management of Data, May 2001.
C. Fraleigh, S. Moon, C. Diot, B. Lyles, and F. Tobagi. “Architecture of a Passive Monitoring System for Backbone IP Networks”. Technical Report TR00-ATL-101-801, Sprint Advanced Technology Laboratories, October 2000.
M.R. Garey and D.S. Johnson. “Computers and Intractability: A Guide to the Theory of NP-Completeness”. W.H. Freeman, 1979.
M.M. Halldórsson. “Approximations of Weighted Independent Set and Hereditary Subset Problems”. Journal of Graph Algorithms and Applications, 4(1), 2000.
H.V. Jagadish, J. Madar, and R. Ng. “Semantic Compression and Pattern Extraction with Fascicles”. In Proc. of the 25th Intl. Conf. on Very Large Data Bases, September 1999.
G. Jakobson and M.D. Weissman. “Alarm Correlation”. IEEE Network, November 1993.
Judea Pearl. “Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference”. Morgan Kaufmann Publishers, Inc., 1988.
William Stallings. “SNMP, SNMPv2, SNMPv3, and RMON 1 and 2”. Addison-Wesley Longman, Inc., 1999. (Third Edition).
S. Yemini, S. Kliger, E. Mozes, Y. Yemini, and D. Ohsie. “High Speed & Robust Event Correlation”. IEEE Communications Magazine, May 1996.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Garofalakis, M., Rastogi, R. (2002). Network Data Mining and Analysis: The \( \mathcal{N}\mathcal{E}\mathcal{M}\mathcal{E}\mathcal{S}\mathcal{I}\mathcal{S} \) Project. In: Chen, MS., Yu, P.S., Liu, B. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2002. Lecture Notes in Computer Science(), vol 2336. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-47887-6_1
Download citation
DOI: https://doi.org/10.1007/3-540-47887-6_1
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-43704-8
Online ISBN: 978-3-540-47887-4
eBook Packages: Springer Book Archive