Abstract
A big data application usually needs to build a pipeline on the top of workflow engine which connects relevant periodic workflow jobs. It’s crucial to timely alert pipeline issues, provide an issue diagnosis subsystem to find out root cause from a variety of sources, and measure pipeline/service by predefined metrics. In this paper, we identify three indispensable qualities monitor systems must fulfill namely timeliness, accuracy and flexibility. We find that the conventional monitoring tools lack at least one of three qualities, and introduce a general purpose MAD (Monitoring, Alerting and Diagnosis) system for big data applications to keep data freshness, collect measurement metrics to meet SLA.
Keywords
This work is specially supported by the Science and Technology Plan General Program of Beijing Municipal Education Commission (KM201510037001), Chinese Mountaineering Association (CMA2014-B-A04) and Intelligence Logistics System Beijing Key Laboratory (NO:BZ0211)
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Khanna, G., Varadharajan, P., et al.: Automated online monitoring of distributed applications through external monitors. IEEE Trans. Dependable Secure Comput. 3(2), 115–129 (2006)
Khanna, G., Cheng, M.Y., et al.: Automated rule-based diagnosis through a distributed monitor system. IEEE Trans. Dependable Secure Comput. 4(4), 266–279 (2007)
Chen, H., Jiang, G., et al.: Invariants based failure diagnosis in distributed computing systems. In: IEEE Symposium on Reliable Distributed Systems, pp: 160–166 (2010)
Joshi, K.R., Hiltunen, M.A., et al.: Probabilistic model-driven recovery in distributed systems. IEEE Trans. Dependable Secure Comput. 8(6), 913–928 (2011)
Ganglia - a scalable distributed monitoring system for high-performance computing systems. http://ganglia.sourceforge.net/
Nagios - the industry standard in IT infrastructure monitoring. http://www.nagios.org/
Splunk - the leading platform for Operational Intelligence. http://www.splunk.com/
Apache Hadoop. http://wiki.apache.org/hadoop
Jeffery, D., Sanjay, G.: MapReduce: simplified data processing on large clusters (2004). http://labs.google.com/papers/mapreduce.html
Hadoop - Yahoo! Lauches world’s largest hadoop production applications. http://developer.yahoo.com/blogs/hadoop/posts/2008/02/yahoo-worlds-largest-product-hadoop/
Ronnie, C., Bob, J., et al.: SCOPE: easy and efficient parallel processing of massive data sets. In: VLDB 2008, pp. 24–30 (2008)
Mohammad, I., Angelo, K.H. Oozie: torwards a scalable workflow management system for hadoop. In: SWEET 2012, 20 May 2012
Patrick, H., Mahadev, K., et al.: ZooKeeper: wait-free coordination for internet-scale. In: Usenix (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Shi, M., Yuan, R. (2015). MAD: A Monitor System for Big Data Applications. In: He, X., et al. Intelligence Science and Big Data Engineering. Big Data and Machine Learning Techniques. IScIDE 2015. Lecture Notes in Computer Science(), vol 9243. Springer, Cham. https://doi.org/10.1007/978-3-319-23862-3_30
Download citation
DOI: https://doi.org/10.1007/978-3-319-23862-3_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-23861-6
Online ISBN: 978-3-319-23862-3
eBook Packages: Computer ScienceComputer Science (R0)