Abstract
We present an algorithm for predicting both common and rare events. Statistics show that occurrences of rare events are usually associated with common events. Therefore, we argue that predicting common events correctly is an important step toward correctly predicting rare events. The new algorithm assumes that frequencies of events exhibit a power-law distribution. The algorithm consists of components for detecting rare event types and common event types, while minimizing computational overhead. For experiments, we attempt to predict various fault types that can occur in distributed systems. The simulation study driven by the system failure data collected at the Pacific Northwest National Laboratory (PNNL) shows that fault-mitigation based on the new prediction mechanism provides 15 % better system availability than the existing prediction methods. Furthermore, it allows only 10 % of all possible system loss caused by rare faults in the simulation data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
JASON: The MIRTE Corporation: Rare Events. Technical report, JSR 09–108 (2009)
USENIX - Computer Failure Data Repository. http://cfdr.usenix.org/data.html#pnl
Kaelbling, L., Littman, M., Moore, A.: Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237–285 (1996)
Kivinen, J., Smola, A., Williamson, R.: Online Learning with Kernels (2003)
Stanley, H., Gabaix, X., Gopikrishnan, P., Plerou, V.: Economic fluctuations and statistical physics: quantifying extremely rare and less rare events in finance. Physica A 382(1), 286–301 (2007)
Kuznar, L., Frederick, W.: Environmental constraints and sigmoid utility: implications for value, risk sensitivity, and social status. Ecol. Econ. 46(2), 293–306 (2003)
Clauset, A., Shalizi, C., Newman, M.: Power-law distributions in empirical data. J. SIAM Rev. 51, 661–703 (2009)
Tenenbaum, J., Griffiths, T.: Generalization, similarity, and bayesian inference. J. Behav. Brain Sci. 24(4), 629–640 (2001)
Oakley, J., O’Hagan, A.: Bayesian inference for the uncertainty distribution of computer model outputs. J. Biometrika 89(4), 769–784 (2002). http://biomet.oupjournals.org/cgi/doi/10.1093/biomet/89.4.769
Silva, W., Milidi, R.: Algorithms for combining belief functions. Int. J. Approximate Reasoning 7(1–2), 73–94 (1992)
Smets, P.: Belief functions: the disjunctive rule of combination and the generalized Bayesian theorem. Int. J. Approximate Reasoning 9(1), 1–35 (1993)
Schein, A., Ungar, L.: Active learning for logistic regression: an evaluation. J. Mach. Learn. 68(3), 235–265 (2007). http://www.springerlink.com/index/10.1007/s10994-007-5019-5
Mitchell, T.: Machine Learning. McGraw Hill Series in Computer Science. McGraw-Hill, New York (1997)
King, G., Zeng, L.: Logistic regression in rare events data. Polit. Anal. 9(2), 137–163 (2001)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Jung, M., Oh, J.C. (2016). Rare Event-Prediction with a Hybrid Algorithm Under Power-Law Assumption. In: Fujita, H., Ali, M., Selamat, A., Sasaki, J., Kurematsu, M. (eds) Trends in Applied Knowledge-Based Systems and Data Science. IEA/AIE 2016. Lecture Notes in Computer Science(), vol 9799. Springer, Cham. https://doi.org/10.1007/978-3-319-42007-3_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-42007-3_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-42006-6
Online ISBN: 978-3-319-42007-3
eBook Packages: Computer ScienceComputer Science (R0)