Abstract
Computer systems generate large amounts of event logs related to various operational aspects (positive and negative). Extracting from them useful information (e.g. targeted at dependability and resilience issues) is a challenging problem widely discussed in the literature and still needing deeper studies. We have developed a new holistic approach using enhanced event classification (based on original text mining algorithms) combined with multidimensional statistical analysis of various properties in vocabulary (words, phrases), time, spatial, local and global correlations. It has been incorporated in the developed tools and verified on event data sets collected from different computers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Berry, M.W., Kogan, J.: Text Mining Applications and Theory. Wiley, Chichester (2010)
Chen, C., Singh, N., Yajnik, D.: Log analytics for dependable enterprise telephony. In: Proceedings of 9th European Dependable Computing Conference, pp. 94–101 (2012)
Chuvakin, A., Schmid, K., Phillips, C., Moulder, P.: Logging and log management, the authoritative guide to understanding the concepts surrounding logging and log management. Elsevier (2013). http://dx.doi.org/10.1016/B978-1-59-749635-3.00024-5
Cinque, M., Cotroneo, D, Della, Corte, R., Pecchia, A.: Assessing direct monitoring techniques to analyze failures of critical industrial systems. In: Proceedings of IEEE 25th International Symposium on Software Reliability Engineering, pp. 212–222 (2014)
Di Martino, C., Kalbarczyk, Z., Kramer, W., Iyer, R.: Measuring and understanding extreme-scale application resilience: a field study of 5,000,000 HPC application runs. In: IEEEE/IFIP International Conference on Dependable Systems and Networks, pp. 25–36 (2015)
Fu, X., Ren, R., Zhan, J., Zhou, W., Jia, Z., Lu, G.: Logmaster: mining event correlations in logs of large-scale cluster systems. In: Proceedings of the 31st IEEE Symposium on Reliable Distributed Systems, pp. 71–80 (2012)
Gainaru, A., Cappelo, F., Snir, M., Kramer, W.: Fault prediction under the microscope: a closer look into HPC systems. In: Proceedings of the International Conference for High Performance Computing, pp 1–12 (2012)
He, P., Zhu, J., He, S., Li, J., Lyu, M.R.: An evaluation study on log parsing and its use in log mining. In: Proceedings of the 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, pp. 654–661 (2016)
He, S., Zhu, J., He, P., Lyu, M.R.: Experience report: system log analysis for anomaly detection. In: Proceedings of the International Symposium on Software Reliability Engineering (ISSRE), pp 207–218 (2016)
Kubacki, M., Sosnowski, J.: Multidimensional log analysis. In: Proceedings of European Dependable Computing Conference, pp. 193–196 (2016)
Law, D., Gruss, R., Abrahams, A.S.: Automated defect discovery for dishwasher appliances from online consumer reviews. Expert Syst. Appl. 67, 84–94 (2017)
Lin, Q., Zhang, H., Lou, J.G., Zhang, Y., Chen, X.: Log clustering based problem identification for online service systems. In: Proceedings of the 38th International Conference on Software Engineering (2016)
Lou, J., Fu, Q., Yang, S., Xu, Y., Li, J.: Mining invariants from console logs for system problem detection. In: Proceedings of the USENIX Annual Technical Conference (2010)
Makanju, A., Zincir-Heywoodet, A.N., Milios, E.E.: A lightweight algorithm for message type extraction in system application logs. IEEE Trans. Knowl. Data Eng. 24(11), 1921–1936 (2012)
Nagappan, M., Robinson, B.: Creating operational profiles of software systems by transforming their log files to directed cyclic graphs. In: Proceedings of the 6th International Workshop on Traceability in Emerging Forms of Software Engineering, pp. 54–57. ACM (2011)
Peccia, A., Cinque, M., Carrozza, G., Cotroneo, D.: Industry practices and event logging: assessment of a critical software development process. In: Proceedings of the IEEE/ACM 37th IEEE International Conference on Software Engineering, pp. 169–178 (2015)
Sosnowski, J., Kubacki, M., Krawczyk, H.: Monitoring event logs within a cluster system. In: Zamojski, W., Mazurkiewicz, J., Sugier, J., Walkowiak, T., Kacprzyk, J. (eds.) Complex Systems and Dependability. AINSC, vol. 170, pp. 259–271. Springer, Heidelberg (2013). doi:10.1007/978-3-642-30662-4_17
Stearley, J., Oliner, A.: Bad words: finding faults in spirit’s syslogs. In: Proceedings of International Conference on Cluster Computing and the Grid (2008)
Tang, L., Li, T., Perng, C.-S.: LogSig: generating system events from raw textual logs. In: Proceedings of the 20th ACM International Conference on Information and Knowledge Management, pp. 785–794 (2011)
Xu, W., Huang, L., Fox, A., Patterson, D., Jordon, M.I.: Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM Symposium on Operating Systems Principles, pp. 117–132 (2009)
Zhu, J., He, P., Fu, Q., Zhang, H., Lyu, R., Zhang, D.: Learning to log: helping developers make informed logging decisions. In: Proceedings of the 37th International Conference on Software Engineering, pp. 415–424 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Kubacki, M., Sosnowski, J. (2017). Holistic Processing and Exploring Event Logs. In: Romanovsky, A., Troubitsyna, E. (eds) Software Engineering for Resilient Systems. SERENE 2017. Lecture Notes in Computer Science(), vol 10479. Springer, Cham. https://doi.org/10.1007/978-3-319-65948-0_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-65948-0_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-65947-3
Online ISBN: 978-3-319-65948-0
eBook Packages: Computer ScienceComputer Science (R0)