Abstract
With the rapid development of the internet technology, we have entered the era of big data, people product massive amount of data on the internet. Through the analysis and data mining of Web logs, we can dig out valuable information such as user’s behavior preferences. But handling massive amounts of data, the traditional single machine can no longer meet the requirements. With the continuous development of big data technology, massive Hadoop log data can be analyzed through the framework of big data. In this paper, the Hadoop large data platform is built, the MapReduce programming model is used to preprocess the network log, and the Hive data warehouse is used to analyze the processed data in multi dimension. The analysis results have good guiding significance for mastering the user browsing behavior, promoting the promotion effect, optimizing the structure and experience of the website.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Edwards, M.F., Rambani, A.S., Zhu, Y.T., et al.: Design of hadoop-based framework for analytics of large synchrophasor datasets. Procedia Comput. Sci. 12(4), 254–258 (2012)
Chansler, R., Kuang, H., Radia, S., Shvachko, K.: The hadoop distributed file system. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST 2010) (MSST), Incline Village, NV, pp. 1–10 (2010). https://doi.org/10.1109/msst.2010.5496972
Dean, J.F., Ghemawat, S.S.: MapReduce: simplified data processing on large clusters. ACM 51(1), 107–113 (2008). https://doi.org/10.1145/1327452.1327492
Kotiyal, B.F., Kumar, A.S., Pant, B.T., et al.: Big data: mining of log file through hadoop. In: International Conference on Human Computer Interactions, pp. 1–7. IEEE (2014). https://doi.org/10.1109/ich-ci-ieee.2013.6887797
Wang, C.H., Tsai, C.T., Fan, C.C., et al.: A hadoop based weblog analysis system (2014). https://doi.org/10.1109/u-media.2014.9
Suguna, S.F., Vithya, M.S., Eunaicy, J.I.C.: Big data analysis in e-commerce system using Hadoop MapReduce. In: International Conference on Inventive Computation Technologies, pp. 1–6 (2017). https://doi.org/10.1109/inventive.2016.7824798
Du, J.F., Zhang, Z.S., Zhao, C.T.: Analysis on the digging of social network based on user search behavior. Int. J. Smart Home 10(5), 297–304 (2016). https://doi.org/10.14257/ij-sh.2016.10.5.27
Dewangan, S K, Pandey, S., Verma, T.: A distributed framework for event log analysis using MapReduce. In: International Conference on Advanced Communication Control and Computing Technologies, pp. 503–506. IEEE (2017). https://doi.org/10.1109/icaccct.2016.7831690
He, G.F., Ren, S.S., Yu, D.T., et al.: Analysis of enterprise user behavior on hadoop. In: Sixth International Conference on Intelligent Human-Machine Systems and Cybernetics, pp. 230–233. IEEE (2014). https://doi.org/10.1109/ihmsc.2014.158
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, W., Niu, X., Yang, C., Kang, H., Chen, Z., Wang, Y. (2019). An Analysis and Research of Network Log Based on Hadoop. In: Krömer, P., Zhang, H., Liang, Y., Pan, JS. (eds) Proceedings of the Fifth Euro-China Conference on Intelligent Data Analysis and Applications. ECC 2018. Advances in Intelligent Systems and Computing, vol 891. Springer, Cham. https://doi.org/10.1007/978-3-030-03766-6_61
Download citation
DOI: https://doi.org/10.1007/978-3-030-03766-6_61
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-03765-9
Online ISBN: 978-3-030-03766-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)