The Research and Implementation of a Distributed Crawler System Based on Apache Flink

  • Feng YeEmail author
  • Zongfei Jing
  • Qian Huang
  • Cheng Hu
  • Yong Chen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11338)


Web information is growing at an explosive rate. The crawling ability of the single-machine crawler becomes the bottleneck, so distributed web crawling techniques become the focus of research. However, the existing distributed web crawler systems have some shortcomings. Thread management for solving thread synchronization and resource competition is usually designed by using pure multi-thread asynchronous methods. But the execution of this mechanism observably reduces the performance. Moreover, the deduplication algorithms lead to low efficiency in dealing with large data sets or the problem of occupying large storage space. Therefore, we propose and implement a distributed web crawler system based on Apache Flink, which combines and integrates the Mesos/Marathon framework. It can make full use of the computing resources of the cluster and significantly improve the efficiency of the web crawler system. Taking the data of Netease news pages as an example, the experimental results show that the distributed crawler proposed has higher execution efficiency and reliability.


The distributed crawler Apache Flink URL filter The distributed crawling management 


  1. 1.
    Cho, J., Garcia-Molina, H.: Parallel crawlers. In: 11th International Conference Proceedings on World Wide Web, pp. 124–135. Association for Computing Machinery, Honolulu (2002)Google Scholar
  2. 2.
    Boldi, P., Codenotti, B., Santini, M., Vigna, S.: UbiCrawler: a scalable fully distributed web crawler. Softw.-Pract. Exp. 34(8), 711–726 (2004)CrossRefGoogle Scholar
  3. 3.
    Cambazoglu, B.B., Karaca, E., Kucukyilmaz, T., Turk, A., Aykanat, C.: Architecture of a grid-enabled web search engine. Inf. Process. Manag. 43(3), 609–623 (2007)CrossRefGoogle Scholar
  4. 4.
    Singh, A., Srivatsa, M., Liu, L., Miller, T.: Apoidea: a decentralized peer-to-peer architecture for crawling the world wide web. In: Callan, J., Crestani, F., Sanderson, M. (eds.) SIGIR 2003 Workshop on Distributed Information Retrieval 2003, LNCS, vol. 2924, pp. 126–142. Springer, Heidelberg (2004). Scholar
  5. 5.
    Loo, B.T., Krishnamurthy, S., Cooper, O.: Distributed Web Crawling over DHTs. University of California at Berkely, Berkely (2004)Google Scholar
  6. 6.
    Heydon, A., Najork, M.: Mercator: a scalable, extensible web crawler. World Wide Web-Internet Web Inf. Syst. 2(4), 219–229 (1999)CrossRefGoogle Scholar
  7. 7.
    Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system. In: the19th ACM Symposium Proceedings on Operating Systems Principles, pp. 29–43. Association for Computing Machinery, Lake George (2003)Google Scholar
  8. 8.
    Achsan, H.T.Y., Wibowo, W.C.: A fast distributed focused-web crawling. Proc. Eng. 69, 492–499 (2014)CrossRefGoogle Scholar
  9. 9.
    Huang, Q., Li, Q., Yan, Z., Fu, H.: A novel incremental parallel web crawler based on focused crawling. J. Comput. Inf. Syst. 9(6), 2461–2469 (2013)Google Scholar
  10. 10.
    Su, L., Wang, F.: Web crawler model of fetching data speedily based on hadoop distributed system. In: IEEE International Conference Proceedings on Software Engineering and Service Science, pp. 927–931. IEEE Computer Society, Beijing (2016)Google Scholar
  11. 11.
    Yang, Y., Yang, J.: Design and implementation of a multi-area distributed crawler based on Skipnet-YL network. In: Pacific-Asia Conference Proceedings on Circuits Communications and System, pp. 4–6. IEEE Computer Society, Wuhan (2011)Google Scholar
  12. 12.
    Friedman, E., Tzoumas, K.: Introduction to Apache Flink: Stream Processing for Real Time and Beyond. O’Reilly Media, Sebastopol (2016)Google Scholar
  13. 13.
    Deshpande, T.: Learning Apache Flink. Packt Publishing, Birmingham (2017)Google Scholar
  14. 14.
    Gen, L.: Research and Optimization of Web Crawler System under Distributed Environment. Beijing University of Posts and Telecommunications, Beijing (2014)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Feng Ye
    • 1
    • 3
    Email author
  • Zongfei Jing
    • 2
  • Qian Huang
    • 1
  • Cheng Hu
    • 1
  • Yong Chen
    • 3
  1. 1.Hohai UniversityNanjingPeople’s Republic of China
  2. 2.University of Chinese Academy of SciencesBeijingPeople’s Republic of China
  3. 3.Nanjing Longyuan Micro-Electronic CompanyNanjingPeople’s Republic of China

Personalised recommendations