Analysis of the Web Graph Aggregated by Host and Pay-Level Domain

  • Agostino FunelEmail author
Conference paper
Part of the Studies in Computational Intelligence book series (SCI, volume 813)


In this paper the web is analyzed as a graph aggregated by host and pay-level domain (PLD). The web graph datasets, publicly available, have been released by the Common Crawl Foundation ( and are based on a web crawl performed during the period May-June-July 2017. The host graph has \(\sim \)1.3 billion nodes and \(\sim \)5.3 billion arcs. The PLD graph has \(\sim \)91 million nodes and \(\sim \)1.1 billion arcs. We study the distributions of degree and sizes of strongly/weakly connected components (SCC/WCC) focusing on power laws detection using statistical methods. The statistical plausibility of the power law model is compared with that of several alternative distributions. While there is no evidence of power law tails on host level, they emerge on PLD aggregation for indegree, SCC and WCC size distributions. Finally, we analyze distance-related features by studying the cumulative distributions of the shortest path lengths, and give an estimation of the diameters of the graphs.


Web graph Network analysis Structural network properties Network models 



The computing resources and the related technical support used for this work have been provided by CRESCO/ENEAGRID High Performance Computing infrastructure and its staff [10]. CRESCO/ENEAGRID High Performance Computing infrastructure is funded by ENEA, the Italian National Agency for New Technologies, Energy and Sustainable Economic Development and by Italian and European research programmes, see for information.


  1. 1.
    Alstott, J., Bullmore, E., Plenz, D.: Powerlaw: a Python package for analysis of heavy-tailed distributions. PLoS ONE 9(1), e85777 (2014)Google Scholar
  2. 2.
    Barabasi, A., Albert, R.: Emergence of scaling in random networks. Science 286, 509–512 (1999)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Broder, A., et al.: Graph structure in the web. Comput. Netw. 33(1–6), 309–320 (2000)Google Scholar
  4. 4.
    Clauset, A., Shalizzi, C.R., Newman, M.E.J.: Power law distributions in empirical data. SIAM Rev. 51(4), 661–703 (2009)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Donato, D., Leonardi, S., Millozzi, S., Tsaparas, P.: Mining the inner structure of the Web graph. J. Phys. A: Math. Theor. 41(22), 224017 (2008)Google Scholar
  6. 6.
    Kumar, R., Raghavan, P., Rajagopalan, S., Stata, R., Tomkins, A.: Trawling the Web for emerging cyber-communities. Comput. Netw. 31(11–16), 1481–1493 (1999)CrossRefGoogle Scholar
  7. 7.
    Leskovec, J., Sosič, R.: Snap: a general-purpose network analysis and graph-mining library. ACM Trans. Intell. Syst. Technol. (TIST) 8(1), 1 (2016)CrossRefGoogle Scholar
  8. 8.
    Meusel, R., Vigna, S., Lehmberg, O., Bizer, C.: The graph structure in the web - analyzed on different aggregation levels. J. Web Sci. 1, 33–47 (2015)CrossRefGoogle Scholar
  9. 9.
    Palmer, C.R., Gibbons, P.B., Faloutsos, C.: ANF: a fast and scalable tool for data mining in massive graphs. In: Proceedings of KDD ’02 (2002)Google Scholar
  10. 10.
    Ponti, G., et al.: The role of medium size facilities in the HPC ecosystem: the case of the new CRESCO4 cluster integrated in the ENEAGRID infrastructure. In: Proceedings of HPCS, pp. 1030–1033 (2014)Google Scholar
  11. 11.
    Serrano, M.A., Maguitman, A., Bogu\(\tilde{\text{n}}\)á, M., Fortunato, S., Vespignani, A.: Decoding the structure of the WWW: a comparative analysis of web crawls. ACM Trans. Web 1(2) (2007)Google Scholar
  12. 12.
    Zhu, J.J.H., Meng, T., Xie, Z., Li, G., Li, X.: A teapot graph and its hierarchical structure of the chinese web. In: Proceedings of WWW ’08 (2008)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.ENEARomeItaly

Personalised recommendations