How to Evaluate the Effectiveness of URL Normalizations

  • Sang Ho Lee
  • Sung Jin Kim
  • Hyo Sook Jeong
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3597)


Syntactically different URLs could represent the same web page on the World Wide Web, and duplicate representation for web pages causes web applications to handle a large amount of same web pages unnecessarily. In the standard communities, there are on-going efforts to define the URL normalization that helps eliminate duplicate URLs. On the other hand, there are research efforts to extend the standard URL normalization methods to reduce false negatives further while allowing false positives on a limited level. This paper presents a method that evaluates the effectiveness of a URL normalization method in terms of page loss/gain/change and the URL reduction. Over 94 million URLs were extracted from web pages for our experiment and interesting statistical results are reported in this paper.


Normalization Method Extended Normalization Uniform Resource Locator Path Component Page Request 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Burner, M.: Crawling Towards Eternity: Building an Archive of the World Wide Web. Web Techniques Magazine 2(5), 37–40 (1997)Google Scholar
  2. 2.
    Heydon, A., Najork, M.: Mercator: A Scalable, Extensible Web Crawler. International Journal of WWW 2(4), 219–229 (1999)Google Scholar
  3. 3.
    Kim, S.J., Lee, S.H.: Implementation of a Web Robot and Statistics on the Korean Web. LNCS, vol. 2713, pp. 341–350. Springer, Heidelberg (2003)Google Scholar
  4. 4.
    Shkapenyuk, V., Suel, T.: Design and Implementation of a High-performance Distributed Web Crawler. In: Proceedings of 18th Data Engineering Conference, pp. 357–368 (2002)Google Scholar
  5. 5.
    Berners-Lee, T., Fielding, R., Masinter, L.: Uniform Resource Identifiers (URI): Generic Syntax (2004),
  6. 6.
    Lee, S.H., Kim, S.J.: On URL Normalization. In: Proceedings of the Workshop on Ubiquitous Web Systems and Intelligence (2005) (to appear)Google Scholar
  7. 7.

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Sang Ho Lee
    • 1
  • Sung Jin Kim
    • 2
  • Hyo Sook Jeong
    • 1
  1. 1.School of ComputingSoongsil UniversitySeoulKorea
  2. 2.School of Computer Science and EngineeringSeoul National UniversitySeoulKorea

Personalised recommendations