Encyclopedia of Database Systems

2018 Edition
| Editors: Ling Liu, M. Tamer Özsu

Fully Automatic Web Data Extraction

  • Cai-Nicolas ZieglerEmail author
Reference work entry
DOI: https://doi.org/10.1007/978-1-4614-8265-9_1159


Automatic wrapper induction; Web content extraction; Web information extraction


Web documents contain abundant hypertext markup information, both for indicating structure as well as for giving page rendering hints, next to informative textual content. Fully-automatic Web data extraction is geared towards extracting all relevant textual information from HTML documents, without requiring human intervention throughout the process. Commonly, two types of automatic Web extraction paradigms are distinguished in this vein. First, the extraction of one single block of informative content, e.g., in case of news pages, which is also referred to as page cleaning [4]. Second, the extraction of recurring patterns across multiple blocks, typically the case for the extraction of search engine results. In the latter case, the extraction system will commonly also assign labelsto the single atoms of each identified recurring block, such as the search result record’s title, snippet,...

This is a preview of subscription content, log in to check access.

Recommended Reading

  1. 1.
    Crescenzi V, Mecca G, Merialdo P. RoadRunner: towards automatic data extraction from large web sites. In: Proceedings of the 27th International Conference on Very Large Data Bases; 2001. p. 109–18.Google Scholar
  2. 2.
    Debnath S, Mitra P, Giles CL. Automatic extraction of informative blocks from webpages. In: Proceedings of the 2005 ACM Symposium on Applied Computing; 2005. p. 1722–6.Google Scholar
  3. 3.
    Glance N, Hurst M, Nigam K, Siegler M, Stockton R, Tomokiyo T. Deriving marketing intelligence from online discussion. In: Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2005. p. 419–28.Google Scholar
  4. 4.
    Hofmann K, Weerkamp W. Web corpus cleaning using content and structure. In: Fairon C, Naerts H, Kilgarrif A, de Schryver G, editors. Building and exploring web Corpora. vol. 4, UCL; 2007.p. 145–54.Google Scholar
  5. 5.
    Kovacevic M, Dilligenti M, Gori M, Milutinovic V. Recognition of common areas in a web page using a visualization approach. In: Proceedings of the 10th International Conference on Artificial Intelligence: Methodology, Systems, and Applications; 2002. p. 203–12.CrossRefGoogle Scholar
  6. 6.
    Kushmerick N, Weld D, Doorenbos R. Wrapper induction for information extraction. In: Proceedings of the 15th International Joint Conference on AI; 1997. p. 119–28.Google Scholar
  7. 7.
    Lin SH, Ho JM. Discovering informative content blocks from web documents. In: Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2002.p. 588–93.Google Scholar
  8. 8.
    Liu B, Grossman R, Zhai Y. Mining data records in web pages. In: Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2003. p. 601–6.Google Scholar
  9. 9.
    Muslea I, Minton S, Knoblock C. Hierarchical wrapper induction for semistructured information sources. Auton Agent Multi-Agent Syst. 2001;4(1–2):93–114.CrossRefGoogle Scholar
  10. 10.
    Simon K, Lausen G. ViPER: augmenting automatic information extraction with visual perceptions. In: Proceedings of the 14th ACM International Conference on Information and Knowledge Management; 2005. p. 381–8.Google Scholar
  11. 11.
    Ziegler CN, Skubacz M. Towards automated reputation and brand monitoring on the web. In: Proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence; 2006. p. 1066–70.Google Scholar
  12. 12.
    Ziegler CN, Skubacz M. Content extraction from news pages using particle swarm optimization on an linguistic and structural features. In: Proceedings of the 2007 IEEE/WIC/ACM International Conference on Web Intelligence; 2007. p. 242–9.Google Scholar
  13. 13.
    Zhao H, Meng W, Wu Z, Raghavan V, Yu C. Fully automatic wrapper generation for search engines. In: Proceedings of the 14th International World Wide Web Conference; 2005. p. 66–75.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Siemens AGMunichGermany

Section editors and affiliations

  • Georg Gottlob
    • 1
  1. 1.Computing Lab.Oxford Univ.OxfordUK