Advertisement

Main Content Extraction from Web Documents Using Text Block Context

  • Myungwon Kim
  • Youngjin Kim
  • Wonmoon Song
  • Ara Khil
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8056)

Abstract

Due to various Web authoring tools, the new web standards, and improved web accessibility, a wide variety of Web contents are being produced very quickly. In such an environment, in order to provide appropriate Web services to users’ needs it is important to quickly and accurately extract relevant information from Web documents and remove irrelevant contents such as advertisements. In this paper, we propose a method that extracts main content accurately from HTML Web documents. In the method, a decision tree is built and used to classify each block of text whether it is a part of the main content. For classification we use contextual features around text blocks including word density, link density, HTML tag distribution, and distances between text blocks. We experimented with our method using a published data set and a data set that we collected. The experiment results show that our method performs 19% better in F-measure compared to the existing best performing method.

Keywords

Web Document Analysis Content Extraction Tag Distribution Block Distance Context 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
  2. 2.
    Deng, C., Shipeng, Y., Ji-Rong, W., Wei-Ying, M.: VIPS: a Vision-based Page Segmentation Algorithm. Microsoft Technical Report(MSR-TR-2003-79) (2003)Google Scholar
  3. 3.
    Suhit, G., Gail, E.K., David, N., Peter, G.: DOM-based Content Extraction of HTML Documents. In: 12th International Conference on World Wide Web, pp. 207–214 (2003)Google Scholar
  4. 4.
    Suhit, G., Gail, E.K., Peter, G., Michael, F.C., Justin, S.: Automating Content Extraction of HTML Documents. World Wide Web 8(2), 179–224 (2005)CrossRefGoogle Scholar
  5. 5.
    Jeff, P., Dan, R.: Extracting Article Text from the Web with Maximum Subsequence Segmentation. In: The 18th International Conference on World Wide Web, pp. 971–980 (2009)Google Scholar
  6. 6.
    Stefan, E.: A lightweight and efficient tool for cleaning Web pages. In: The 6th International Conference on Language Resources and Evaluation (2008)Google Scholar
  7. 7.
    Stefan, E.: StupidOS: A high-precision approach to boilerplate removal. In: Building and Exploring Web Corpora: Proceedings of the 3rd Web as Corpus Workshop, pp. 123–133 (2007)Google Scholar
  8. 8.
    Young, S., Hasan, J., Farshad, F.: Autonomic Wrapper Induction using Minimal Type System from Web Data. In: Artificial Intelligence, pp. 130–135 (2005)Google Scholar
  9. 9.
    Christian, K., Peter, F., Wolfgang, N.: Boilerplate Detection using Shallow Text Features. In: The Third ACM International Conference on Web Search and Data Mining, pp. 441–450 (2010)Google Scholar
  10. 10.
    Jian, F., Ping, L., Suk Hwan, L., Sam, L., Parag, J., Jerry, L.: Article Clipper- A System for Web Article Extraction. In: 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 743–746 (2011)Google Scholar
  11. 11.
    Tim, W., William, H.H., Jiawei, H.: CETR - Content Extraction via Tag Ratios. In: 19th International Conference on World Wide Web, pp. 971–980 (2010)Google Scholar
  12. 12.
    Tim, W., William, H.H.: Text Extraction from the Web via Text-to-Tag Ratio. In: The 19th International Conference on Database and Expert Systems Application, pp. 23–28 (2008)Google Scholar
  13. 13.
  14. 14.
    W3C (February 2013), http://www.w3.org/TR/html401/
  15. 15.
    Jiawei, H., Micheline, K.: Data Mining: Concepts and Techniques. Morgan Kaufmann (2006)Google Scholar
  16. 16.
    Ian, H.W., Eibe, F.: Data Mining: Practical Machine Learning Tools and Techniques. Elsevier (2005)Google Scholar
  17. 17.
    Waikato Univ. (February 2013), http://www.cs.waikato.ac.nz/ml/weka/
  18. 18.
    Andy, C., Marc G.: (February 2012), http://nekohtml.sourceforge.net/
  19. 19.
    L3S Research Center (February 2013), http://www.l3s.de/~kohlschuetter/boilerplate/
  20. 20.

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Myungwon Kim
    • 1
  • Youngjin Kim
    • 1
  • Wonmoon Song
    • 1
  • Ara Khil
    • 1
  1. 1.Dept. of ComputingSoongsil UniversitySeoulKorea Republic

Personalised recommendations