Abstract
Today’s Web pages are commonly made up of more than merely one cohesive block of information. For instance, news pages from popular media channels such as Financial Times or Washington Post consist of no more than 30%-50% of textual news, next to advertisements, link lists to related articles, disclaimer information, and so forth.
However, for many search-oriented applications such as the detection of relevant pages for an in-focus topic, dissecting the actual textual content from surrounding page clutter is an essential task, so as to maintain appropriate levels of relevance signalling accuracy.
We present a novel approach that extracts real content from news Web pages in an unsupervised fashion. Our method is based on distilling linguistic and structural features from text blocks in HTML pages, having a Particle Swarm Optimizer (PSO) learn feature thresholds for optimal classification performance.
Empirical evaluations and benchmarks show that our approach works very well when applied to several hundreds of news pages from popular media in 5 languages.
Originally published in Proc. of the 2007 IEEE/WIC/ACMInt’l Conf. on Web Intelligence; the version at hand has been slightly extended
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval. Addison-Wesley, Reading (1999)
Baumgartner, R., Flesca, S., Gottlob, G.: Visual Web information extraction with Lixto. In: Proceedings of the 27th International Conference on Very Large Databases, Roma, Italy, pp. 119–128 (2001)
Cai, D., Yu, S., Wen, J.-R., Ma, W.-Y.: Block-based Web search. In: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 456–463. ACM Press, Sheffield, UK (2004)
Cunningham, H., Maynard, D., Bontcheva, K., Tablan, V.: GATE: An architecture for development of robust HLT applications. In: ACL 2002: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 168–175. Association for Computational Linguistics, Philadelphia, PA, USA (2001)
de Reis, D.C., Golgher, P., Silva, A., Laender, A.: Automatic Web news extraction using tree edit distance. In: Proceedings of the 13th International Conference on World Wide Web, pp. 502–511. ACM Press, New York (2004)
Glance, N., Hurst, M., Tomokiyo, T.: Blogpulse: Automated trend discovery for weblogs. In: Proceedings of the WWW 2004 Workshop on the Weblogging Ecosystem, New York, NY, USA (2004)
Goller, J.: STAN: Structural analysis for Web documents. In: Proceedings of the Second Internation Workshop on Web Document Analysis, Ediburg, UK, pp. 15–18 (2003)
Gupta, S., Kaiser, G., Grimm, P., Chiang, M., Starren, J.: Automating content extraction of HTML documents. World Wide Web 8(2), 179–224 (2005)
Gupta, S., Kaiser, G., Neistadt, D., Grimm, P.: DOM-based content extraction of HTML documents. In: Proceedings of the 12th International Conference on World Wide Web, pp. 207–214. ACM Press, Budapest (2003)
Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of IEEE International Conference on Neural Networks, pp. 1942–1948. IEEE Computer Society Press, Piscataway, NJ, USA (1995)
Kushmerick, N., Weld, D., Doorenbos, R.: Wrapper induction for information extraction. In: International Joint Conference on Artificial Intelligence, pp. 729–737. Morgan Kaufmann, Nagoya, Japan (1997)
Lin, S.-H., Ho, J.-M.: Discovering informative content blocks from Web documents. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 588–593. ACM Press, Edmonton (2002)
Ntoulas, A., Najork, M., Manasse, M., Fetterly, D.: Detecting spam Web pages through content analysis. In: Proceedings of the 15th International Conference on World Wide Web, pp. 83–92. ACM Press, Edinburgh (2006)
Quinlan, R.: Improved use of continuous attributes in C4.5. Journal of Artificial Intelligence Research 4, 77–90 (1996)
Simon, K., Lausen, G.: VIPER: Augmenting automatic information extraction with visual perceptions. In: Proceedings of the 2005 ACM CIKM Conference on Information and Knowledge Management, pp. 381–388. ACM Press, Bremen, Germany (2005)
Sun, A., Lim, E.-P.: Web unit mining: Finding and classifying subgraphs of Web pages. In: Proceedings of the 12th International Conference on Information and Knowledge Management, pp. 108–115. ACM Press, New Orleans, LA, USA (2003)
Tseng, Y.-F., Kao, H.-Y.: The mining and extraction of primary informative blocks and data objects from systematic Web pages. In: Proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence, pp. 370–373. IEEE Computer Society Press, Hong Kong, China (2006)
Yang, Y., Zhang, H.-J.: HTML page analysis based on visual cues. In: Proceedings of the Sixth International Conference on Document Analysis and Recognition, pp. 859–864. IEEE Computer Society Press, Washington, DC, USA (2001)
Zhai, Y., Liu, B.: Web data extraction based on partial tree alignment. In: Proceedings of the 14th International Conference on World Wide Web, pp. 76–85. ACM Press, Chiba, Japan (2005)
Ziegler, C.-N., Skubacz, M.: Towards automated reputation and brand monitoring on the web. In: Proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence, pp. 1066–1070. IEEE Computer Society Press, Hong Kong, China (2006)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2012 Springer-Verlag GmbH Berlin Heidelberg
About this chapter
Cite this chapter
Ziegler, CN., Skubacz, M. (2012). Content Extraction from News Pages Using Particle Swarm Optimization. In: Mining for Strategic Competitive Intelligence. Studies in Computational Intelligence, vol 406. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-27714-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-642-27714-6_8
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-27713-9
Online ISBN: 978-3-642-27714-6
eBook Packages: EngineeringEngineering (R0)