Abstract
Web pages contain a combination of informative contents and redundant contents which are primarily used for navigation, advertisements, copyright and decoration. Detecting templates correctly and precisely thus becomes a vital part for many applications. Methods for template detection have been studied extensively. However, they are insufficient to detect multiple templates in a Web site. In this paper, we propose a novel segment-based template detection method to identify templates. Our method works in three steps. First, for each Web site we construct a SSOM (Site-oriented Segment Object Model) tree from sampled pages in a Web collection, through aligning the pages’ SOM (Segment Object Model) trees. Second, we construct a template from the SSOM tree. At last, the template can be used to detect templates for the Web site: Given a page in the Web site, its template contents are gained with mapping between its SOM tree and the SSOM tree and classifying. The proposed method is evaluated with two mining tasks, Web page clustering and classification. It leads to a significant improvement when compared to previous template detection methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bar-Yossef, Z., Rajagopalan, S.: Template detection via data mining and its applications. In: Proceedings of the 11th International Conference on World Wide Web, pp. 580–591. ACM (2002)
Broder, A.Z., Glassman, S.C., Manasse, M.S., Zweig, G.: Syntactic clustering of the web. Computer Networks and ISDN Systems 29(8), 1157–1166 (1997)
Cai, D., Yu, S., Wen, J.R., Ma, W.Y.: Vips: a vision-based page segmentation algorithm. Tech. rep., Microsoft technical report, MSR-TR-2003-79 (2003)
Chakrabarti, D., Kumar, R., Punera, K.: Page-level template detection via isotonic smoothing. In: Proceedings of the 16th International Conference on World Wide Web, pp. 61–70. ACM (2007)
Debnath, S., Mitra, P., Pal, N., Giles, C.L.: Automatic identification of informative sections of web pages. IEEE Transactions on Knowledge and Data Engineering 17(9), 1233–1246 (2005)
Fernandes, D., de Moura, E.S., da Silva, A.S., Ribeiro-Neto, B., Braga, E.: A site oriented method for segmenting web pages. In: Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 215–224. ACM (2011)
Gibson, D., Punera, K., Tomkins, A.: The volume and evolution of web page templates. In: Special Interest Tracks and Posters of the 14th International Conference on World Wide Web, pp. 830–839. ACM (2005)
Kao, H.Y., Chen, M.S., Lin, S.H., Ho, J.M.: Entropy-based link analysis for mining web informative structures. In: Proceedings of the Eleventh International Conference on Information and Knowledge Management, pp. 574–581. ACM (2002)
Lin, S.H., Ho, J.M.: Discovering informative content blocks from web documents. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 588–593. ACM (2002)
Ma, L., Goharian, N., Chowdhury, A., Chung, M.: Extracting unstructured data from template generated web documents. In: Proceedings of the Twelfth International Conference on Information and Knowledge Management, pp. 512–515. ACM (2003)
Reis, D.D.C., Golgher, P.B., Silva, A., Laender, A.: Automatic web news extraction using tree edit distance. In: Proceedings of the 13th International Conference on World Wide Web, pp. 502–511. ACM (2004)
Song, R., Liu, H., Wen, J.R., Ma, W.Y.: Learning block importance models for web pages. In: Proceedings of the 13th International Conference on World Wide Web, pp. 203–211. ACM (2004)
Vieira, K., da Silva, A.S., Pinto, N., de Moura, E.S., Cavalcanti, J., Freire, J.: A fast and robust method for web page template detection and removal. In: Proceedings of the 15th ACM International Conference on Information and Knowledge Management, pp. 258–267. ACM (2006)
Wang, J., Lochovsky, F.H.: Data-rich section extraction from html pages. In: Proceedings of the Third International Conference on Web Information Systems Engineering, WISE 2002, pp. 313–322. IEEE (2002)
Yi, L., Liu, B.: Web page cleaning for web mining through feature weighting. In: International Joint Conference on Artificial Intelligence, vol. 18, pp. 43–50. Lawrence Erlbaum Associates Ltd. (2003)
Yi, L., Liu, B., Li, X.: Eliminating noisy information in web pages for data mining. In: Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 296–305. ACM (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Gao, B., Fan, Q. (2014). Multiple Template Detection Based on Segments. In: Perner, P. (eds) Advances in Data Mining. Applications and Theoretical Aspects. ICDM 2014. Lecture Notes in Computer Science(), vol 8557. Springer, Cham. https://doi.org/10.1007/978-3-319-08976-8_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-08976-8_3
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-08975-1
Online ISBN: 978-3-319-08976-8
eBook Packages: Computer ScienceComputer Science (R0)