Abstract
Dynamic web sites commonly return information in the form of lists and tables. Although hand crafting an extraction program for a specific template is time-consuming but straightforward, it is desirable to automatically generate template extraction programs from examples of lists and tables in html documents. We describe a novel technique, Post-supervised Learning, which exploits unsupervised learning to avoid the need for training examples, while minimally involving the user to achieve high accuracy. We have developed unsupervised algorithms to extract the number of rows and adopted a dynamic programming algorithm for extracting columns. Our system, called TIDE (Template Induction for web Data Extraction), achieves high performance with minimal user input compared to fully supervised techniques.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Deitel, H., Deitel, P., Nieto, T.: Internet and World Wide Web: How to Program. Prentice-Hall, Upper Saddle River, NJ 07458 (2000)
Lerman, K., Knoblock, C., Minton, S.: Automatic data extraction from lists and tables in Web sources. In: Automatic Text Extraction and Mining workshop (ATEM-01), IJCAI-01, Seattle, WA, USA (2001)
Knoblock, C., Lerman, K., Minton, S., Muslea, I.: A machine learning approach to accurately and reliably extracting data from the web. In: IJCAI-2001 Workshop on Text Learning: Beyond Supervision, Seattle, WA, USA (2001)
Kushmerick, N.: Wrapper induction: Efficiency and expressiveness. Artificial Intelligence 118 (2000) 15–68
Kushmerick, N.: Wrapper induction for information extraction. Technical report, Dept. of Computer Science, U. of Washington, TR UW-CSE-97-11-04 (1997)
Muslea, I., Minton, S., Knoblock., C.: Hierarchical wrapper induction for semistructured information sources. Journal of Autonomous Agents and Multi-Agent Systems 4 (2001) 93–114
Lerman, K., Minton, S.: Learning the common structure of data. In: In Proceedings of the 17th National Conference on Artificial Intelligence (AAAI-2000), AAAI Press, Menlo Park (2000) 609–614
Cheeseman, P., Stutz, J.: Bayesian classification (AUTOCLASS): Theory and results. Advances in Knowledge Discovery and Data Mining, AAAI/MIT Press (1996) 153–180
Hanson, J., Stutz, R., Cheeseman, P.: Bayesian classification theory. Technical report, NASA Ames TR FIA-90-12-7-01 (1991)
Carrasco, R., Oncina, J.: Learning stochastic regular grammars by means of a state merging method. In: Proceedings of the Second International Colloquium on Grammatical Inference and Applications (ICGI94). Volume 862 of Lecture Notes on Artificial Intelligence., Berlin, Springer Verlag (1994) 139–152
Lewis, H., Papadimitriou, C.: Elements of the Theory of Computation. Prentice-Hall, Upper Saddle River, NJ 07458 (1998)
Degroot, M., Schervish, M.: Probability and Statistics. Addison-Wesley Pub. Co., Cambridge, Massachusetts (1975)
Rozgonyi, T.G.: Statistics for Engineers. http://engineering.uow.edu.au/Courses/Stats/File1586.html, (Accessed on Oct. 28, 2002)
http://www.travelocity.com/: Travelocity travel site. (Accessed on Oct. 23, 2002)
Cormen, T., Leiserson, C., Rivest, R.: Introduction to Algorithms. MIT Press, Cambridge, Massachusetts (1989) 276, 277
Liang, J., Milios, E., Zincir-Heywood, N.: A robot capable of crawling secure dynamic web sites. Technical report, Faculty of Computer Science, Dalhousie University, Halifax, Nova Scotia, Canada (2002)
Cohen, W., Jensen, L.: A structured wrapper induction system for extracting information from semi-structured documents. In: Automatic Text Extraction and Mining workshop (ATEM-01), IJCAI-01, Seattle, WA, USA (2001)
http://www.borders.com/: Amazon online shopping site. (Accessed on November 12, 2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Shi, Z., Milios, E., Zincir-Heywood, N. (2003). Post-supervised Template Induction for Dynamic Web Sources. In: Xiang, Y., Chaib-draa, B. (eds) Advances in Artificial Intelligence. Canadian AI 2003. Lecture Notes in Computer Science, vol 2671. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44886-1_21
Download citation
DOI: https://doi.org/10.1007/3-540-44886-1_21
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40300-5
Online ISBN: 978-3-540-44886-0
eBook Packages: Springer Book Archive