Abstract
Dynamic web sites commonly return information in the form of lists and tables. Although hand crafting an extraction program for a specific template is time-consuming but straightforward, it is desirable to automatically generate template extraction programs from examples of lists and tables in html documents. Supervised approaches have been shown to achieve high accuracy, but they require manual labelling of training examples, which is also time consuming. Fully unsupervised approaches, which extract rows and columns by detecting regularities in the data, cannot provide sufficient accuracy for practical domains. We describe a novel technique, Post-supervised Learning, which exploits unsupervised learning to avoid the need for training examples, while minimally involving the user to achieve high accuracy. We have developed unsupervised algorithms to extract the number of rows and adopted a dynamic programming algorithm for extracting columns. Our method achieves high performance with minimal user input compared to fully supervised techniques.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Carrasco, R. and Oncina, J. (1994). Learning Stochastic Regular Grammars by Means of a State Merging Method. In Proceedings of the Second International Colloquium on Grammatical Inference and Applications (ICGI94), volume 862 of Lecture Notes on Artificial Intelligence, (pp. 139–152). Berlin, Springer Verlag.
Cheeseman, P. and Stutz, J. (1996). Bayesian Classification (AUTOCLASS): Theory and Results. Advances in Knowledge Discovery and Data Mining (pp. 153–180). AAAI/MIT Press.
Cohen, W. and Jensen, L. (2001). A Structured Wrapper Induction System for Extracting Information from Semi-Structured Documents. In Automatic Text Extraction and Mining workshop (ATEM-01), IJCAI-01, Seattle, WA, USA.
Cormen, T., Leiserson, C., and Rivest, R. (1989). Introduction to Algorithms. Cambridge, Massachusetts: MIT Press.
Degroot, M. and Schervish, M. (1975). Probability and Statistics. Cambridge, Massachusetts: Addison-Wesley Pub. Co.
Deitel, H., Deitel, P., and Nieto, T. (2000). Internet and World Wide Web: How to Program. Upper Saddle River, NJ 07458: Prentice-Hall.
Structures Learned by Stochastic Optimization. In AAAI/IAAI, (pp. 584–589).
Hanson, J., Stutz, R., and Cheeseman, P. (1991). Bayesian Classification Theory. Technical report, NASA Ames TR FIA-90-12-7-01.
Hearst, M. (1998). Information Integration. IEEE Intelligent Systems, 13(5), 12–16.
Knoblock, C., Lerman, K., Minton, S., and Muslea, I. (2001). A Machine Learning Approach to Accurately and Reliably Extracting Data from the Web. In IJCAI-2001 Workshop on Text Learning: Beyond Supervision, Seattle, WA, USA.
Kushmerick, N. (1997). Wrapper Induction for Information Extraction. Technical report, Dept. of Computer Science, U. of Washington, TR UW-CSE-97-11-04.
Kushmerick, N. (2000). Wrapper Induction: Efficiency and Expressiveness. Artificial Intelligence, 118: 15–68.
Lerman, K., Knoblock, C., and Minton, S. (2001). Automatic Data Extraction from Lists and Tables in Web Sources. In Automatic Text Extraction and Mining workshop (ATEM-01), IJCAI-01. Seattle, WA, USA.
Lerman, K. and Minton, S. (2000). Learning the Common Structure of Data. In In Proceedings of the 17th National Conference on Artificial Intelligence (AAAI-2000), AAAI Press (pp. 609–614). Menlo Park.
Lewis, H. and Papadimitriou, C. (1998). Elements of the Theory of Computation. Upper Saddle River, NJ 07458: Prentice-Hall.
Liang, J., Milios, E., and Zincir-Heywood, N. (2002). A Robot Capable of Crawling Secure Dynamic Web Sites. Technical Report, Faculty of Computer Science, Dalhousie University, Halifax, Nova Scotia, Canada.
McCallum, Andrew, Freitag, Dayne, and Pereira, Fernando. (2000). Maximum Entropy Markov Models for Information Extraction and Segmentation. In Proc. 17th International Conf. on Machine Learning (pp. 591–598). Morgan Kaufmann, San Francisco, CA.
Muslea, I., Minton, S., and Knoblock, C. (2001). Hierarchical Wrapper Induction for Semistructured Information Sources. Journal of Autonomous Agents and Multi-Agent Systems, 4: 93–114.
Ng, H., Lim, C., and Koo, J. (1999). Learning to Recognize Tables in Free Text. In Proceedings of the 37th Annual Meeting of ACL (pp. 443–450).
Pinto, D., McCallum, A., Lee, X., and Croft, W. 2003. Table Extraction Using Conditional Random Fields. In Proceedings of the 26th ACM SIGIR.
Pinto, David, McCallum, Andrew, Wei, Xing, and Bruce Croft, W. (2003). Table Extraction Using Conditional Random Fields. In Proceedings of the ACM SIGIR.
Pyreddy, Pallavi and Bruce Croft, W. (1997). TINTIN: A System for Retrieval in Text Tables. In ACM DL, (pp. 193–200).
Shi, Z., Milios, E., and Zincir-Heywood, N. (2002). Post-Supervised Template Induction for Information Extraction From Lists and Tables in Dynamic Web Sources. Technical Report CS-2002-09, Dalhousie University, Faculty of Computer Science.
Weisstein, Eric W. (2005). ȜVariation Coefficient.ȝ From MathWorld—A Wolfram Web Resource. http://www.mathworld.wolfram.com/VariationCoefficient.html Accessed on Apr. 15, 2005.
http://www.borders.com/ (2001). Amazon online shopping site. Accessed on November 12, 2001.
http://www.travelocity.com/ (2002). Travelocity travel site. Accessed on Oct. 23, 2002.
http://www.whitepages.com/ (2002). WhitePages online directory service. Accessed on Oct. 23, 2002.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Shi, Z., Milios, E. & Zincir-Heywood, N. Post-Supervised Template Induction for Information Extraction from Lists and Tables in Dynamic Web Sources. J Intell Inf Syst 25, 69–93 (2005). https://doi.org/10.1007/s10844-005-0861-z
Received:
Revised:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/s10844-005-0861-z