Post-Supervised Template Induction for Information Extraction from Lists and Tables in Dynamic Web Sources
- 52 Downloads
Dynamic web sites commonly return information in the form of lists and tables. Although hand crafting an extraction program for a specific template is time-consuming but straightforward, it is desirable to automatically generate template extraction programs from examples of lists and tables in html documents. Supervised approaches have been shown to achieve high accuracy, but they require manual labelling of training examples, which is also time consuming. Fully unsupervised approaches, which extract rows and columns by detecting regularities in the data, cannot provide sufficient accuracy for practical domains. We describe a novel technique, Post-supervised Learning, which exploits unsupervised learning to avoid the need for training examples, while minimally involving the user to achieve high accuracy. We have developed unsupervised algorithms to extract the number of rows and adopted a dynamic programming algorithm for extracting columns. Our method achieves high performance with minimal user input compared to fully supervised techniques.
Keywordsinformation extraction grammar induction template induction unsupervised learning
Unable to display preview. Download preview PDF.
- Carrasco, R. and Oncina, J. (1994). Learning Stochastic Regular Grammars by Means of a State Merging Method. In Proceedings of the Second International Colloquium on Grammatical Inference and Applications (ICGI94), volume 862 of Lecture Notes on Artificial Intelligence, (pp. 139–152). Berlin, Springer Verlag.Google Scholar
- Cheeseman, P. and Stutz, J. (1996). Bayesian Classification (AUTOCLASS): Theory and Results. Advances in Knowledge Discovery and Data Mining (pp. 153–180). AAAI/MIT Press.Google Scholar
- Cohen, W. and Jensen, L. (2001). A Structured Wrapper Induction System for Extracting Information from Semi-Structured Documents. In Automatic Text Extraction and Mining workshop (ATEM-01), IJCAI-01, Seattle, WA, USA.Google Scholar
- Cormen, T., Leiserson, C., and Rivest, R. (1989). Introduction to Algorithms. Cambridge, Massachusetts: MIT Press.Google Scholar
- Degroot, M. and Schervish, M. (1975). Probability and Statistics. Cambridge, Massachusetts: Addison-Wesley Pub. Co.Google Scholar
- Deitel, H., Deitel, P., and Nieto, T. (2000). Internet and World Wide Web: How to Program. Upper Saddle River, NJ 07458: Prentice-Hall.Google Scholar
- Structures Learned by Stochastic Optimization. In AAAI/IAAI, (pp. 584–589).Google Scholar
- Hanson, J., Stutz, R., and Cheeseman, P. (1991). Bayesian Classification Theory. Technical report, NASA Ames TR FIA-90-12-7-01.Google Scholar
- Knoblock, C., Lerman, K., Minton, S., and Muslea, I. (2001). A Machine Learning Approach to Accurately and Reliably Extracting Data from the Web. In IJCAI-2001 Workshop on Text Learning: Beyond Supervision, Seattle, WA, USA.Google Scholar
- Kushmerick, N. (1997). Wrapper Induction for Information Extraction. Technical report, Dept. of Computer Science, U. of Washington, TR UW-CSE-97-11-04.Google Scholar
- Lerman, K., Knoblock, C., and Minton, S. (2001). Automatic Data Extraction from Lists and Tables in Web Sources. In Automatic Text Extraction and Mining workshop (ATEM-01), IJCAI-01. Seattle, WA, USA.Google Scholar
- Lerman, K. and Minton, S. (2000). Learning the Common Structure of Data. In In Proceedings of the 17th National Conference on Artificial Intelligence (AAAI-2000), AAAI Press (pp. 609–614). Menlo Park.Google Scholar
- Lewis, H. and Papadimitriou, C. (1998). Elements of the Theory of Computation. Upper Saddle River, NJ 07458: Prentice-Hall.Google Scholar
- Liang, J., Milios, E., and Zincir-Heywood, N. (2002). A Robot Capable of Crawling Secure Dynamic Web Sites. Technical Report, Faculty of Computer Science, Dalhousie University, Halifax, Nova Scotia, Canada.Google Scholar
- McCallum, Andrew, Freitag, Dayne, and Pereira, Fernando. (2000). Maximum Entropy Markov Models for Information Extraction and Segmentation. In Proc. 17th International Conf. on Machine Learning (pp. 591–598). Morgan Kaufmann, San Francisco, CA.Google Scholar
- Ng, H., Lim, C., and Koo, J. (1999). Learning to Recognize Tables in Free Text. In Proceedings of the 37th Annual Meeting of ACL (pp. 443–450).Google Scholar
- Pinto, D., McCallum, A., Lee, X., and Croft, W. 2003. Table Extraction Using Conditional Random Fields. In Proceedings of the 26th ACM SIGIR.Google Scholar
- Pinto, David, McCallum, Andrew, Wei, Xing, and Bruce Croft, W. (2003). Table Extraction Using Conditional Random Fields. In Proceedings of the ACM SIGIR.Google Scholar
- Pyreddy, Pallavi and Bruce Croft, W. (1997). TINTIN: A System for Retrieval in Text Tables. In ACM DL, (pp. 193–200).Google Scholar
- Shi, Z., Milios, E., and Zincir-Heywood, N. (2002). Post-Supervised Template Induction for Information Extraction From Lists and Tables in Dynamic Web Sources. Technical Report CS-2002-09, Dalhousie University, Faculty of Computer Science.Google Scholar
- Weisstein, Eric W. (2005). ȜVariation Coefficient.ȝ From MathWorld—A Wolfram Web Resource. http://www.mathworld.wolfram.com/VariationCoefficient.html Accessed on Apr. 15, 2005.
- http://www.borders.com/ (2001). Amazon online shopping site. Accessed on November 12, 2001.
- http://www.travelocity.com/ (2002). Travelocity travel site. Accessed on Oct. 23, 2002.
- http://www.whitepages.com/ (2002). WhitePages online directory service. Accessed on Oct. 23, 2002.