Post-Supervised Template Induction for Information Extraction from Lists and Tables in Dynamic Web Sources

  • Z. Shi
  • E. Milios
  • N. Zincir-Heywood


Dynamic web sites commonly return information in the form of lists and tables. Although hand crafting an extraction program for a specific template is time-consuming but straightforward, it is desirable to automatically generate template extraction programs from examples of lists and tables in html documents. Supervised approaches have been shown to achieve high accuracy, but they require manual labelling of training examples, which is also time consuming. Fully unsupervised approaches, which extract rows and columns by detecting regularities in the data, cannot provide sufficient accuracy for practical domains. We describe a novel technique, Post-supervised Learning, which exploits unsupervised learning to avoid the need for training examples, while minimally involving the user to achieve high accuracy. We have developed unsupervised algorithms to extract the number of rows and adopted a dynamic programming algorithm for extracting columns. Our method achieves high performance with minimal user input compared to fully supervised techniques.


information extraction grammar induction template induction unsupervised learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Carrasco, R. and Oncina, J. (1994). Learning Stochastic Regular Grammars by Means of a State Merging Method. In Proceedings of the Second International Colloquium on Grammatical Inference and Applications (ICGI94), volume 862 of Lecture Notes on Artificial Intelligence, (pp. 139–152). Berlin, Springer Verlag.Google Scholar
  2. Cheeseman, P. and Stutz, J. (1996). Bayesian Classification (AUTOCLASS): Theory and Results. Advances in Knowledge Discovery and Data Mining (pp. 153–180). AAAI/MIT Press.Google Scholar
  3. Cohen, W. and Jensen, L. (2001). A Structured Wrapper Induction System for Extracting Information from Semi-Structured Documents. In Automatic Text Extraction and Mining workshop (ATEM-01), IJCAI-01, Seattle, WA, USA.Google Scholar
  4. Cormen, T., Leiserson, C., and Rivest, R. (1989). Introduction to Algorithms. Cambridge, Massachusetts: MIT Press.Google Scholar
  5. Degroot, M. and Schervish, M. (1975). Probability and Statistics. Cambridge, Massachusetts: Addison-Wesley Pub. Co.Google Scholar
  6. Deitel, H., Deitel, P., and Nieto, T. (2000). Internet and World Wide Web: How to Program. Upper Saddle River, NJ 07458: Prentice-Hall.Google Scholar
  7. Structures Learned by Stochastic Optimization. In AAAI/IAAI, (pp. 584–589).Google Scholar
  8. Hanson, J., Stutz, R., and Cheeseman, P. (1991). Bayesian Classification Theory. Technical report, NASA Ames TR FIA-90-12-7-01.Google Scholar
  9. Hearst, M. (1998). Information Integration. IEEE Intelligent Systems, 13(5), 12–16.CrossRefGoogle Scholar
  10. Knoblock, C., Lerman, K., Minton, S., and Muslea, I. (2001). A Machine Learning Approach to Accurately and Reliably Extracting Data from the Web. In IJCAI-2001 Workshop on Text Learning: Beyond Supervision, Seattle, WA, USA.Google Scholar
  11. Kushmerick, N. (1997). Wrapper Induction for Information Extraction. Technical report, Dept. of Computer Science, U. of Washington, TR UW-CSE-97-11-04.Google Scholar
  12. Kushmerick, N. (2000). Wrapper Induction: Efficiency and Expressiveness. Artificial Intelligence, 118: 15–68.CrossRefMathSciNetGoogle Scholar
  13. Lerman, K., Knoblock, C., and Minton, S. (2001). Automatic Data Extraction from Lists and Tables in Web Sources. In Automatic Text Extraction and Mining workshop (ATEM-01), IJCAI-01. Seattle, WA, USA.Google Scholar
  14. Lerman, K. and Minton, S. (2000). Learning the Common Structure of Data. In In Proceedings of the 17th National Conference on Artificial Intelligence (AAAI-2000), AAAI Press (pp. 609–614). Menlo Park.Google Scholar
  15. Lewis, H. and Papadimitriou, C. (1998). Elements of the Theory of Computation. Upper Saddle River, NJ 07458: Prentice-Hall.Google Scholar
  16. Liang, J., Milios, E., and Zincir-Heywood, N. (2002). A Robot Capable of Crawling Secure Dynamic Web Sites. Technical Report, Faculty of Computer Science, Dalhousie University, Halifax, Nova Scotia, Canada.Google Scholar
  17. McCallum, Andrew, Freitag, Dayne, and Pereira, Fernando. (2000). Maximum Entropy Markov Models for Information Extraction and Segmentation. In Proc. 17th International Conf. on Machine Learning (pp. 591–598). Morgan Kaufmann, San Francisco, CA.Google Scholar
  18. Muslea, I., Minton, S., and Knoblock, C. (2001). Hierarchical Wrapper Induction for Semistructured Information Sources. Journal of Autonomous Agents and Multi-Agent Systems, 4: 93–114.CrossRefGoogle Scholar
  19. Ng, H., Lim, C., and Koo, J. (1999). Learning to Recognize Tables in Free Text. In Proceedings of the 37th Annual Meeting of ACL (pp. 443–450).Google Scholar
  20. Pinto, D., McCallum, A., Lee, X., and Croft, W. 2003. Table Extraction Using Conditional Random Fields. In Proceedings of the 26th ACM SIGIR.Google Scholar
  21. Pinto, David, McCallum, Andrew, Wei, Xing, and Bruce Croft, W. (2003). Table Extraction Using Conditional Random Fields. In Proceedings of the ACM SIGIR.Google Scholar
  22. Pyreddy, Pallavi and Bruce Croft, W. (1997). TINTIN: A System for Retrieval in Text Tables. In ACM DL, (pp. 193–200).Google Scholar
  23. Shi, Z., Milios, E., and Zincir-Heywood, N. (2002). Post-Supervised Template Induction for Information Extraction From Lists and Tables in Dynamic Web Sources. Technical Report CS-2002-09, Dalhousie University, Faculty of Computer Science.Google Scholar
  24. Weisstein, Eric W. (2005). ȜVariation Coefficient.ȝ From MathWorld—A Wolfram Web Resource. Accessed on Apr. 15, 2005.
  25. (2001). Amazon online shopping site. Accessed on November 12, 2001.
  26. (2002). Travelocity travel site. Accessed on Oct. 23, 2002.
  27. (2002). WhitePages online directory service. Accessed on Oct. 23, 2002.

Copyright information

© Springer Science + Business Media, Inc. 2005

Authors and Affiliations

  • Z. Shi
    • 1
  • E. Milios
    • 1
  • N. Zincir-Heywood
    • 1
  1. 1.Faculty of Computer ScienceDalhousie UniversityHalifaxCanada

Personalised recommendations