The Role of Unlabeled Data in Supervised Learning
Most computational models of supervised learning rely only on labeled training examples, and ignore the possible role of unlabeled data. This is true both for cognitive science models of learning such as SOAR [Newell 1990] and ACT–R [Anderson, et al. 1995], and for machine learning and data mining algorithms such as decision tree learning and inductive logic programming (see, e.g., [Mitchell 1997]). In this paper we consider the potential role of unlabeled data in supervised learning. We present an algorithm and experimental results demonstrating that unlabeled data can significantly improve learning accuracy in certain practical problems. We then identify the abstract problem structure that enables the algorithm to successfully utilize this unlabeled data, and prove that unlabeled data will boost learning accuracy for problems in this class. The problem class we identify includes problems where the features describing the examples are redundantly sufficient for classifying the example; a notion we make precise in this paper. This problem class includes many natural learning problems faced by humans, such as learning a semantic lexicon over noun phrases in natural language, and learning to recognize objects from multiple sensor inputs. We argue that models of human and animal learning should consider more strongly the potential role of unlabeled data, and that many natural learning problems fit the class we identify.
Unable to display preview. Download preview PDF.
- A Anderson et al. , Production system models of complex cognition. In Proceedings of the Seventeenth Annual Conference of the Cognitive Science Society (pp. 9–12). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
- de Sa , Learning classification with unlabeled data, NIPS–6, 1994.Google Scholar
- de Sa and Ballard , Category learning through multi–modality sensing, Neural Computation 10(5), 1998.Google Scholar
- Newell , Unified theories of cognition. Cambridge, MA: Harvard University Press, 1990.Google Scholar
- Yarowsky , Unsupervised word sense disambiguation rivaling supervised methods. Proceedings of the 33 rd Annual Meeting of the ACL, pp. 189–196.Google Scholar