From machine learning to machine reasoning
- 4.4k Downloads
A plausible definition of “reasoning” could be “algebraically manipulating previously acquired knowledge in order to answer a new question”. This definition covers first-order logical inference or probabilistic inference. It also includes much simpler manipulations commonly used to build large learning systems. For instance, we can build an optical character recognition system by first training a character segmenter, an isolated character recognizer, and a language model, using appropriate labelled training sets. Adequately concatenating these modules and fine tuning the resulting system can be viewed as an algebraic operation in a space of models. The resulting model answers a new question, that is, converting the image of a text page into a computer readable text.
This observation suggests a conceptual continuity between algebraically rich inference systems, such as logical or probabilistic inference, and simple manipulations, such as the mere concatenation of trainable learning systems. Therefore, instead of trying to bridge the gap between machine learning systems and sophisticated “all-purpose” inference mechanisms, we can instead algebraically enrich the set of manipulations applicable to training systems, and build reasoning capabilities from the ground up.
KeywordsMachine learning Reasoning Recursive networks
- Ahmed, A., Yu, K., Xu, W., Gong, Y., & Xing, E. P. (2008). Training hierarchical feed-forward visual recognition models using transfer learning from pseudo tasks. In Proc. 10th European conference on computer vision (ECCV). Google Scholar
- Bakır, G. H., Hofmann, T., Schölkopf, B., Smola, A. J., Taskar, B., & Vishwanathan, S. V. N. (Eds.) (2007). Predicting structured data. Cambridge: MIT Press. Google Scholar
- Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of deep networks. In Advances in neural information processing systems (Vol. 19, pp. 153–160). Cambridge: MIT Press. Google Scholar
- Bordes, A., Weston, J., Collobert, R., & Bengio, Y. (2011). Learning structured embeddings of knowledge bases. In Proc. 25th conference on artificial intelligence (AAAI). Google Scholar
- Bottou, L. (2008). Artificial intelligence in seven years? Seminar presentation, University of Montreal, June 2008. http://www.iro.umontreal.ca/~lisa/seminaires/26-06-2008.html.
- Bottou, L. (2011). From machine learning to machine reasoning, February 2011. arXiv:1102.1808v3.
- Bottou, L., & Gallinari, P. (1991). A framework for the cooperation of learning algorithms. In Advances in neural information processing systems (Vol. 3). San Mateo: Morgan Kaufmann. Google Scholar
- Bottou, L., LeCun, Y., & Bengio, Y. (1997). Global training of document processing systems using graph transformer networks. In Proc. of computer vision and pattern recognition (pp. 489–493). New York: IEEE Press. Google Scholar
- Buntine, W. (1994). Operations for learning with graphical models. The Journal of Artificial Intelligence Research, 2, 159–225. Google Scholar
- Collobert, R. (2011). Deep learning for efficient discriminative parsing. In Proc. artificial intelligence and statistics (AISTAT). Google Scholar
- Collobert, R., & Weston, J. (2007). Fast semantic extraction using a novel neural network architecture. In Proc. 45th annual meeting of the association of computational linguistics (ACL) (pp. 560–567). Google Scholar
- Etter, V. (2009). Semantic vector machines. Master’s thesis, EPFL. Google Scholar
- Friedman, N., Getoor, L., Koller, D., & Pfeffer, A. (1999). Learning probabilistic relational models. In Proc. sixteenth international joint conference on artificial intelligence (pp. 1300–1307). Google Scholar
- Grangier, D., Bottou, L., & Collobert, R. (2009). Deep convolutional networks for scene parsing. ICML 2009 Deep Learning Workshop. http://david.grangier.info/scene_parsing.
- Hoiem, D., Stein, A., Efros, A. A., & Hebert, M. (2007). Recovering occlusion boundaries from a single image. In Proc. international conference on computer vision (CVPR). Google Scholar
- LeCun, Y., Bottou, L., & HuangFu, J. (2004). Learning methods for generic object recognition with invariance to pose and lighting. In Proc. computer vision and pattern recognition. Google Scholar
- Lighthill, J. (1973). Artificial intelligence: a general survey. In Artificial intelligence: a paper symposium. Science Research Council. Google Scholar
- Lonardi, S., Sperduti, A., & Starita, A. (1994). Encoding pyramids by labeling RAAM. In Proc. neural networks for signal processing. Google Scholar
- Miller, M. (2006). Personal communication. Google Scholar
- Neville, J., & Jensen, D. (2003). Collective classification with relational dependency networks. In Proc. second international workshop on multi-relational data mining (pp. 77–91). Google Scholar
- NIPS (1987–2010). Advances in neural processing information systems. Volumes 0 to 22. Google Scholar
- Paccanaro, A., & Hinton, G. E. (2001). Learning hierarchical structures with linear relational embedding. In Advances in neural information processing systems (Vol. 14). Cambridge: MIT Press. Google Scholar
- Pearl, J. (1988). Probabilistic reasoning in intelligent systems. San Mateo: Morgan Kaufmann. Google Scholar
- Pearl, J. (2000). Causality: models, reasoning, and inference. Cambridge: Cambridge University Press. Google Scholar
- Piaget, J. (1937). La construction du réel chez l’enfant. Neuchatel: Delachaux et Niestlé. Google Scholar
- Plate, T. (1994). Distributed Representations and Nested Compositional Structure. PhD thesis, Department of Computer Science, University of Toronto. Google Scholar
- Riesenhuber, M., & Poggio, T. (2003). How visual cortex recognizes objects: the tale of the standard model. The Visual Neurosciences, 2, 1640–1653. Google Scholar
- Robinson, J. A. (1965). A machine-oriented logic based on the resolution principle. Communications of the ACM, 5, 23–41. Google Scholar
- Socher, R., Ng, A., & Manning, C. (2010). Learning continuous phrase representations and syntactic parsing with recursive neural networks. NIPS Deep Learning workshop presentation, November 2010. http://deeplearningworkshopnips2010.wordpress.com/schedule/oral1.
- Socher, R., Lin, C., Ng, A. Y., & Manning, C. D. (2011). Parsing natural scenes and natural language with recursive neural networks. In Proc. 28th international conference on machine learning (ICML). Google Scholar
- Sperduti, A. (1994). In advances in neural information processing systems: Vol. 5. Encoding labeled graphs by labeling RAAM. San Mateo: Morgan Kaufmann. Google Scholar
- Ponce, J., Lazebnik, S., & Schmid, C. (2006). Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In Proc. computer vision and pattern recognition (Vol. II, pp. 2169–2178). Google Scholar
- Welling, M. (2009). Herding dynamic weights to learn. In Proc. 26th international conference on machine learning (pp. 1121–1128). Google Scholar