Semantic Vector Spaces for Broadening Consideration of Consequences

  • Douglas Summers-StayEmail author


Reasoning systems with too simple a model of the world and human intent are unable to consider potential negative side effects of their actions and modify their plans to avoid them (e.g., avoiding potential errors). However, hand-encoding the enormous and subtle body of facts that constitutes common sense into a knowledge base has proved too difficult despite decades of work. Distributed semantic vector spaces learned from large text corpora, on the other hand, can learn representations that capture shades of meaning of common-sense concepts and perform analogical and associational reasoning in ways that knowledge bases are too rigid to perform, by encoding concepts and the relations between them as geometric structures. These have, however, the disadvantage of being unreliable, poorly understood, and biased in their view of the world by the source material. This chapter will discuss how these approaches may be brought together in a way that combines the best properties of each for understanding the world and human intentions in a richer way.


  1. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete problems in AI safety. arXiv preprint arXiv:1606.06565.Google Scholar
  2. Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450.Google Scholar
  3. Blouw, P and Eliasmith, C. (2005) A neurally plausible encoding of word order information into a semantic vector space. 35th Annual conference of the cognitive science society Vol. 1910.Google Scholar
  4. Dash, D., Voortman, M., and De Jongh, M. (2013, August). Sequences of Mechanisms for Causal Reasoning in Artificial Intelligence. In IJCAIGoogle Scholar
  5. Deerwester, S., et al, Improving Information Retrieval with Latent Semantic Indexing, Proceedings of the 51st Annual Meeting of the American Society for Information Science 25, 1988, pp. 36–40.Google Scholar
  6. Dietterich, T. G., and Horvitz, E. J. (2015). Rise of concerns about AI: reflections and directions. Communications of the ACM, 58(10), 38-40.Google Scholar
  7. Faruqui, M., Dodge, J., Jauhar, S. K., Dyer, C., Hovy, E., and Smith, N. A. (2014). Retrofitting word vectors to semantic lexicons. arXiv preprint arXiv:1411.4166.Google Scholar
  8. Hawkins, J., and Blakeslee, S. (2007). On intelligence. Macmillan.Google Scholar
  9. Hayes, P. J. (1978). The naive physics manifesto. Institut pour les études sémantiques et cognitives/Université de Genève.Google Scholar
  10. Hinton, G. E. (1984). Distributed representations.Google Scholar
  11. Hofstadter, D. (1985). Metamagical themas: Questing for the essence of mind and pattern. Basic books.Google Scholar
  12. Hofstadter, D, and Sander, E. Surfaces and Essences. Basic Books, 2013.Google Scholar
  13. Huth, A. G., Nishimoto, S., Vu, A. T., and Gallant, J. L. (2012). A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 76(6), 1210-1224.Google Scholar
  14. Kanerva, P. (1988). Sparse distributed memory. MIT press.Google Scholar
  15. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Skip-thought vectors. In Advances in neural information processing systems (pp. 3294-3302).Google Scholar
  16. Leech, R., Mareschal, D., and Cooper, R. P. (2008). Analogy as relational priming: A developmental and computational perspective on the origins of a complex cognitive skill. Behavioral and Brain Sciences, 31(04), 357-378.Google Scholar
  17. Lenat, D. B., Prakash, M., & Shepherd, M. (1985). CYC: Using common sense knowledge to overcome brittleness and knowledge acquisition bottlenecks. AI magazine, 6(4), 65.Google Scholar
  18. Levy, O., and Goldberg, Y. (2014). Dependency-Based Word Embeddings. In ACL (2) (pp. 302-308).Google Scholar
  19. Lin, Y., Liu, Z., Luan, H., Sun, M., Rao, S., and Liu, S. (2015). Modeling relation paths for representation learning of knowledge bases. arXiv preprint arXiv:1506.00379.Google Scholar
  20. Neelakantan, A., Roth, B., and Mc-Callum, A. (2015, March). Compositional vector space models for knowledge base inference. In 2015 AAAI Spring Symposium Series.Google Scholar
  21. Reed, S. E., Zhang, Y., Zhang, Y., and Lee, H. (2015). Deep visual analogy-making. In Advances in Neural Information Processing Systems (pp. 1252-1260).Google Scholar
  22. Rei, M., and Briscoe, T. (2014, June). Looking for Hyponyms in Vector Space. In CoNLL (pp. 68-77).Google Scholar
  23. Rissman, J., and Wagner, A. D. (2012). Distributed representations in memory: insights from functional brain imaging. Annual review of psychology, 63, 101.Google Scholar
  24. Rothe, S., and Schütze, H. (2015). Autoextend: Extending word embeddings to embeddings for synsets and lexemes. arXiv preprint arXiv:1507.01127.Google Scholar
  25. Russell, S. (2014, November 14). Of Myths And Moonshine. Retrieved from
  26. Sadeghi, F., Zitnick, C. L., and Farhadi, A. (2015). Visalogy: Answering visual analogy questions. In Advances in Neural Information Processing Systems (pp. 1882-1890).Google Scholar
  27. Speer, R., Havasi, C., and Lieberman, H. (2008, July). AnalogySpace: Reducing the Dimensionality of Common Sense Knowledge. In AAAI (Vol. 8, pp. 548-553).Google Scholar
  28. Turney, P. D. (2006). Similarity of semantic relations. Computational Linguistics, 32(3), 379-416.Google Scholar
  29. Upchurch, P., Snavely, N., and Bala, K. (2016). From A to Z: Supervised Transfer of Style and Content Using Deep Neural Network Generators. arXiv preprint arXiv:1603.02003.Google Scholar
  30. Vosniadou, S., and Ortony, A. (1989). Similarity and analogical reasoning. Cambridge University Press.Google Scholar
  31. Wang, Z., Zhang, J., Feng, J., and Chen, Z. (2014, October). Knowledge Graph and Text Jointly Embedding. In EMNLP (pp. 1591-1601).Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.US RDECOM, Artificial Intelligence Researcher, Army Research LaboratoryAdelphiUSA

Personalised recommendations