Training Neural Tensor Networks with the Never Ending Language Learner

  • Flávio A. O. Santos
  • Filipe B. do Nascimento
  • Matheus S. Santos
  • Hendrik T. Macedo
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 738)


Neural Networks have become the state-of-the-art technique in the field of Natural Language Processing (NLP). Many models attempt to learn and extend facts on graph-based knowledge bases (KBs). These datasets and models are valuable resources for many NLP tasks but are occasionally limited by data incompleteness. Previous work limited the number of relationships the model would learn from. In this paper we attempt to train a Neural Tensor Network (NTN) using 97 relationships from the Never Ending Language Learner (NELL) knowledge base. We compare its performance with previous NTNs trained with 11 relationships from Wordnet and 13 relationships from Freebase. Our model has achieved significant accuracy given the limited number of tuples per relationship in NELL’s KB.


Neural Tensor Network Never Ending Language Learning Knowledge base 



The authors thank FAPITEC-SE for granting a graduate scholarship to Flávio Santos, CNPq for granting a graduate scholarship to Filipe Nascimento and a productivity scholarship to Hendrik Macedo [DT-II, Processo 310446/2014-7] and LCAD-UFS for providing a cluster for the execution of the experiments.


  1. 1.
    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G.S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I.J. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Józefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D.G. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P.A. Tucker, V. Vanhoucke, V. Vasudevan, F.B. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, X. Zheng, Tensorflow: large-scale machine learning on heterogeneous distributed systems, CoRR, abs/1603.04467 (2016)Google Scholar
  2. 2.
    S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, Z. Ives, Dbpedia: a nucleus for a web of open data, in The Semantic Web (Springer, Berlin, 2007), pp. 722–735Google Scholar
  3. 3.
    C.M. Bishop, Pattern Recognition and Machine Learning (Springer, New York, 2006)Google Scholar
  4. 4.
    K. Bollacker, C. Evans, P. Paritosh, T. Sturge, J. Taylor, Freebase: a collaboratively created graph database for structuring human knowledge, in Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data (ACM, New York, 2008), pp. 1247–1250Google Scholar
  5. 5.
    A. Bordes, J. Weston, R. Collobert, Y. Bengio, et al., Learning structured embeddings of knowledge bases, in AAAI, vol. 6 (2011), p. 6Google Scholar
  6. 6.
    A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, O. Yakhnenko, Translating embeddings for modeling multi-relational data, in Advances in Neural Information Processing Systems (2013), pp. 2787–2795Google Scholar
  7. 7.
    L. Bottou, Stochastic gradient descent tricks, in Neural Networks: Tricks of the Trade (Springer, Berlin, 2012), pp. 421–436Google Scholar
  8. 8.
    A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E.R. Hruschka Jr., T.M. Mitchell, Toward an architecture for never-ending language learning, in AAAI, vol. 5 (2010), p. 3Google Scholar
  9. 9.
    A. Carlson, J. Betteridge, R.C. Wang, E.R. Hruschka Jr., T.M. Mitchell, Coupled semi-supervised learning for information extraction, in Proceedings of the Third ACM International Conference on Web Search and Data Mining (ACM, New York, 2010), pp. 101–110Google Scholar
  10. 10.
    X. Chen, A. Shrivastava, A. Gupta, Neil: extracting visual knowledge from web data, in Proceedings of the IEEE International Conference on Computer Vision (2013), pp. 1409–1416Google Scholar
  11. 11.
    L. Deng, D. Yu, et al., Deep learning: methods and applications. Found. Trends Signal Process. 7(3–4), 197–387 (2014)Google Scholar
  12. 12.
    M. Gardner, P.P. Talukdar, J. Krishnamurthy, T. Mitchell, Incorporating vector space similarity in random walk inference over knowledge bases (2014)Google Scholar
  13. 13.
    Y. Goldberg, A primer on neural network models for natural language processing. J. Artif. Intell. Res. (JAIR) 57, 345–420 (2016)Google Scholar
  14. 14.
    M. Gutmann, A. Hyvärinen, Noise-contrastive estimation: a new estimation principle for unnormalized statistical models, in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (2010), pp. 297–304Google Scholar
  15. 15.
    T. Mikolov, I. Sutskever, K. Chen, G.S. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, in Advances in Neural Information Processing Systems (2013), pp. 3111–3119Google Scholar
  16. 16.
    G.A. Miller, Wordnet: a lexical database for english. Commun. ACM 38(11), 39–41 (1995)Google Scholar
  17. 17.
    T.M. Mitchell, J. Allen, P. Chalasani, J. Cheng, O. Etzioni, M. Ringuette, J.C. Schlimmer, Theo: a framework for self-improving systems. Architectures for Intelligence (1991), pp. 323–355Google Scholar
  18. 18.
    T.M. Mitchell, W.W. Cohen, E.R. Hruschka Jr., P.P. Talukdar, J. Betteridge, A. Carlson, B.D. Mishra, M. Gardner, B. Kisiel, J. Krishnamurthy, et al., Never ending learning, in AAAI (2015), pp. 2302–2310Google Scholar
  19. 19.
    M. Samadi, M.M. Veloso, M. Blum, Openeval: web information query evaluation, in AAAI (2013)Google Scholar
  20. 20.
    R. Socher, D. Chen, C.D. Manning, A.Y. Ng, Reasoning with Neural Tensor Networks for knowledge base completion, in Advances in Neural Information Processing Systems (2013), pp. 926–934Google Scholar
  21. 21.
    H.-Y. Wang, W.-Y. Ma, Integrating semantic knowledge into lexical embeddings based on information content measurement, in EACL 2017 (2017), p. 509Google Scholar
  22. 22.
    C. Xu, Y. Bai, J. Bian, B. Gao, G. Wang, X. Liu, T.-Y. Liu, Rc-net: a general framework for incorporating knowledge into word representations, in Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (ACM, New York, 2014), pp. 1219–1228Google Scholar
  23. 23.
    M. Yu, M. Dredze, Improving lexical embeddings with semantic knowledge, in ACL (2) (2014), pp. 545–550Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Flávio A. O. Santos
    • 1
  • Filipe B. do Nascimento
    • 2
  • Matheus S. Santos
    • 2
  • Hendrik T. Macedo
    • 1
  1. 1.Computer Science Postgraduate ProgramFederal University of SergipeSão CristóvãoBrazil
  2. 2.Computer Science DepartmentFederal University of SergipeSão CristóvãoBrazil

Personalised recommendations