Advertisement

Binarized Knowledge Graph Embeddings

  • Koki KishimotoEmail author
  • Katsuhiko Hayashi
  • Genki Akai
  • Masashi Shimbo
  • Kazunori Komatani
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11437)

Abstract

Tensor factorization has become an increasingly popular approach to knowledge graph completion (KGC), which is the task of automatically predicting missing facts in a knowledge graph. However, even with a simple model like CANDECOMP/PARAFAC (CP) tensor decomposition, KGC on existing knowledge graphs is impractical in resource-limited environments, as a large amount of memory is required to store parameters represented as 32-bit or 64-bit floating point numbers. This limitation is expected to become more stringent as existing knowledge graphs, which are already huge, keep steadily growing in scale. To reduce the memory requirement, we present a method for binarizing the parameters of the CP tensor decomposition by introducing a quantization function to the optimization problem. This method replaces floating point–valued parameters with binary ones after training, which drastically reduces the model size at run time. We investigate the trade-off between the quality and size of tensor factorization models for several KGC benchmark datasets. In our experiments, the proposed method successfully reduced the model size by more than an order of magnitude while maintaining the task performance. Moreover, a fast score computation technique can be developed with bitwise operations.

Keywords

Knowledge graph completion Tensor factorization Model compression 

References

  1. 1.
    Bengio, Y., Léonard, N., Courville, A.C.: Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR abs/1308.3432 (2013). http://arxiv.org/abs/1308.3432
  2. 2.
    Bollacker, K.D., Evans, C., Paritosh, P., Sturge, T., Taylor, J.: Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, 10–12 June 2008, pp. 1247–1250 (2008).  https://doi.org/10.1145/1376616.1376746
  3. 3.
    Bordes, A., Chopra, S., Weston, J.: Question answering with subgraph embeddings. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, Doha, Qatar, 25–29 October 2014, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 615–620 (2014). http://aclweb.org/anthology/D/D14/D14-1067.pdf
  4. 4.
    Bordes, A., Usunier, N., García-Durán, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of A meeting held 5–8 December 2013, Lake Tahoe, Nevada, United States, pp. 2787–2795 (2013). http://papers.nips.cc/paper/5071-translating-embeddings-for-modeling-multi-relational-data
  5. 5.
    Courbariaux, M., Bengio, Y., David, J.: Binaryconnect: training deep neural networks with binary weights during propagations. In: Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, Montreal, Quebec, Canada, 7–12 December 2015, pp. 3123–3131 (2015). http://papers.nips.cc/paper/5647-binaryconnect-training-deep-neural-networks-with-binary-weights-during-propagations
  6. 6.
    Dettmers, T., Minervini, P., Stenetorp, P., Riedel, S.: Convolutional 2D knowledge graph embeddings. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, 2–7 February 2018, New Orleans, Louisiana, USA (2018). https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17366
  7. 7.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010, pp. 249–256 (2010). http://www.jmlr.org/proceedings/papers/v9/glorot10a.html
  8. 8.
    Guo, Y.: A survey on methods and theories of quantized neural networks. CoRR abs/1808.04752 (2018). http://arxiv.org/abs/1808.04752
  9. 9.
    Hayashi, K., Shimbo, M.: On the equivalence of holographic and complex embeddings for link prediction. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, 30 July–4 August, Volume 2: Short Papers, pp. 554–559 (2017).  https://doi.org/10.18653/v1/P17-2088
  10. 10.
    Hitchcock, F.L.: The expression of a tensor or a polyadic as a sum of products. J. Math. Phys 6(1), 164–189 (1927)CrossRefGoogle Scholar
  11. 11.
    Kazemi, S.M., Poole, D.: Simple embedding for link prediction in knowledge graphs. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, Canada, 3–8 December 2018. pp. 4289–4300 (2018). http://papers.nips.cc/paper/7682-simple-embedding-for-link-prediction-in-knowledge-graphs
  12. 12.
    Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455–500 (2009).  https://doi.org/10.1137/07070111XMathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Kolda, T.G., Bader, B.W., Kenny, J.P.: Higher-order web link analysis using multilinear algebra. In: Proceedings of the 5th IEEE International Conference on Data Mining (ICDM 2005), Houston, Texas, USA, 27–30 November 2005, pp. 242–249 (2005).  https://doi.org/10.1109/ICDM.2005.77
  14. 14.
    Lacroix, T., Usunier, N., Obozinski, G.: Canonical tensor decomposition for knowledge base completion. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, 10–15 July 2018, pp. 2869–2878 (2018). http://proceedings.mlr.press/v80/lacroix18a.html
  15. 15.
    Lam, M.: Word2bits - quantized word vectors. CoRR abs/1803.05651 (2018). http://arxiv.org/abs/1803.05651
  16. 16.
    Liu, H., Wu, Y., Yang, Y.: Analogical inference for multi-relational embeddings. In: Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6–11 August 2017, pp. 2168–2178 (2017). http://proceedings.mlr.press/v70/liu17d.html
  17. 17.
    Ma, Y., Crook, P.A., Sarikaya, R., Fosler-Lussier, E.: Knowledge graph inference for spoken dialog systems. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, 19–24 April 2015, pp. 5346–5350 (2015).  https://doi.org/10.1109/ICASSP.2015.7178992
  18. 18.
    Nickel, M., Rosasco, L., Poggio, T.A.: Holographic embeddings of knowledge graphs. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, Arizona, USA, 12–17 February 2016, pp. 1955–1961 (2016). http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12484
  19. 19.
    Nickel, M., Tresp, V.: Logistic tensor factorization for multi-relational data. CoRR abs/1306.2084 (2013). http://arxiv.org/abs/1306.2084
  20. 20.
    Nickel, M., Tresp, V., Kriegel, H.: A three-way model for collective learning on multi-relational data. In: Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, 28 June–2 July 2011, pp. 809–816 (2011)Google Scholar
  21. 21.
    Palumbo, E., Rizzo, G., Troncy, R., Baralis, E., Osella, M., Ferro, E.: An empirical comparison of knowledge graph embeddings for item recommendation. In: Proceedings of the First Workshop on Deep Learning for Knowledge Graphs and Semantic Technologies (DL4KGS) Co-located with the 15th Extended Semantic Web Conerence (ESWC 2018), Heraklion, Crete, Greece, 4 June 2018, pp. 14–20 (2018). http://ceur-ws.org/Vol-2106/paper2.pdf
  22. 22.
    Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: XNOR-net: imagenet classification using binary convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016, Part IV. LNCS, vol. 9908, pp. 525–542. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_32CrossRefGoogle Scholar
  23. 23.
    Socher, R., Chen, D., Manning, C.D., Ng, A.Y.: Reasoning with neural tensor networks for knowledge base completion. In: Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013, Proceedings of a meeting held 5–8 December 2013, Lake Tahoe, Nevada, United States, pp. 926–934 (2013). http://papers.nips.cc/paper/5028-reasoning-with-neural-tensor-networks-for-knowledge-base-completion
  24. 24.
    Suchanek, F.M., Kasneci, G., Weikum, G.: Yago: a core of semantic knowledge. In: Proceedings of the 16th International Conference on World Wide Web, WWW 2007, Banff, Alberta, Canada, 8–12 May 2007, pp. 697–706 (2007).  https://doi.org/10.1145/1242572.1242667
  25. 25.
    Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., Bouchard, G.: Complex embeddings for simple link prediction. In: Proceedings of the 33rd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, 19–24 June 2016, pp. 2071–2080 (2016). http://jmlr.org/proceedings/papers/v48/trouillon16.html
  26. 26.
    Yang, B., Yih, W., He, X., Gao, J., Deng, L.: Embedding entities and relations for learning and inference in knowledge bases. CoRR abs/1412.6575 (2014). http://arxiv.org/abs/1412.6575

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Koki Kishimoto
    • 1
    Email author
  • Katsuhiko Hayashi
    • 1
    • 3
  • Genki Akai
    • 1
  • Masashi Shimbo
    • 2
    • 3
  • Kazunori Komatani
    • 1
  1. 1.Osaka UniversityOsakaJapan
  2. 2.Nara Institute of Science and TechnologyNaraJapan
  3. 3.RIKEN Center for Advanced Intelligence ProjectTokyoJapan

Personalised recommendations