Revisiting Correlations between Intrinsic and Extrinsic Evaluations of Word Embeddings

  • Yuanyuan Qiu
  • Hongzheng Li
  • Shen Li
  • Yingdi Jiang
  • Renfen HuEmail author
  • Lijiao Yang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11221)


The evaluation of word embeddings has received a considerable amount of attention in recent years, but there have been some debates about whether intrinsic measures can predict the performance of downstream tasks. To investigate this question, this paper presents the first study on the correlation between results of intrinsic evaluation and extrinsic evaluation with Chinese word embeddings. We use word similarity and word analogy as the intrinsic tasks, Named Entity Recognition and Sentiment Classification as the extrinsic tasks. A variety of Chinese word embeddings trained with different corpora and context features are used in the experiments. From the data analysis, we reach some interesting conclusions: there are strong correlations between intrinsic and extrinsic evaluations, and the performance of different tasks can be affected by training corpora and context features to varying degrees.


Word embedding Intrinsic evaluation Extrinsic evaluation 



This work is supported by the Fundamental Research Funds for the Central Universities, China Postdoctoral Science Foundation funded project (No. 2018M630095) and National Language Committee Research Program of China (No. ZDI135-42).


  1. 1.
    Bakarov, A.: A survey of word embeddings evaluation methods. arXiv preprint arXiv:1801.09536 (2018)
  2. 2.
    Bansal, M., Gimpel, K., Livescu, K.: Tailoring continuous word representations for dependency parsing. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 809–815 (2014)Google Scholar
  3. 3.
    Batchkarov, M., Kober, T., Reffin, J., Weeds, J., Weir, D.: A critique of word similarity as a method for evaluating distributional semantic models. In: Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pp. 7–12 (2016)Google Scholar
  4. 4.
    Camacho-Collados, J., Pilehvar, M.T., Collier, N., Navigli, R.: Semeval-2017 task 2: multilingual and cross-lingual semantic word similarity. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 15–26 (2017)Google Scholar
  5. 5.
    Chen, C.Y., Ma, W.Y.: Word embedding evaluation datasets and wikipedia title embedding for Chinese. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pp. 825–831 (2018)Google Scholar
  6. 6.
    Chen, X., Xu, L., Liu, Z., Sun, M., Luan, H.B.: Joint learning of character and word embeddings. In: IJCAI, pp. 1236–1242 (2015)Google Scholar
  7. 7.
    Chiu, B., Korhonen, A., Pyysalo, S.: Intrinsic evaluation of word vectors fails to predict extrinsic performance. In: Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pp. 1–6 (2016)Google Scholar
  8. 8.
    Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12(Aug), 2493–2537 (2011)zbMATHGoogle Scholar
  9. 9.
    Evans, J.D.: Straightforward Statistics for the Behavioral Sciences. Brooks/Cole, Boston (1996)Google Scholar
  10. 10.
    Faruqui, M., Tsvetkov, Y., Rastogi, P., Dyer, C.: Problems with evaluation of word embeddings using word similarity tasks. In: Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pp. 30–35 (2016)Google Scholar
  11. 11.
    Finkelstein, L., et al.: Placing search in context: the concept revisited. In: Proceedings of the 10th international conference on World Wide Web, pp. 406–414. ACM (2001)Google Scholar
  12. 12.
    Gao, B., Bian, J., Liu, T.Y.: Wordrep: a benchmark for research on learning word representations. arXiv preprint arXiv:1407.1640 (2014)
  13. 13.
    Ghannay, S., Favre, B., Esteve, Y., Camelin, N.: Word embedding evaluation and combination. In: LREC, pp. 300–305 (2016)Google Scholar
  14. 14.
    Gurnani, N.: Hypothesis testing based intrinsic evaluation of word embeddings. arXiv preprint arXiv:1709.00831 (2017)
  15. 15.
    Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)
  16. 16.
    Levy, O., Goldberg, Y.: Dependency-based word embeddings. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Vol. 2: Short Papers), vol. 2, pp. 302–308 (2014)Google Scholar
  17. 17.
    Levy, O., Goldberg, Y.: Linguistic regularities in sparse and explicit word representations. In: Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pp. 171–180 (2014)Google Scholar
  18. 18.
    Levy, O., Goldberg, Y., Dagan, I.: Improving distributional similarity with lessons learned from word embeddings. Trans. Assoc. Comput. Linguist. 3, 211–225 (2015)Google Scholar
  19. 19.
    Li, S., Zhao, Z., Hu, R., Li, W., Liu, T., Du, X.: Analogical reasoning on Chinese morphological and semantic relations. arXiv preprint arXiv:1805.06504 (2018)
  20. 20.
    Li, Y., Li, W., Sun, F., Li, S.: Component-enhanced Chinese character embeddings. arXiv preprint arXiv:1508.06669 (2015)
  21. 21.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
  22. 22.
    Nayak, N., Angeli, G., Manning, C.D.: Evaluating word embeddings using a representative suite of practical tasks. In: Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pp. 19–23 (2016)Google Scholar
  23. 23.
    Schnabel, T., Labutov, I., Mimno, D., Joachims, T.: Evaluation methods for unsupervised word embeddings. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 298–307 (2015)Google Scholar
  24. 24.
    Tjong Kim Sang, E.F.: Introduction to the CoNLL-2002 shared task: language-independent named entity recognition. In: Proceedings of CoNLL-2002, Taipei, Taiwan, pp. 155–158 (2002)Google Scholar
  25. 25.
    Turian, J., Ratinov, L., Bengio, Y.: Word representations: a simple and general method for semi-supervised learning. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 384–394. Association for Computational Linguistics (2010)Google Scholar
  26. 26.
    Wu, Y., Li, W.: Overview of the NLPCC-ICCPOL 2016 shared task: Chinese word similarity measurement. In: Lin, C.-Y., Xue, N., Zhao, D., Huang, X., Feng, Y. (eds.) ICCPOL/NLPCC -2016. LNCS (LNAI), vol. 10102, pp. 828–839. Springer, Cham (2016). Scholar
  27. 27.
    Xu, J., Liu, J., Zhang, L., Li, Z., Chen, H.: Improve Chinese word embeddings by exploiting internal structure. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1041–1050 (2016)Google Scholar
  28. 28.
    Zhai, M., Tan, J., Choi, J.D.: Intrinsic and extrinsic evaluations of word embeddings. In: AAAI, pp. 4282–4283 (2016)Google Scholar
  29. 29.
    Zhao, Z., Liu, T., Li, S., Li, B., Du, X.: Ngram2vec: Learning improved word representations from Ngram co-occurrence statistics. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 244–253 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Yuanyuan Qiu
    • 1
    • 2
  • Hongzheng Li
    • 3
  • Shen Li
    • 1
    • 2
  • Yingdi Jiang
    • 1
    • 2
  • Renfen Hu
    • 1
    • 2
    Email author
  • Lijiao Yang
    • 1
    • 2
  1. 1.Institute of Chinese Information ProcessingBeijing Normal UniversityBeijingChina
  2. 2.UltraPower-BNU Joint Laboratory for Artificial IntelligenceBeijing Normal UniversityBeijingChina
  3. 3.School of Computer Science and TechnologyBeijing Institute of TechnologyBeijingChina

Personalised recommendations