To Improve, or Not to Improve; How Changes in Corpora Influence the Results of Machine Learning Tasks on the Example of Datasets Used for Paraphrase Identification

  • Krystyna Chodorowska
  • Barbara Rychalska
  • Katarzyna Pakulska
  • Piotr Andruszkiewicz
Part of the Studies in Big Data book series (SBD, volume 40)


In this paper we attempt to verify the influence of data quality improvements on results of machine learning tasks. We focus on measuring semantic similarity and use the SemEval 2016 datasets. To achieve consistent annotations, we made all sentences grammatically and lexically correct, and developed formal semantic similarity criteria. The similarity detector used in this research was designed for the SemEval English Semantic Textual Similarity (STS) task. This paper addresses two fundamental issues: first, how each characteristic of the chosen sets affects performance of similarity detection software, and second, which improvement techniques are most effective for provided sets and which are not. Having analyzed these points, we present and explain the not obvious results we obtained.


Corpora quality Annotation rules SemEval Semantic textual similarity 


  1. 1.
    ACL 2008 Third Workshop on Statistical Machine Translation: WMT2008 Development Dataset (2008).
  2. 2.
    Agirre, E., Banea, C., Cardie, C., Cer, D.M., Diab, M.T., Gonzalez-Agirre, A., Guo, W., Mihalcea, R., Rigau, G., Wiebe, J.: Semeval-2014 task 10: multilingual semantic textual similarity. In: Nakov, P., Zesch, T. (eds.) Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval@COLING 2014, Dublin, Ireland, August 23–24, pp. 81–91. The Association for Computer Linguistics (2014).
  3. 3.
    Bethard, S., Cer, D.M., Carpuat, M., Jurgens, D., Nakov, P., Zesch, T. (eds.): Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16–17, 2016. The Association for Computer Linguistics (2016).
  4. 4.
    Labadié, A., Prince, V.: The impact of corpus quality and type on topic based text segmentation evaluation. In: Proceedings of the International Multiconference on Computer Science and Information Technology, IMCSIT 2008, Wisla, Poland, 20–22 October 2008. pp. 313–319. IEEE (2008).
  5. 5.
    Microsoft Research: Microsoft Research Paraphrase Corpus (2010).
  6. 6.
    Microsoft Research: Microsoft Research Video Description Corpus (2010).
  7. 7.
    Miller, G.A.: Wordnet: a lexical database for english. Commun. ACM 38, 39–41 (1995)CrossRefGoogle Scholar
  8. 8.
    Rus, V., Banjade, R., Lintean, M.: On Paraphrase Identification Corpora (2015).
  9. 9.
    Rychalska, B., Pakulska, K., Chodorowska, K., Walczak, W., Andruszkiewicz, P.: Samsung Poland NLP Team at SemEval-2016 task 1: Necessity for diversity; combining recursive autoencoders, wordnet and ensemble methods to measure semantic similarity. In: Bethard et al., [3], pp. 602–608.
  10. 10.
    Socher, R., Huang, E.H., Pennington, J., Ng, A.Y., Manning, C.D.: Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection (2011)Google Scholar
  11. 11.
    Sultan, M.A., Bethard, S., Sumner, T.: DLS@CU: Sentence Similarity From Word Alignment and Semantic Vector (2015).
  12. 12.
    Talvensaari, T.: Effects of aligned corpus quality and size in corpus-based CLIR. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) Advances in Information Retrieval, 30th European Conference on IR Research, ECIR 2008, Glasgow, UK, March 30–April 3, 2008. Proceedings. Lecture Notes in Computer Science, vol. 4956, pp. 114–125. Springer, Berlin (2008).
  13. 13.
    Ul-Qayyum, Z., Altaf, W.: Paraphrase Identification Using Semantic Heuristic Features (2012).
  14. 14.
    Zhou, Y., Liu, P., Zong, C.: Approaches to improving corpus quality for statistical machine translation. Int. J. Comput. Proc. Oriental Lang. 23(4), 327–348 (2011). Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2019

Authors and Affiliations

  • Krystyna Chodorowska
    • 1
  • Barbara Rychalska
    • 1
  • Katarzyna Pakulska
    • 1
  • Piotr Andruszkiewicz
    • 1
  1. 1.Samsung R&D Institute PolandWarsawPoland

Personalised recommendations