Detection of Duplicates in Quora and Twitter Corpus

  • Sujith ViswanathanEmail author
  • Nikhil Damodaran
  • Anson Simon
  • Anon George
  • M. Anand Kumar
  • K. P. Soman
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 750)


Detection of duplicate sentences from a corpus containing a pair of sentences deals with identifying whether two sentences in the pair convey the same meaning or not. This detection of duplicates helps in deduplication, a process in which duplicates are removed. Traditional natural language processing techniques are less accurate in identifying similarity between sentences, such similar sentences can also be referred as paraphrases. Using Quora and Twitter paraphrase corpus, we explored various approaches including several machine learning algorithms to obtain a liable approach that can identify the duplicate sentences given a pair of sentences. This paper discusses the performance of six supervised machine learning algorithms in two different paraphrase corpus, and it focuses on analyzing how accurately the algorithms classify sentences present in the corpus as duplicates and non-duplicates.


Deduplication Natural language processing Paraphrase Quora Twitter Machine learning 


  1. 1.
    Anand Kumar, M., Singh, S., Kavirajan, B., Soman, K.: Shared task on detecting paraphrases in indian languages (dpil): An overview. Lecture Notes in computer science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) pp. 128–140 (2018)Google Scholar
  2. 2.
    Blacoe, W., Lapata, M.: A comparison of vector-based representations for semantic composition. In: Proceedings of the 2012 joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 546–556. Association for Computational Linguistics (2012)Google Scholar
  3. 3.
    Cordeiro, J., Dias, G., Brazdil, P.: A metric for paraphrase detection. In: International Multi-Conference on Computing in the Global Information Technology, 2007. ICCGI 2007, pp. 7–7. IEEE (2007)Google Scholar
  4. 4.
    Fernando, S., Stevenson, M.: A semantic similarity approach to paraphrase detection. In: Proceedings of the 11th Annual Research Colloquium of the UK Special Interest Group for Computational Linguistics, pp. 45–52 (2008)Google Scholar
  5. 5.
    Huang, C.H., Yin, J., Hou, F.: A text similarity measurement combining word semantic information with tf-idf method. Jisuanji Xuebao(Chin. J. Comput.) 34(5), 856–864 (2011)Google Scholar
  6. 6.
    Iyer, S., Dandekar, N., Csernai, K.: First quora dataset release: Question pairs (2017)Google Scholar
  7. 7.
    Joao, C., Gaël, D., Pavel, B.: New functions for unsupervised asymmetrical paraphrase detection. J. Software 2(4), 12–23 (2007)CrossRefGoogle Scholar
  8. 8.
    Mahalakshmi, S., Anand Kumar, M., Soman, K.: Paraphrase detection for tamil language using deep learning algorithm. Int. J. of Appld. Engg. Res 10(17), 13929–13934 (2015)Google Scholar
  9. 9.
    Mueller, J., Thyagarajan, A.: Siamese recurrent architectures for learning sentence similarity. In: AAAI, pp. 2786–2792 (2016)Google Scholar
  10. 10.
    Praveena, R., Kumar, M.A., Soman, K.P.: Chunking based malayalam paraphrase identification using unfolding recursive autoencoders. In: 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI) pp. 922–928 (2017)Google Scholar
  11. 11.
    Socher, R., Huang, E.H., Pennin, J., Manning, C.D., Ng, A.Y.: Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In: Advances in neural information processing systems, pp. 801–809 (2011)Google Scholar
  12. 12.
    Xu, W., Callison-Burch, C., Dolan, B.: Semeval-2015 task 1: Paraphrase and semantic similarity in twitter (pit). In: Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pp. 1–11 (2015)Google Scholar
  13. 13.
    Xu, W., Ritter, A., Callison-Burch, C., Dolan, W.B., Ji, Y.: Extracting lexically divergent paraphrases from twitter. Trans. Assoc. Comput. Linguist. 2, 435–448 (2014)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  • Sujith Viswanathan
    • 1
    Email author
  • Nikhil Damodaran
    • 1
  • Anson Simon
    • 1
  • Anon George
    • 1
  • M. Anand Kumar
    • 1
  • K. P. Soman
    • 1
  1. 1.Center for Computational Engineering and Networking (CEN)Amrita Vishwa Vidyapeetham, Amrita School of EngineeringCoimbatoreIndia

Personalised recommendations