Advertisement

Emoji Prediction: A Transfer Learning Approach

  • Linrui Zhang
  • Yisheng Zhou
  • Tatiana Erekhinskaya
  • Dan Moldovan
Conference paper
  • 25 Downloads
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1130)

Abstract

We present a transfer learning model for the Emoji Prediction task described at SemEval-2018 Task 2. Given a text of tweet, the task aims to predict the most likely emoji to be used within such tweet. The proposed method used a pre-training and fine-tuning strategy, which applies the pre-learned knowledge from several upstream tasks to downstream Emoji Prediction task, solving the data scarcity issue suffered by most of the SemEval-2018 participants using supervised learning strategy. Our transfer learning-based model can outperform state-of-the-art system (best performer at SemEval-2018) by 2.53% in macro F-score. Except from providing details of our system, this paper also intends to provide a comparison between supervised learning models and transfer learning models in solving Emoji Prediction task.

Keywords

Deep learning Transfer learning Emoji prediction 

References

  1. 1.
    Cappallo, S., Mensink, T., Snoek, C.G.: Image2emoji: zero-shot emoji prediction for visual media. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 1311–1314. ACM (2015)Google Scholar
  2. 2.
    Barbieri, F., Ballesteros, M., Saggion, H.: Are emojis predictable? In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 105–111. Association for Computational Linguistics, Valencia, Spain (2017)Google Scholar
  3. 3.
    Barbieri, F., Camacho-Collados, J., Ronzano, F., Anke, L.E., Ballesteros, M., Basile, V., Saggion, H.: SemEval-2018 Task 2: multilingual emoji prediction. In: Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018). Association for Computational Linguistics, New Orleans, LA, United States (2018)Google Scholar
  4. 4.
    Coltekin, C., Rama, T.: Tubingenoslo at semeval-2018 task 2: SVMs perform better than RNNs in emoji prediction. In: Proceedings of the 12th International Workshop on Semantic Evaluation, pp. 32–36. Association for Computational Linguistics, New Orleans, LA, United States (2018)Google Scholar
  5. 5.
    Beaulieu, J., Owusu, D.A.: Umduluth-cs8761 at semeval-2018 task 2: emojis: too many choices? In: Proceedings of the 12th International Workshop on Semantic Evaluation, pp. 32–36. Association for Computational Linguistics, New Orleans, LA, United States (2018)Google Scholar
  6. 6.
    Naziontis, C., Athanasiou, N., Chronopoulou, A., Kolovou, A., Paraskevopoulos, G., Ellians, N., Narayanan, S., Potamianos, A.: Ntua-slp at semeval-2018 task1: predicting affective content in tweets with deep attentive rnns and transfer learning. arXiv prepreint arXiv:1804.06658
  7. 7.
    Liu, M.: Emonlp at semeval-2018 task 2: English emoji prediction with gradient boosting regression tree method and bidirectional LSTM. In: Proceedings of the 12th International Workshop on Semantic Evaluation, pp. 32–36. Association for Computational Linguistics, New Orleans, LA, United States (2018)Google Scholar
  8. 8.
    Coltekin, C., Rama, T.: Discriminating similar languages with linear SVMs and neural networks. In: Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects, pp. 15–24. Osaka, Japan (2016)Google Scholar
  9. 9.
    Medvedeva, M., Kroon, M., Plank, B.: When sparse traditional models outperform dense neural networks: the curious case of discriminating between similar languages. In: Proceedings of the Fourth Workshop on NLP for similar language, Varieties and Dialects, pp. 156–163. Association for Computational Linguistics, Valencia, Spain (2017)Google Scholar
  10. 10.
    West, J., Venture, D., Warnick, S.: Spring research presentation: a theoretical foundation for inductive transfer. Brigham Young Univ. Coll. Phys. Math. Sci. 1, 32 (2007)Google Scholar
  11. 11.
    Radford, A., Narasimhan, K., Salimans, T., Sutskever. I.: Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/opennai-assets/researchcovers/languageunsupercised/languageunderstandingpaper.pdf (2018)
  12. 12.
    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  13. 13.
    Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 (2019)
  14. 14.
    Baziotis, C., Pelekis, N., Doulkeridis, C.: Datastories at semeval-2017 task 4: deep lstm with attention for message-level and topic-based sentiment analysis. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 747–754. Vancouver, Canada (2017)Google Scholar
  15. 15.
    Joulin, A., Grave, E., Bojanowski, R., Mikolov, T.: Bag of tricks for efficient text classification. arXiv:1607.01759 (2016)

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Linrui Zhang
    • 1
  • Yisheng Zhou
    • 1
  • Tatiana Erekhinskaya
    • 1
  • Dan Moldovan
    • 1
  1. 1.RichardsonUSA

Personalised recommendations