Distant Supervision for Emotion Classification with Discrete Binary Values

  • Jared Suttles
  • Nancy Ide
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7817)


In this paper, we present an experiment to identify emotions in tweets. Unlike previous studies, which typically use the six basic emotion classes defined by Ekman, we classify emotions according to a set of eight basic bipolar emotions defined by Plutchik (Plutchik’s “wheel of emotions”). This allows us to treat the inherently multi-class problem of emotion classification as a binary problem for four opposing emotion pairs. Our approach applies distant supervision, which has been shown to be an effective way to overcome the need for a large set of manually labeled data to produce accurate classifiers. We build on previous work by treating not only emoticons and hashtags but also emoji, which are increasingly used in social media, as an alternative for explicit, manual labels. Since these labels may be noisy, we first perform an experiment to investigate the correspondence among particular labels of different types assumed to be indicative of the same emotion. We then test and compare the accuracy of independent binary classifiers for each of Plutchik’s four binary emotion pairs trained with different combinations of label types. Our best performing classifiers produce results between 75-91%, depending on the emotion pair; these classifiers can be combined to emulate a single multi-label classifier for Plutchik’s eight emotions that achieves accuracies superior to those reported in previous multi-way classification studies.


Emotional Content Sentiment Analysis Computational Linguistics Human Language Technology Emotional Token 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agarwal, A., Xie, B., Vovsha, I., Rambow, O., Passonneau, R.: Sentiment Analysis of Twitter Data. In: Proceedings of the Workshop on Language in Social Media (LSM 2011), pp. 30–38. Association for Computational Linguistics, Portland (2011)Google Scholar
  2. 2.
    Alm, C.O., Roth, D., Sproat, R.: Emotions from text: Machine learning for text-based emotion prediction. In: Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pp. 579–586. Association for Computational Linguistics, Vancouver (2005)CrossRefGoogle Scholar
  3. 3.
    Ansari, S.: Automatic emotion tone detection in twitter (2010)Google Scholar
  4. 4.
    Balabantaray, R.C., Mohammad, M., Sharma, N.: Multi-class twitter emotion classification: A new approach. International Journal of Applied Information Systems 4(1), 48–53 (2012)Google Scholar
  5. 5.
    Bollen, J., Pepe, A., Mao, H.: Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. CoRR (2009)Google Scholar
  6. 6.
    Choudhury, M.D., Counts, S., Gamon, M.: Not all moods are created equal! exploring human emotional states in social media. In: ICWSM (2012)Google Scholar
  7. 7.
    Chuang, Z.J., Wu, C.H.: Multi-modal emotion recognition from speech and text. International Journal of Computational Linguistics and Chinese Language Processing 9(2), 45–62 (2004)Google Scholar
  8. 8.
    Danisman, T., Alpkocak, A.: Emotion classification of audio signals using ensemble of support vector machines. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Pieraccini, R., Weber, M. (eds.) PIT 2008. LNCS (LNAI), vol. 5078, pp. 205–216. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  9. 9.
    Derks, D., Bos, A.E.R., Von Grumbkow, J.: Emoticons and online message interpretation. Social Science Computer Review 26(3), 379–388 (2008)CrossRefGoogle Scholar
  10. 10.
    Dodds, P.S., Harris, K.D., Kloumann, I.M., Bliss, C.A., Danforth, C.M.: Temporal patterns of happiness and information in a global social network: Hedonometrics and twitter. CoRR (2011)Google Scholar
  11. 11.
    Ekman, P.: Universals and cultural differences in facial expressions of emotions. Nebraska Symposium on Motivation 19, 207–283 (1972)Google Scholar
  12. 12.
    Ekman, P.: An argument for basic emotions. Cognition & Emotion 6(3-4), 169–200 (1992)CrossRefGoogle Scholar
  13. 13.
    Go, A., Bhayani, R., Huang, L.: Twitter sentiment classification using distant supervision. Processing, 1–6 (2009)Google Scholar
  14. 14.
    González-Ibáñez, R., Muresan, S., Wacholder, N.: Identifying sarcasm in twitter: A closer look. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 581–586. Association for Computational Linguistics, Portland (2011)Google Scholar
  15. 15.
    Lee, C.M., Narayanan, S.S.: Toward detecting emotions in spoken dialogs. In: IEEE Transactions on Speech and Audio Processing, pp. 293–303 (2005)Google Scholar
  16. 16.
    Mihalcea, R., Liu, H.: A corpus-based approach to finding happiness. In: AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs, pp. 139–144. AAAI (2006)Google Scholar
  17. 17.
    Mintz, M., Bills, S., Snow, R., Jurafsky, D.: Distant supervision for relation extraction without labeled data. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 1003–1011. Association for Computational Linguistics (2009)Google Scholar
  18. 18.
    Mohammad, S.: From once upon a time to happily ever after: Tracking emotions in novels and fairy tales. In: Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pp. 105–114. Association for Computational Linguistics, Portland (2011)Google Scholar
  19. 19.
    Mohammad, S.: Emotional Tweets. In: SEM 2012: The First Joint Conference on Lexical and Computational Semantics – vol. 1: Proceedings of the main conference and the shared task and vol. 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), June 7-8, pp. 246–255. Association for Computational Linguistics, Montréal (2012)Google Scholar
  20. 20.
    Neviarouskaya, A., Prendinger, H., Ishizuka, M.: Analysis of affect expressed through the evolving language of online communication. In: Chin, D.N., Zhou, M.X., Lau, T.A., Puerta, A.R. (eds.) Proceedings of the 12th International Conference on Intelligent User Interfaces, pp. 278–281 (2009)Google Scholar
  21. 21.
    Pak, A., Paroubek, P.: Twitter as a corpus for sentiment analysis and opinion mining. In: Calzolari, N., Choukri, K., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S., Rosner, M., Tapias, D. (eds.) Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC 2010), European Language Resources Association (ELRA), Valletta (2010)Google Scholar
  22. 22.
    Plutchik, R.: Emotion: Theory, research, and experience. In: Theories of Emotion, vol. 1. Academic Press, New York (1980)Google Scholar
  23. 23.
    Provine, R., Spencer, R., Mandell, D.: Emotional Expression Online: Emoticons punctuate website text messages. Journal of Language and Social Psychology 26(3), 299–307 (2007)CrossRefGoogle Scholar
  24. 24.
    Purver, M., Battersby, S.: Experimenting with distant supervision for emotion classification. In: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp. 482–491. Association for Computational Linguistics, Avignon (2012)Google Scholar
  25. 25.
    Read, J.: Recognising Affect in Text using Pointwise-Mutual Information. Master’s thesis, University of Sussex (2004)Google Scholar
  26. 26.
    Read, J.: Using emoticons to reduce dependency in machine learning techniques for sentiment classification. In: Proceedings of the ACL Student Research Workshop, pp. 43–48. Association for Computational Linguistics, Stroudsburg (2005)CrossRefGoogle Scholar
  27. 27.
    Roberts, K., Roach, M.A., Johnson, J., Guthrie, J., Harabagiu, S.M.: EmpaTweet: Annotating and Detecting Emotions on Twitter. In: Calzolari, N., Choukri, K., Declerck, T., Doğan, M.U., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S. (eds.) Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC 2012), European Language Resources Association (ELRA), Istanbul (2012)Google Scholar
  28. 28.
    Seol, Y.S., Kim, D.J., Kim, H.W.: Emotion Recognition from Text Using Knowledge-based ANN. In: Proceedings of ITC-CSCC (2009)Google Scholar
  29. 29.
    Tanaka, Y., Takamura, H., Okumura, M.: Extraction and classification of facemarks. In: Proceedings of the 10th International Conference on Intelligent User Interfaces, IUI 2005, pp. 28–34. ACM, New York (2005)Google Scholar
  30. 30.
    Wang, W., Chen, L., Thirunarayan, K., Sheth, A.: Harnessing Twitter Big Data for Automatic Emotion Identification. In: International Conference on Social Computing (2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Jared Suttles
    • 1
  • Nancy Ide
    • 1
  1. 1.Department of Computer ScienceVassar CollegePoughkeepsieUSA

Personalised recommendations