Abstract
As social media use grows and increasingly becomes a forum for social debate in politics, social issues, sports, and brand sentiment; accurately classifying social media sentiment remains an important computational challenge. Social media posts present numerous challenges for text classification. This paper presents an approach to introduce guided decision trees into the design of a crowdsourcing platform to extract additional data features, reduce task cognitive complexity, and improve the quality of the resulting labeled text corpus. We compare the quality of the proposed approach with off-the-shelf sentiment classifiers and a crowdsourced solution without a decision tree using a tweet sample from the social media firestorm #CancelColbert. We find that the proposed crowdsource with decision tree approach results in a training corpus with higher quality, necessary for effective classification of social media content.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
E. Kouloumpis, T. Wilson, J.D. Moore, Twitter sentiment analysis: The good the bad and the omg!, in ICWSM 11 (2011), pp. 11538–541
A. Bermingham, A.F. Smeaton, Classifying sentiment in microblogs: Is brevity an advantage?, in 19th ACM International Conference on Information and Knowledge Management, ACM (2010), pp. 1833–1836
H. Yu, V. Hatzivassiloglou, Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences, in EMNLP (2003)
D. Maynard, M.A. Greenwood, Who cares about sarcastic tweets? Investigating the impact of sarcasm on sentiment analysis, in Language Resources and Evaluation Conference (2014), pp. 4238–4243
A. Reyes, P. Rosso, T. Veale, A multidimensional approach for detecting irony in Twitter, in Language Resources and Evaluation, vol. 47, no. 1 (2013), pp. 239–268
O. Tsur, D. Davidov, A. Rappoport, A great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews, in Fourth International AAAI Conference on Weblogs and Social Media (2010)
R.W. Gibbs, H.L. Colston, in Irony in Language and Thought (Routledge (Taylor and Francis), New York, 2007)
C. Bosco, V. Patti, A. Bolioli, Developing corpora for sentiment analysis: The case of irony and senti-TUT. IEEE Intell. Syst. 28(2), 55–63 (2013)
S. Park, M. Ko, J. Kim, Y. Liu, J. Song, The politics of comments: Predicting political orientation of news stories with commenters’ sentiment patterns, in ACM 2011 Conference on Computer Supported Cooperative Work, ACM (2011), pp. 113–122
K.-L. Liu, W.-J. Li, M. Guo, Emoticon smoothed language models for twitter sentiment analysis, in AAAI (2012)
J. Dunn, Facebook totally dominates the list of most popular social media apps, [online] Business Insider, Available at http://www.businessinsider.com/facebook-dominates-most-popular-social-media-apps-chart-2017-7 (2017)
H. Lamba, M.M. Malik, J. Pfeffer, A tempest in a teacup? Analyzing firestorms on twitter, in Advances in Social Networks Analysis and Mining (ASONAM), 2015 IEEE/ACM International Conference, IEEE (2015), pp. 17–24
J. Kang, Campaign to ‘Cancel’ Colbert, [online] The New Yorker, Available at https://www.newyorker.com/news/news-desk/the-campaign-to-cancel-colbert (2014)
F. Morstatter, J. Pfeffer, H. Liu, When is it biased?: Assessing the representativeness of twitter’s streaming API, in 23rd International Conference on World Wide Web ACM (2014), pp. 555–556
M.M. Bradley, P.J. Lang, Affective norms for english words (ANEW) instruction manual and affective ratings, Technical Report C-1, The Center for Research in Psychophysiology University of Florida (2009)
F.Ã…. Nielsen, A new ANEW: Evaluation of a word list for sentiment analysis in microblogs, arXiv preprint arXiv:1103.2903 (2011)
X. Zhu, S. Kiritchenko, S. Mohammad, NRC-Canada-2014: Recent improvements in the sentiment analysis of tweets, in SemEval@ COLING (2014), pp. 443–447
S. Kiritchenko, X. Zhu, S.M. Mohammad, Sentiment analysis of short informal texts. J. Artif. Intell. Res. 50, 723–762 (2014)
B. Liu, Sentiment analysis and opinion mining, in Synthesis Lectures on Human Language Technologies, vol. 5, no. 1 (2012), pp. 1–167
R. Artstein, M. Poesio, Inter-coder agreement for computational linguistics. Comput. Linguist. 34(4), 555–596 (2008)
P.S. Bayerl, K.I. Paul, What determines inter-coder agreement in manual annotations? A meta-analytic investigation. Comput. Linguist. 37(4), 699–725 (2011)
A.F. Hayes, K. Krippendorff, Answering the call for a standard reliability measure for coding data, in Communication Methods and Measures, vol. 1, no. 1, pp. 77–89 (2007)
K.A. Neuendorf, The Content Analysis Guidebook (Sage, 2016)
Acknowledgements
This work was supported by the Office of Naval Research, Grant No. N00014-17-1-2981/127025
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
McCulloh, I., Cohen, R., Takacs, R. (2020). Better Quality Classifiers for Social Media Content: Crowdsourcing with Decision Trees. In: Elçi, A., Sa, P., Modi, C., Olague, G., Sahoo, M., Bakshi, S. (eds) Smart Computing Paradigms: New Progresses and Challenges. Advances in Intelligent Systems and Computing, vol 767. Springer, Singapore. https://doi.org/10.1007/978-981-13-9680-9_8
Download citation
DOI: https://doi.org/10.1007/978-981-13-9680-9_8
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-9679-3
Online ISBN: 978-981-13-9680-9
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)