Tweet Classification Using Sentiment Analysis Features and TF-IDF Weighting for Improved Flu Trend Detection

  • Ali Alessa
  • Miad FaezipourEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10934)


Social Networking Sites (SNS) such as Twitter are widely used by users of diverse ages. The rate of the data in SNS has made it become an efficient resource for real-time analysis. Thus, SNS data can effectively be used to track disease outbreaks and provide necessary warnings earlier than official agencies such as the American Center of Disease Control and Prevention. In this study, we show that sentiment analysis features and weighting techniques such as Term Frequency-Inverse Document Frequency (TF-IDF) can improve the accuracy of flu tweet classification. Various machine learning algorithms were evaluated to classify tweets to either flu-related or unrelated and then adopt the one with better accuracy. The results show that the proposed approach is useful for flu disease surveillance models/systems.


Influenza Machine learning Sentiment Social networking site TF-IDF 


  1. 1.
    Moorhead, S.A., Hazlett, D.E., Harrison, L., Carroll, J.K., Irwin, A., Hoving, C.: A new dimension of health care: systematic review of the uses, benefits, and limitations of social media for health communication. J. Med. Internet Res. 15(4), e85 (2013)CrossRefGoogle Scholar
  2. 2.
    Nurwidyantoro, A., Winarko, E.: Event detection in social media: a survey. In: 2013 International Conference on ICT for Smart Society (ICISS), pp. 1–5. IEEE (2013)Google Scholar
  3. 3.
    Itoh, M., Yokoyama, D., Toyoda, M., Tomita, Y., Kawamura, S., Kitsuregawa, M.: Visual fusion of mega-city big data: an application to traffic and tweets data analysis of metro passengers. In: 2014 IEEE International Conference on Big Data (Big Data), pp. 431–440. IEEE (2014)Google Scholar
  4. 4.
    Wang, X., Zeng, K., Zhao, X.-L., Wang, F.-Y.: Using web data to enhance traffic situation awareness. In: 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 195–199. IEEE (2014)Google Scholar
  5. 5.
    Zhang, S.: Using Twitter to enhance traffic incident awareness. In: 2015 IEEE 18th International Conference on Intelligent Transportation Systems, pp. 2941–2946. IEEE (2015)Google Scholar
  6. 6.
    Kosala, R., Adi, E., et al.: Harvesting real time traffic information from Twitter. Procedia Eng. 50, 1–11 (2012)CrossRefGoogle Scholar
  7. 7.
    Abel, F., Hauff, C., Houben, G.-J., Stronkman, R., Tao, K.: Twitcident: fighting fire with information from social web streams. In: Proceedings of the 21st International Conference on World Wide Web, pp. 305–308. ACM (2012)Google Scholar
  8. 8.
    Terpstra, T., de Vries, A., Stronkman, R., Paradies, G.L.: Towards a realtime Twitter analysis during crises for operational crisis management. Simon Fraser University, Burnaby, BC, Canada (2012)Google Scholar
  9. 9.
    Adam, N., Eledath, J., Mehrotra, S., Venkatasubramanian, N.: Social media alert and response to threats to citizens (SMART-C). In: 2012 8th International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom), pp. 181–189. IEEE (2012)Google Scholar
  10. 10.
    Abel, F., Hauff, C., Houben, G.-J., Stronkman, R., Tao, K.: Semantics + filtering + search = twitcident. Exploring information in social web streams. In: Proceedings of the 23rd ACM Conference on Hypertext and Social Media, pp. 285–294. ACM (2012)Google Scholar
  11. 11.
    Sakaki, T., Okazaki, M., Matsuo, Y.: Earthquake shakes Twitter users: real-time event detection by social sensors. In: Proceedings of the 19th International Conference on World Wide Web, pp. 851–860. ACM (2010)Google Scholar
  12. 12.
    Qusef, A., Ismail, K.: Social media in project communications management. In: 2016 7th International Conference on Computer Science and Information Technology (CSIT), pp. 1–5, July 2016Google Scholar
  13. 13.
    Treboux, J., Cretton, F., Evéquoz, F., Calvé, A.L., Genoud, D.: Mining and visualizing social data to inform marketing decisions. In: 2016 IEEE 30th International Conference on Advanced Information Networking and Applications (AINA), pp. 66–73, March 2016Google Scholar
  14. 14.
    Wan, S., Paris, C., Georgakopoulos, D.: Social media data aggregation and mining for internet-scale customer relationship management. In: 2015 IEEE International Conference on Information Reuse and Integration (IRI), pp. 39–48, August 2015Google Scholar
  15. 15.
    Burgess, J., Bruns, A.: Twitter archives and the challenges of “big social data” for media and communication research. M/C J. 15(5) (2012)Google Scholar
  16. 16.
    Yang, B., Guo, W., Chen, B., Yang, G., Zhang, J.: Estimating mobile traffic demand using Twitter. IEEE Wirel. Commun. Lett. 5(4), 380–383 (2016)CrossRefGoogle Scholar
  17. 17.
    Jackoway, A., Samet, H., Sankaranarayanan, J.: Identification of live news events using Twitter. In: Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Location-Based Social Networks, pp. 25–32. ACM (2011)Google Scholar
  18. 18.
    Ishikawa, S., Arakawa, Y., Tagashira, S., Fukuda, A.: Hot topic detection in local areas using Twitter and Wikipedia. In: ARCS Workshops (ARCS), pp. 1–5. IEEE (2012)Google Scholar
  19. 19.
    Petrovic, S., Osborne, M., Lavrenko, V.: The Edinburgh Twitter corpus. In: Proceedings of the NAACL HLT 2010 Workshop on Computational Linguistics in a World of Social Media, pp. 25–26 (2010)Google Scholar
  20. 20.
    Osborne, M., Petrovic, S., McCreadie, R., Macdonald, C., Ounis, I.: Bieber no more: first story detection using Twitter and Wikipedia. In: SIGIR 2012 Workshop on Time-Aware Information Access (2012)Google Scholar
  21. 21.
    Naveed, N., Gottron, T., Kunegis, J., Alhadi, A.C.: Bad news travel fast: a content-based analysis of interestingness on Twitter. In: Proceedings of the 3rd International Web Science Conference, WebSci 2011, pp. 8:1–8:7. ACM, New York (2011)Google Scholar
  22. 22.
    Corley, C.D., Cook, D.J., Mikler, A.R., Singh, K.P.: Text and structural data mining of influenza mentions in web and social media. Int. J. Environ. Res. Public Health 7(2), 596–615 (2010)CrossRefGoogle Scholar
  23. 23.
    Hwang, M.-H., Wang, S., Cao, G., Padmanabhan, A., Zhang, Z.: Spatiotemporal transformation of social media geostreams: a case study of Twitter for flu risk analysis. In: Proceedings of the 4th ACM SIGSPATIAL International Workshop on GeoStreaming, pp. 12–21. ACM (2013)Google Scholar
  24. 24.
    Polgreen, P.M., Chen, Y., Pennock, D.M., Nelson, F.D., Weinstein, R.A.: Using internet searches for influenza surveillance. Clin. Infect. Dis. 47(11), 1443–1448 (2008)CrossRefGoogle Scholar
  25. 25.
    Goel, S., Hofman, J.M., Lahaie, S., Pennock, D.M., Watts, D.J.: Predicting consumer behavior with web search. Proc. Natl. Acad. Sci. 107(41), 17486–17490 (2010)CrossRefGoogle Scholar
  26. 26.
    Scharkow, M., Vogelgesang, J.: Measuring the public agenda using search engine queries. Int. J. Public Opin. Res. 23(1), 104–113 (2011)CrossRefGoogle Scholar
  27. 27.
    Dugas, A.F., Hsieh, Y.-H., Levin, S.R., Pines, J.M., Mareiniss, D.P., Mohareb, A., Gaydos, C.A., Perl, T.M., Rothman, R.E.: Google flu trends: correlation with emergency department influenza rates and crowding metrics. Clin. Infect. Dis. 54(4), 463–469 (2012)CrossRefGoogle Scholar
  28. 28.
    Morrison, J.L., Breitling, R., Higham, D.J., Gilbert, D.R.: GeneRank: using search engine technology for the analysis of microarray experiments. BMC Bioinform. 6(1), 1 (2005)CrossRefGoogle Scholar
  29. 29.
    Ginsberg, J., Mohebbi, M.H., Patel, R.S., Brammer, L., Smolinski, M.S., Brilliant, L.: Detecting influenza epidemics using search engine query data. Nature 457(7232), 1012–1014 (2009)CrossRefGoogle Scholar
  30. 30.
    Lee, K., Agrawal, A., Choudhary, A.: Real-time disease surveillance using Twitter data: demonstration on flu and cancer. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1474–1477. ACM (2013)Google Scholar
  31. 31.
    Suh, B., Hong, L., Pirolli, P., Chi, Ed H.: Want to be retweeted? Large scale analytics on factors impacting retweet in Twitter network. In: 2010 IEEE Second International Conference on Social Computing (socialcom), pp. 177–184. IEEE (2010)Google Scholar
  32. 32.
    Broniatowski, D.A., Paul, M.J., Dredze, M.: National and local influenza surveillance through Twitter: an analysis of the 2012–2013 influenza epidemic. PLoS One 8(12), e83672 (2013)CrossRefGoogle Scholar
  33. 33.
    Lamb, A., Paul, M.J., Dredze, M.: Separating fact from fear: tracking flu infections on Twitter. In: North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2013), June 2013Google Scholar
  34. 34.
    Aramaki, E., Maskawa, S., Morita, M.: Twitter catches the flu: detecting influenza epidemics using Twitter. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1568–1576. Association for Computational Linguistics (2011)Google Scholar
  35. 35.
    Santos, J.C., Matos, S.: Analysing Twitter and web queries for flu trend prediction. Theor. Biol. Med. Model. 11(1), S6 (2014)CrossRefGoogle Scholar
  36. 36.
    Cui, X., Yang, N., Wang, Z., Cheng, H., Zhu, W., Li, H., Ji, Y., Liu, C.: Chinese social media analysis for disease surveillance. Pers. Ubiquit. Comput. 19(7), 1125–1132 (2015)CrossRefGoogle Scholar
  37. 37.
    Byrd, K., Mansurov, A., Baysal, O.: Mining Twitter data for influenza detection and surveillance. In: IEEE/ACM International Workshop on Software Engineering in Healthcare Systems (SEHS), pp. 43–49. IEEE (2016)Google Scholar
  38. 38.
    Sanders, N.J.: Sanders-Twitter Sentiment Corpus (2011). Accessed 20 Oct 2017
  39. 39.
    Amazon MTurk: Amazon Mechanical Turk (MTurk)Google Scholar
  40. 40.
    Bird, S.: NLTK: the natural language toolkit. In: Proceedings of the COLING/ACL on Interactive Presentation Sessions, pp. 69–72. Association for Computational Linguistics (2006)Google Scholar
  41. 41.
    Singh, J., Gupta, V.: A systematic review of text stemming techniques. Artif. Intell. Rev. 48(2), 157–217 (2017)CrossRefGoogle Scholar
  42. 42.
    Joachims, T.: A probabilistic analysis of the Rocchio algorithm with TFIDF for text categorization. Technical report, Carnegie-mellon univ pittsburgh pa dept of computer science (1996)Google Scholar
  43. 43.
    Loria, S., Keen, P., Honnibal, M., Yankovsky, R., Karesh, D., Dempsey, E., et al.: TextBlob: simplified text processing. Secondary TextBlob: Simplified Text Processing (2014)Google Scholar
  44. 44.
    Zweig, M.H., Campbell, G.: Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin. Chem. 39(4), 561–577 (1993)Google Scholar
  45. 45.
    Alessa, A., Faezipour, M.: A review of influenza detection and prediction through social networking sites. Theor. Biol. Med. Model. 15(2), 1–27 (2018)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of EngineeringUniversity of BridgeportBridgeportUSA

Personalised recommendations