Advertisement

An Online Trend Detection Strategy for Twitter Using Mann–Kendall Non-parametric Test

  • Sourav Malakar
  • Saptarsi GoswamiEmail author
  • Amlan Chakrabarti
Conference paper
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 11)

Abstract

Twitter is one of the most popular online social networking and micro-blogging service that enables its users to post and share text-based messages called Tweets. The data generated daily in terms of tweets are enormous and represents a rich source of information. To elicit actionable intelligence, various natural language processing (NLP) and text mining techniques are applied. Detecting of trends from twitters represents an important set of problems with a wide variety of applications and has huge appeal to diverse communities. In this paper, a simple trend detection technique based on term frequency has been proposed. In the first step, term document matrix of the tweet stream is created and top words are identified. The top word list is dynamically updated based on new streams. Time series is generated for the top words. Trends of the words are detected using Mann–Kendall non-parametric test. The method has been applied on few topical twitter datasets and proved to be quite effective.

Keywords

Twitter Text mining Time series 

References

  1. 1.
    Nirmala, C.R., Roopa, G.M., Naveen Kumar, K.R.: Twitter data analysis for unemployment crisis. In: 2015 International Conference on Applied and Theoretical Computing and Communication Technology (iCATccT), pp. 420–423. IEEE (2015)Google Scholar
  2. 2.
    Singhal, K., Agrawal, B., Mittal, N.: Modeling Indian general elections: sentiment analysis of political Twitter data. In: Information Systems Design and Intelligent Applications, pp. 469–477. Springer India (2015)Google Scholar
  3. 3.
    Barnwal, A.K., Choudhary, G.K., Swarnim, R., Kedia, A., Goswami, S., Das, A.: Application of Twitter in health care sector for India. In: 3rd IEEE International Conference on Recent Advances in Information Technology (Accepted)Google Scholar
  4. 4.
    Mathioudakis, M., Koudas, N.: Twitter monitor: trend detection over the twitter stream. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, pp. 1155–1158. ACM (2010)Google Scholar
  5. 5.
    Kim, D., Kim, D., Rho, S., Hwang, E.: Detecting trend and bursty keywords using characteristics of Twitter stream data. Int. J. Smart Home 7(1), 209–220 (2013)Google Scholar
  6. 6.
    Lau, J.H., Collier, N., Baldwin, T.: On-line trend analysis with topic models: Twitter trends detection topic model online. In: COLING, pp. 1519–1534 (2012)Google Scholar
  7. 7.
    Bolelli, L., Ertekin, Ş., Giles, C.L.: Topic and trend detection in text collections using latent Dirichlet allocation. In: European Conference on Information Retrieval, 6 April 2009, pp. 776–780. Springer, BerlinGoogle Scholar
  8. 8.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3(Jan), 993–1022 (2003)Google Scholar
  9. 9.
    Naaman, M., Becker, H., Gravano, L.: Hip and trendy: characterizing emerging trends on Twitter. J. Am. Soc. Inf. Sci. Technol. 62(5), 902–918 (2011)CrossRefGoogle Scholar
  10. 10.
    Kumar, S., Maskara, S., Chandak, N., Goswami, S.: Article: empirical study of relationship between Twitter mood and stock market from an Indian context. Int. J. Appl. 8, 33–37Google Scholar
  11. 11.
    Bolelli, L., Ertekin, S., Lee Giles, C.: Topic and trend detection in text collections using latent Dirichlet allocation. In: ECIR 09 Proceedings of the 31st European Conference on IR Research on Advances in Information Retrieval, pp. 776–780Google Scholar
  12. 12.
    Walther, M., Kaisser, M.: Geo-spatial event detection in the twitter stream. In: Advances in Information Retrieval, pp. 356–367. Springer, Berlin (2013)Google Scholar
  13. 13.
    Ishikawa, S., Arakawa, Y., Tagashira, S., Fukuda, A.: Hot topic detection in local areas using Twitter and Wikipedia. In: ARCS Workshops (ARCS), 2012, pp. 1–5. IEEE (2012)Google Scholar
  14. 14.
    Miyabe, M., Miura, A., Aramaki, E.: Use trend analysis of Twitter after the Great East Japan earthquake. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work Companion, pp. 175–178. ACM (2012)Google Scholar
  15. 15.
    Aiello, L.M., Petkos, G., Martin, C., Corney, D., Papadopoulos, S., Skraba, R., Goker, A., Kompatsiaris, I., Jaimes, A.: Sensing trending topics in Twitter. IEEE Trans. Multimed. 15(6), 1268–1282 (2013)Google Scholar
  16. 16.
    Yue, S., Pilon, P., Cavadias, G.: Power of the Mann-Kendall and Spearman’s rho tests for detecting monotonic trends in hydrological series. J. Hydrol. 259(1), 254–271 (2002)CrossRefGoogle Scholar
  17. 17.
    Guhathakurta, P., Sreejith, O.P., Menon, P.A.: Impact of climate change on extreme rainfall events and flood risk in India. J. Earth Syst. Sci. 120(3), 359–373 (2011)CrossRefGoogle Scholar
  18. 18.
    Liu, Q., Yang, Z., Cui, B.: Spatial and temporal variability of annual precipitation during 1961–2006 in Yellow River Basin, China. J. Hydrol. 361(3), 330–338 (2008)Google Scholar
  19. 19.
    Neeti, N., Ronald Eastman, J.: A contextual Mann‐Kendall approach for the assessment of trend significance in image time series. Trans. GIS 15(5), 599–611 (2011)Google Scholar
  20. 20.
    Grottke, M., Li, L., Vaidyanathan, K., Trivedi, K.S.: Analysis of software aging in a web server. IEEE Trans. Reliab. 55(3), 411–420 (2006)CrossRefGoogle Scholar
  21. 21.
    R Core Team: R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, http://www.R-project.org/ (2013)
  22. 22.
    Gentry, J.: Twitter: R based Twitter Client. R package version 1.1.9, https://CRAN.R-project.org/package=twitteR (2015)
  23. 23.
    Feinerer, I., Hornik, K., Meyer, D.: Text mining infrastructure in R. J. Stat. Softw. (2008)Google Scholar
  24. 24.
    Zeileis, A., Grothendieck, G.: Zoo: S3 infrastructure for regular and irregular time series. J. Stat. Softw. 14(6), 1–27 (2005), http://jstatsoft.org/v14/i06/

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  • Sourav Malakar
    • 1
  • Saptarsi Goswami
    • 1
    Email author
  • Amlan Chakrabarti
    • 1
  1. 1.KolkataIndia

Personalised recommendations