Detecting Health Events on the Social Web to Enable Epidemic Intelligence

  • Marco Fisichella
  • Avaré Stewart
  • Alfredo Cuzzocrea
  • Kerstin Denecke
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7024)


Content analysis and clustering of natural language documents becomes crucial in various domains, even in public health. Recent pandemics such as Swine Flu have caused concern for public health officials. Given the ever increasing pace at which infectious diseases can spread globally, officials must be prepared to react sooner and with greater epidemic intelligence gathering capabilities. Information should be gathered from a broader range of sources, including the Web which in turn requires more robust processing capabilities. To address this limitation, in this paper, we propose a new approach to detect public health events in an unsupervised manner. We address the problems associated with adapting an unsupervised learner to the medical domain and in doing so, propose an approach which combines aspects from different feature-based event detection methods. We evaluate our approach with a real world dataset with respect to the quality of article clusters. Our results show that we are able to achieve a precision of 62% and a recall of 75% evaluated using manually annotated, real-world data.


Retrospective medical event detection Clustering Epidemic Intelligence 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Allan, J., Papka, R., Lavrenko, V.: On-line new event detection and tracking. In: SIGIR 1998: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 37–45. ACM, New York (1998)Google Scholar
  2. 2.
    Collier, N., Doan, S., Kawazeo, A., Goodwin, R.M., Conway, M., Tateno, Y., Ngo, Q.H., Dien, D., Kawtrakul, A., Takeuchi, K., Shigematsu, M., Taniguchi, K.: Biocaster: detecting public health rumors with a web-based text mining system (2008),
  3. 3.
    Hartley, D., et al.: The of international event-based biosurveillance. Emerging Health Threats (2009),
  4. 4.
    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society 39(1), 1–38 (1977)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Fung, G.P.C., Yu, J.X., Yu, P.S., Lu, H.: Parameter free bursty events detection in text streams. In: VLDB 2005: Proceedings of the 31st International Conference on Very Large Data Bases, pp. 181–192. VLDB Endowment (2005)Google Scholar
  6. 6.
    Grishman, R., Huttunen, S., Yangarber, R.: Information extraction for enhanced access to disease outbreak reports. J. of Biomedical Informatics 35(4), 236–246 (2002), CrossRefGoogle Scholar
  7. 7.
    He, Q., Chang, K., Lim, E.P.: Analyzing feature trajectories for event detection. In: SIGIR, pp. 207–214 (2007)Google Scholar
  8. 8.
    Hofmann, T.: Probabilistic latent semantic analysis. In: UAI, pp. 289–296 (1999)Google Scholar
  9. 9.
    Kawamae, N.: Latent interest-topic model: finding the causal relationships behind dyadic data. In: CIKM, pp. 649–658 (2010)Google Scholar
  10. 10.
    Keller, M., Blench, M., Tolentino, H., et al.: Use of unstructured event-based reports for global infectious disease surveillance 15(5) (May 2009)Google Scholar
  11. 11.
    Kohlschütter, C., Fankhauser, P., Nejdl, W.: Boilerplate detection using shallow text features. In: Davison, B.D., Suel, T., Craswell, N., Liu, B. (eds.) WSDM, pp. 441–450. ACM, New York (2010), Google Scholar
  12. 12.
    Li, Z., Wang, B., Li, M., Ma, W.Y.: A probabilistic model for retrospective news event detection. In: SIGIR, pp. 106–113 (2005)Google Scholar
  13. 13.
    Madoff, L.C.: Promed-mail: An early warning system for emerging disease 2(39), 227–232 (July 2004)Google Scholar
  14. 14.
    Ming, Z., Wang, K., Chua, T.S.: Prototype hierarchy based clustering for the categorization and navigation of web collections. In: SIGIR, pp. 2–9 (2010)Google Scholar
  15. 15.
    Nallapati, R., Ahmed, A., Xing, E.P., Cohen, W.W.: Joint latent topic models for text and citations. In: KDD, pp. 542–550 (2008)Google Scholar
  16. 16.
    Paquet, C., Coulombier, D., Kaiser, R., Ciotti, M.: Epidemic intelligence: a new framework for strengthening disease surveillance in Europe. Euro Surveillence 11(12), 212–214 (2006), Google Scholar
  17. 17.
    Steinberger, R., Fuart, F., van der Groot, E., Best, C., von Etter, P., Yangarber, R.: Text mining from the web for medical intelligence. Mining Massive Data Sets for Security 19, 295–310 (2008)Google Scholar
  18. 18.
    Steyvers, M., Griffiths, T.: Probabilistic Topic Models. Lawrence Erlbaum Associates, Mahwah (2007)Google Scholar
  19. 19.
    Yang, Y., Pierce, T., Carbonell, J.: A study of retrospective and on-line event detection. In: SIGIR 1998: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 28–36. ACM, New York (1998)Google Scholar
  20. 20.
    Yangarber, R.: Verification of facts across document boundaries. In: Proceedings International Workshop on Intelligent Information Access (2006)Google Scholar
  21. 21.
    Zhang, D., Zhai, C., Han, J., Srivastava, A., Oza, N.: Topic modeling for olap on multidimensional text databases: topic cube and its applications. Stat. Anal. Data Min. 2(5-6), 378–395 (2009)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Marco Fisichella
    • 1
  • Avaré Stewart
    • 1
  • Alfredo Cuzzocrea
    • 2
  • Kerstin Denecke
    • 1
  1. 1.Forschungszentrum L3SHannoverGermany
  2. 2.ICAR-CNR and University of CalabriaItaly

Personalised recommendations