Epidemic Outbreak and Spread Detection System Based on Twitter Data

Abstract

Social Network systems, such as Twitter, can serve as important data sources to provide collective intelligence and awareness of health problems in real time. The challenges of utilizing social media data include that the volume of data is large but distributed and of a highly unstructured form. Appropriate data gathering, scrubbing and aggregating efforts for these data are required to transform them for meaningful use. In this paper, we discuss such a social media data ETL (Extract-Transform-Load) method, to provide a user-friendly, dynamic method for visualizing outbreaks and the spread of developing epidemics in space and time. We have developed the Epidemics Outbreak and Spread Detection System (EOSDS) as a prototype that makes use of the rich information retrievable in real time from Twitter. EOSDS provides three different visualization methods of spreading epidemics, static map, distribution map, and filter map, to investigate public health threats in the space and time dimensions. The results of these visualizations in our experiments correlate well with relevant CDC official reports, a gold standard used by health informatics scientists. In our experiments, the EOSDS also detected an unusual situation not shown in the CDC reports, but confirmed by online news media.