An accurate forecasting of outbreaks of influenza-like illness (ILI) could support public health officials to suggest public health actions earlier. We investigated the performance of three different feature spaces in different models to forecast the weekly ILI rate in Syria using EWARS data from World Health Organization (WHO). Time series feature space was first used and we applied the seven models which are Naïve, Average, Seasonal naïve, drift, dynamic harmonic regression (Dhr), seasonal and trend decomposition using loess (STL) and TBATS. The Second feature space is like some state-of-the-art, which we named \(53-weeks-before\_52-first-order-difference\) feature space. The third one, we proposed and named \(n-years-before\_m-weeks-around\) (YnWm) feature space. Machine learning (ML) and deep learning (DL) model were applied to the second and third feature spaces (generalized linear model (GLM), support vector regression (SVR), gradient boosting (GB), random forest (RF) and long short term memory (LSTM)).
It was indicated that the LSTM model of four layers with \(1-year-before\_4-weeks-around\) feature space gave more accurate results than other models and reached the lowest MAPE of \(3.52\%\) and the lowest RMSE of 0.01662. I hope that this modelling methodology can be applied in other countries and therefore help prevent and control influenza worldwide.
Influenza epidemic results in three to five million cases of severe illnesses and approximately 290,000 to 650,000 deaths worldwide each year . WHO’s Early Warning, Alert and Response System (EWARS) is designed to improve disease outbreak detection in emergency settings. The system has been built in SYRIA since 2012 to collect and detect near real-time information on several outbreaks including influenza . The Existing researches on modeling influenza epidemic falls into two categories: Mechanistic and Statistical models. They are summarized in literature reviews [3,4,5,6] and in the CDC comparisons [7, 8]. Researches under statistical category vary according to the different Features and methods used. Some researchers used the number of patients in the past as features [9,10,11,12,13], while others integrated other data sources to predict the number of patients in the future. Examples of these sources are climatological data [14, 15], search engine queries [16,17,18,19], public comments on social media like Twitter [20, 21], online information-seeking behavior on websites like Wikipedia [22, 23] and a combination of multiple data streams [15, 24,25,26]. Different methods on these features were applied. Some researches treated the problem as an instance of more general time series forecasting using time series methods (ARIMA, ARIMA-STL, GARMA) [9, 10, 17, 27], while others used ML methods including Stacked linear regression [24, 26], AdaBoost regression with decision trees , GB , SVR [26, 28], elastic net  RF [11, 12, 28], Artificial Neural Network (ANN) [12, 20]. Recently, a DL method Called LSTM has attracted much interest in ILI prediction and gave excellent results, which are more accurate than those of other methods [12, 13, 15, 29]. In addition to investigating the performance of the three different feature spaces with multiple time-series, ML and DL based methods to predict the weekly ILI rate in Syria; we proposed novel future spaces \(n-years-before\_m-weeks-around\) that integrate into state-of-the-art ML and DL methods. There are two important contributions of this paper. First, the use of \(n-years-before\_m-weeks-around\) future spaces to predict ILI rate. Second, analyzing multiple models performance over the EWARS data from WHO in Syria.
Materials and methods
We collected SYRIA flu data from the EWARS reports published by WHO on the website . We only used the Flu Data from the first week of 2014 to the 42nd week of 2018. To avoid any possible population variations, we adopted the ILI rates as predictors (x) and responses (y) of our models.
Figure 1a illustrates the raw data. The Y-axis represents the weekly ILI rate, and the X-axis represents the time series. The seasonality is obvious, as shown in Fig 1b. We split the data into two parts: the first 90% was the training set and the last 10% was the testing set.
In this study, we reviewed three types of feature space. We named them as (time series feature space, \(53-weeks-before\_52-first-order-difference\) feature space and \(n-years-before\_m-weeks-around\) feature space). Tables S1 and S2 illustrate the data set, response, predictors and the pretreatment of the source data. An additional pdf file shows this in more detail (see Additional file 1).
Time series feature space (fs1) We treated the ILI weekly data as time series with seasonal value 52.
\(53- weeks-before\_52-first-order-differences\)feature space (fs2) Some previous studies found that using the ILI rate of the past 53 weeks and the 52 first-order differences helped improve the results of the prediction models for influenza data [11,12,13].
\(n-years-before\_m-weeks-around\)(YnWm) feature spaces (fs3) We reviewed a maximum of 3 years before and 5 weeks around. To predict the ILI rate of week WX of year YX, We used the ILI rate of the past m weeks before WX in YX, the ILI rate of past n seasonal weeks and the ILI rate of the m weeks around seasonal weeks. In the case of \(1-year-before\_4-weeks-around\), the response y is the ILI rate of week 30 of year 2017 then the predictors are the ILI rates of weeks 29, 28, 27, 26 from year 2017 and the ILI rates of weeks 26, 27, 28, 29, 30, 31, 32, 33, 34 from year 2016. The number of predictors and rows changed depending on the value of n and m and we had to drop some data rows so that we get the same training data length for any value of n and m. Figure 2a illustrates the feature spaces with examples of \(3-years-before\_5-weeks-around\) and \(1-year-before\_4-weeks-around\) feature spaces.
The models, programming languages, and libraries, which were used in this study were illustrated in Additional file 1. We trained all models in R Programming Language (version 3.4.4). For time series models, we used the “forecast” package (version 8.4). For ML models, we applied the caret package (Version 6.0-8). For DL models, we used the Keras package (Version 2.2.4) based on Tensorflow (Version 1.10). A personal computer with Intel I7-8550U processor, 8 GB of RAM and an NVIDIA 130 MX GPU was used for the experiments. Each experiment takes approximately 1 to 30 min to train the model. The prediction takes less than 5 s on the same hardware.
We compared different models with different feature spaces using the mean absolute percentage error (MAPE) and root mean squared error (RMSE) as key performance indicators (KPIs).
where At is the actual value and Ft is the forecasted value.
fs1 with time series models
Seven methods (Average, Naïve, Seasonal Naïve, Drift, STL, DHR and TBATS) were applied in a recursive way. We trained the model on the training data set then predicted the ILI rate of week number x then we repeated the process by combining the actual ILI rate of week number x with the training data set. The metrics of all models are presented in Table 1(a), (b). it was indicated that the TBATS model outperformed other six models in predicting weekly ILI rate (MAPE = 4.66%, RMSE = 0.03096). Our explanation of the success of TBATS model is that the seasonality is allowed to change slowly over time in a TBATS model, while DHR, STL, Seasonal Naïve models force the seasonal patterns to repeat periodically without changing and Average, Naïve, Drift models do not depend on the seasonal patterns. We considered TBATS model a baseline in comparing the results of the following models.
fs2 with ML and DL models
We applied four ML methods (GLM, SVR, GB and RF) and two DL methods (LSTM with 3 layers and LSTM with 4 layers). Table 1(a), (b) shows that none of the models achieved better results than the baseline model in MAPE metric but in RMSE metric, the LSTM model of 3 layers (RMSE = 0.02294) outperformed the baseline model and all otehr models with fs1 and fs2 .
fs3 with ML AND DL models
According to \(0<=n<=3\ and\ 0<=m<=5\) we trained 23 feature spaces in four ML methods (GLM, SVR, GB and RF) and two DL methods (LSTM with 3 layers and LSTM with 4 layers). Totally, we made 138 experiments with results shown in Table 1(a), (b). In accordance with MAPE metric, We found that 22 experiments show a decrease in metrics less than that of the baseline model (1 experiment: by using GB model with Y2W2 feature space, 10 experiments: by using LSTM model of 3 layers with “Y1Wm, Y2Wm: excluding m=0 and m=4” feature spaces, 11 experiments: by using LSTM model of 4 layers with “Y1Wm: excluding m=1,Y2Wm: excluding m=5,Y3w1” feature spaces) but In accordance with RMSE metric, We found that 44 experiments show a decrease in metrics less than that of the baseline model (1 experiment: by using GB model with Y2W2 feature space, 20 experiments: by using LSTM model of 3 layers with “YnWm: excluding n=3 with m=3,4,5” feature spaces, 23 experiments: by using LSTM model of 4 layers with “YnWm” feature spaces). we achieved the best result (MAPE = 3.52%, RMSE = 0.01662) by using LSTM model of 4 layers with Y1W4 feature space. in comparison to models with fs2, the results show that for any model used, there are at least one value to n and m in fs3 that achieved better result than the same model with fs3. Our explanation of the success of this feature space is that models can learn the seasonality by n years before and can recognize the trend of the data by m weeks around.
The temporal variation of the real weekly ILI rate and the predicted values obtained from the three models (the best result in each feature space) for the test period were plotted in Fig 2b. As can be seen, the predicted values of weekly ILI rate were in a good agreement with their related observed values and the used models could be used to model the weekly ILI rate. Moreover, LSTM model of 4 layers with fs3 resulted in better predicted values and trend between them for the observed values of ILI rate than the other models especially when there is a change in trend between observations.
The accurate forecast of an outbreak of ILI could support public health officials in taking public health actions, such as allocating or temporarily readjusting medical resources for hospitals and medical centers. The ILI rate varies from year to year depending on multiple factors. Despite this difference, the ILI rate plot in each year takes a semi-constant form with an increase or decrease compared to previous years and could be predicted by using specific observations of past years with statistical models. Performance of statistical models is data dependent and there is no model that performs well in all situations. Therefore, evaluating the performance of different models is of great importance as they provide useful and important information regarding strengths and weaknesses of the models and gives an insight to use better models for forecasting purposes. Some sate of the arts [11, 12] utilize 53 weeks before with 52 first order difference with different statistical models and found that both the recent observations and the later observations with the difference were interesting and had significant influence on the predication. In this study, we proposed novel future spaces, namely \(n-years-before\_m-weeks-around\), and compared to some existing future spaces that utilize historical observations in different ways by integrating it into state-of-the-art ML and DL models. Our results revealed the success of our future space for some values of n and m and its failure for other values to outperform other future spaces in prediction ILI rate in Syria. This fact suggests that combining carefully selected number of Nearby Historical Observations with Carefully selected number of seasonal Historical Observations is advantageous over simply choosing all Historical Observations. While the results presented here are for ILI within Syria, our novel future spaces shows promise to be easily extended to accurately track not only influenza in other countries but also other infectious diseases Through careful tuning of the values of n and m. We believe that it is up to policymakers in organizations concerned with EWARS to decide whether these forecasts are ready for use in decision support at the current level of accuracy.
In conclusion, we performed Naïve, Average, Seasonal naïve, Drift, DHR, STL, TBATS, GLM, SVR, GB, RF and LSTM methods with different feature spaces to predict the weekly ILI rate using EWARS data from WHO in Syria. We found that the TBATS method with time series feature space gave better results than those resulted from all methods in \(53-weeks-before\_52-first-order-differences\) feature space. We also found that the GLM, SVR, GB, RF and LSTM methods with a good choice to n and m in \(n-years-before\_m-weeks-around\) feature space gave better results than those resulted in \(53-weeks-before\_52-first-order-differences\) feature space. In all the models, the LSTM model of 4 layers reached the lowest MAPE (3.52%) and the lowest RMSE (0.01662).
Climatological data and Pharmaceutical Sales could be used to achieve better performance of the used models. We would like to investigate the impact of these parameters in future work.
Availability of data and materials
The data is provided as Additional file 2. The data is also publically available on: http://www.emro.who.int/syr/publications-other/ewars-weekly-bulletin-2014.html, http://www.emro.who.int/syr/publications-other/ewars-weekly-bulletin-2015.html, http://www.emro.who.int/syr/information-resources/2016-ewars-bulletins.html, http://www.emro.who.int/syr/ewars-workshops/ewars-bulletins-2017.html, http://www.emro.who.int/syr/information-resources/ewars-weekly-bulletins-2018.html.
Early Warning Alert and Response System
dynamic harmonic regression
seasonal and trend decomposition using Loess
\(n-years-before\_m-weeks-around\) feature space
generalized linear model
support vector regression
long short term memory
mean absolute percentage error
root mean square error
World Health Organization. Influenza (Seasonal) Fact Sheet. http://www.emro.who.int/syr/publications-other/ewars-weekly-bulletin-2014.html. Accessed 16 July 2019.
World Health Organization Regional Office for the Eastern Mediterranean. Syrian Arab Republic—EWARS: The Early Warning Alert and Response System. http://www.emro.who.int/syr/publications-other/ewars-weekly-bulletin.html. Accessed 16 July 2019.
Brooks LC, Farrow DC, Hyun S, Tibshirani RJ, Rosenfeld R. Nonmechanistic forecasts of seasonal influenza with iterative one-week-ahead distributions. PLoS Comput Biol. 2018;14(6):1006134.
Kandula S, Yamana T, Pei S, Yang W, Morita H, Shaman J. Evaluation of mechanistic and statistical methods in forecasting influenza-like illness. J R Soc Interface. 2018;15(144):20180174.
Unkel S, Farrington CP, Garthwaite PH, Robertson C, Andrews N. Statistical methods for the prospective detection of infectious disease outbreaks: a review. J R Stat Soc Ser A. 2012;175(1):49–82.
Nsoesie EO, Brownstein JS, Ramakrishnan N, Marathe MV. A systematic review of studies on forecasting the dynamics of influenza outbreaks. Influenza Other Respir Viruses. 2014;8(3):309–16.
Brammer L, Kniss K, Epperson S, Blanton L, Mustaquim D, Steffens C, D’Mello T, Perez A, Dhara R, Chaves SS, et al. Influenza activity-united states, 2012–13 season and composition of the 2013–14 influenza vaccine. MMWR Morb Mortal Wkly Rep. 2013;62(23):473.
Biggerstaff M, Johansson M, Alper D, Brooks LC, Chakraborty P, Farrow DC, Hyun S, Kandula S, McGowan C, Ramakrishnan N, et al. Results from the second year of a collaborative effort to forecast influenza seasons in the united states. Epidemics. 2018;24:26–33.
Gao H, Wong KK, Zheteyeva Y, Shi J, Uzicanin A, Rainey JJ. Comparing observed with predicted weekly influenza-like illness rates during the winter holiday break, united states, 2004–2013. PLoS ONE. 2015;10(12):0143791.
Kane MJ, Price N, Scotch M, Rabinowitz P. Comparison of ARIMA and random forest time series models for prediction of avian influenza H5N1 outbreaks. BMC Bioinform. 2014;15(1):276.
Wu H, Cai Y, Wu Y, Zhong R, Li Q, Zheng J, Lin D, Li Y. Time series analysis of weekly influenza-like illness rate using a one-year period of factors in random forest regression. Biosci Trends. 2017;11(3):292–6.
Zhang J, Nawata K. A comparative study on predicting influenza outbreaks. Biosci Trends 2017;11(5):533–41.
Zhang J, Nawata K. Multi-step prediction for influenza outbreak by an adjusted long short-term memory. Epidemiol Infect. 2018;146(7):809–16.
Shaman J, Karspeck A, Yang W, Tamerius J, Lipsitch M. Real-time influenza forecasts during the 2012–2013 season. Nat Commun. 2013;4:2837.
Venna SR, Tavanaei A, Gottumukkala RN, Raghavan VV, Maida AS, Nichols S. A novel data-driven model for real-time influenza forecasting. IEEE Access. 2018;7:7691–701.
Yuan Q, Nsoesie EO, Lv B, Peng G, Chunara R, Brownstein JS. Monitoring influenza epidemics in china with search query from baidu. PLoS ONE. 2013;8(5):64323.
Dugas AF, Jalalpour M, Gel Y, Levin S, Torcaso F, Igusa T, Rothman RE. Influenza forecasting with google flu trends. PLoS ONE. 2013;8(2):56176.
Yang S, Santillana M, Kou SC. Accurate estimation of influenza epidemics using google search data via argo. Proc Natl Acad Sci. 2015;112(47):14473–8.
Santillana M, Zhang DW, Althouse BM, Ayers JW. What can digital disease detection learn from (an external revision to) google flu trends? Am J Prev Med. 2014;47(3):341–7.
Lee K, Agrawal A, Choudhary A. Forecasting influenza levels using real-time social media streams. In: 2017 IEEE international conference on healthcare informatics (ICHI). New York: IEEE; 2017. p. 409–14.
Paul MJ, Dredze M, Broniatowski D. Twitter improves influenza forecasting. PLoS Curr. 2014. https://doi.org/10.1371/currents.outbreaks.90b9ed0f59bae4ccaa683a39865d9117.
Hickmann KS, Fairchild G, Priedhorsky R, Generous N, Hyman JM, Deshpande A, Del Valle SY. Forecasting the 2013–2014 influenza season using wikipedia. PLoS Comput Biol. 2015;11(5):1004239.
McIver DJ, Brownstein JS. Wikipedia usage estimates prevalence of influenza-like illness in the United States in near real-time. PLoS Comput Biol. 2014;10(4):1003581.
Lu FS, Hou S, Baltrusaitis K, Shah M, Leskovec J, Hawkins J, Brownstein J, Conidi G, Gunn J, Gray J, et al. Accurate influenza monitoring and forecasting using novel internet data streams: a case study in the boston metropolis. JMIR Public Health Surveill. 2018;4(1):4.
Ertem Z, Raymond D, Meyers LA. Optimal multi-source forecasting of seasonal influenza. PLoS Comput Biol. 2018;14(9):1006236.
Santillana M, Nguyen AT, Dredze M, Paul MJ, Nsoesie EO, Brownstein JS. Combining search, social media, and traditional data sources to improve influenza surveillance. PLoS Comput Biol. 2015;11(10):1004513.
Kandula S, Shaman J. Near-term forecasts of influenza-like illness: an evaluation of autoregressive time series approaches. Epidemics. 2019;27:41–51.
Poirier C, Lavenu A, Bertaud V, Campillo-Gimenez B, Chazard E, Cuggia M, Bouzillé G. Real time influenza monitoring using hospital big data in combination with machine learning methods: comparison study. JMIR Public Health Surveill. 2018;4(4):11361.
Burdakov A, Ukharov A, Myalkin M, Terekhov V. Forecasting of influenza-like illness incidence in amur region with neural networks. In: International conference on neuroinformatics. Berlin: Springer; 2018. p. 307–14.
World Health Organization Regional Office for the Eastern Mediterranean. Syrian Arab Republic—EWARS Weekly Bulletin. http://www.emro.who.int/syr/publications-other/ewars-weekly-bulletin-2014.html, http://www.emro.who.int/syr/publications-other/ewars-weekly-bulletin-2015.html, http://www.emro.who.int/syr/information-resources/2016-ewars-bulletins.html, http://www.emro.who.int/syr/ewars-workshops/ewars-bulletins-2017.html, http://www.emro.who.int/syr/information-resources/ewars-weekly-bulletins-2018.html. Accessed 16 July 2019.
This work is part of master degree research about “health map prediction in Syria” in Higher Institute for Applied Science and Technology.
Higher Institute for Applied Sciences and Technology provided technical support for the present study, by providing the personal computer used for the experiments. The funder had no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript and decision to publish.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional file 1. Details about feature spaces and programing languages and libraries. .pdf file with three tables: the first table explaint the \(53- weeks-before\_52-first-order-difference\) feature space, the second table explaint the \(N-years-before\_m-weeks-around\) feature space and the third one show the models, programming languages, and libraries, which were used in this study.
About this article
Cite this article
Darwish, A., Rahhal, Y. & Jafar, A. A comparative study on predicting influenza outbreaks using different feature spaces: application of influenza-like illness data from Early Warning Alert and Response System in Syria. BMC Res Notes 13, 33 (2020). https://doi.org/10.1186/s13104-020-4889-5
- Influenza-like illness (ILI)
- Feature space
- Time series analysis
- Long short term memory (LSTM)