Bayesian Best-Arm Identification for Selecting Influenza Mitigation Strategies

  • Pieter J. K. LibinEmail author
  • Timothy Verstraeten
  • Diederik M. Roijers
  • Jelena Grujic
  • Kristof Theys
  • Philippe Lemey
  • Ann Nowé
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11053)


Pandemic influenza has the epidemic potential to kill millions of people. While various preventive measures exist (i.a., vaccination and school closures), deciding on strategies that lead to their most effective and efficient use remains challenging. To this end, individual-based epidemiological models are essential to assist decision makers in determining the best strategy to curb epidemic spread. However, individual-based models are computationally intensive and it is therefore pivotal to identify the optimal strategy using a minimal amount of model evaluations. Additionally, as epidemiological modeling experiments need to be planned, a computational budget needs to be specified a priori. Consequently, we present a new sampling technique to optimize the evaluation of preventive strategies using fixed budget best-arm identification algorithms. We use epidemiological modeling theory to derive knowledge about the reward distribution which we exploit using Bayesian best-arm identification algorithms (i.e., Top-two Thompson sampling and BayesGap). We evaluate these algorithms in a realistic experimental setting and demonstrate that it is possible to identify the optimal strategy using only a limited number of model evaluations, i.e., 2-to-3 times faster compared to the uniform sampling method, the predominant technique used for epidemiological decision making in the literature. Finally, we contribute and evaluate a statistic for Top-two Thompson sampling to inform the decision makers about the confidence of an arm recommendation. Code related to this paper is available at:


Pandemic influenza Multi-armed bandits Fixed budget best-arm identification Preventive strategies Individual-based models 



Pieter Libin and Timothy Verstraeten were supported by a PhD grant of the FWO (Fonds Wetenschappelijk Onderzoek - Vlaanderen). Kristof Theys, Jelena Grujic and Diederik Roijers were supported by a postdoctoral grant of the FWO. The computational resources were provided by an EWI-FWO grant (Theys, KAN2012 We thank the anonymous reviewers for their insightful comments that allowed us to improve this work.

Supplementary material

473908_1_En_28_MOESM1_ESM.pdf (8.6 mb)
Supplementary material 1 (pdf 8771 KB)


  1. 1.
    Audibert, J.Y., Bubeck, S.: Best arm identification in multi-armed bandits. In: COLT-23th Conference on Learning Theory (2010)Google Scholar
  2. 2.
    Basta, N.E., Chao, D.L., Halloran, M.E., Matrajt, L., Longini, I.M.: Strategies for pandemic and seasonal influenza vaccination of schoolchildren in the United States. Am. J. Epidemiol. 170(6), 679–686 (2009)CrossRefGoogle Scholar
  3. 3.
    Britton, T.: Stochastic epidemic models: a survey. Math. Biosci. 225(1), 24–35 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Bubeck, S., Munos, R., Stoltz, G.: Pure exploration in multi-armed bandits problems. In: Gavaldà, R., Lugosi, G., Zeugmann, T., Zilles, S. (eds.) ALT 2009. LNCS (LNAI), vol. 5809, pp. 23–37. Springer, Heidelberg (2009). Scholar
  5. 5.
    Bubeck, S., Munos, R., Stoltz, G.: Pure exploration in finitely-armed and continuous-armed bandits. Theor. Comput. Sci. 412(19), 1832–1852 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Chao, D.L., Halloran, M.E., Obenchain, V.J., Longini Jr., I.M.: FluTE, a publicly available stochastic influenza epidemic simulation model. PLoS Comput. Biol. 6(1), e1000656 (2010)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Chao, D.L., Halstead, S.B., Halloran, M.E., Longini, I.M.: Controlling Dengue with Vaccines in Thailand. PLoS Negl. Trop. Dis. 6(10), e1876 (2012)CrossRefGoogle Scholar
  8. 8.
    Chapelle, O., Li, L.: An empirical evaluation of Thompson sampling. In: Advances in Neural Information Processing Systems, pp. 2249–2257 (2011)Google Scholar
  9. 9.
    Dorigatti, I., Cauchemez, S., Pugliese, A., Ferguson, N.M.: A new approach to characterising infectious disease transmission dynamics from sentinel surveillance: application to the Italian 2009/2010 A/H1N1 influenza pandemic. Epidemics 4(1), 9–21 (2012)CrossRefGoogle Scholar
  10. 10.
    Enserink, M.: Crisis underscores fragility of vaccine production system. Science 306(5695), 385 (2004)CrossRefGoogle Scholar
  11. 11.
    Eubank, S., Kumar, V., Marathe, M., Srinivasan, A., Wang, N.: Structure of social contact networks and their impact on epidemics. DIMACS Ser. Discrete Math. Theor. Comput. Sci 70(0208005), 181 (2006)MathSciNetGoogle Scholar
  12. 12.
    Even-Dar, E., Mannor, S., Mansour, Y.: Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. J. Mach. Learn. Res. 7(Jun), 1079–1105 (2006)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Ferguson, N.M., Cummings, D.A.T., Cauchemez, S., Fraser, C.: Others: strategies for containing an emerging influenza pandemic in Southeast Asia. Nature 437(7056), 209 (2005)CrossRefGoogle Scholar
  14. 14.
    Fraser, C., Cummings, D.A.T., Klinkenberg, D., Burke, D.S., Ferguson, N.M.: Influenza transmission in households during the 1918 pandemic. Am. J. Epidemiol. 174(5), 505–514 (2011)CrossRefGoogle Scholar
  15. 15.
    Fumanelli, L., Ajelli, M., Merler, S., Ferguson, N.M., Cauchemez, S.: Model-based comprehensive analysis of school closure policies for mitigating influenza epidemics and pandemics. PLoS Comput. Biol. 12(1), e1004681 (2016)CrossRefGoogle Scholar
  16. 16.
    Garivier, A., Kaufmann, E.: Optimal best arm identification with fixed confidence. In: Conference on Learning Theory, pp. 998–1027 (2016)Google Scholar
  17. 17.
    Germann, T.C., Kadau, K., Longini, I.M., Macken, C.A.: Mitigation strategies for pandemic influenza in the United States. Proc. Nat. Acad. Sci. U.S.A. 103(15), 5935–5940 (2006)CrossRefGoogle Scholar
  18. 18.
    Halloran, M.E., Longini, I.M., Nizam, A., Yang, Y.: Containing bioterrorist smallpox. Science (New York, N.Y.) 298(5597), 1428–1432 (2002)CrossRefGoogle Scholar
  19. 19.
    Hartfield, M., Alizon, S.: Introducing the outbreak threshold in epidemiology. PLoS Pathog 9(6), e1003277 (2013)CrossRefGoogle Scholar
  20. 20.
    Herbert, R.: Some aspects of the sequential design of experiments. Bull. Am. Math. Soc. 58(5), 527–535 (1952)MathSciNetzbMATHCrossRefGoogle Scholar
  21. 21.
    Hoffman, M., Shahriari, B., Freitas, N.: On correlation and budget constraints in model-based bandit optimization with application to automatic machine learning. In: Artificial Intelligence and Statistics, pp. 365–374 (2014)Google Scholar
  22. 22.
    Honda, J., Takemura, A.: Optimality of Thompson sampling for Gaussian bandits depends on priors. In: AISTATS, pp. 375–383 (2014)Google Scholar
  23. 23.
    Jennison, C., Johnstone, I.M., Turnbull, B.W.: Asymptotically optimal procedures for sequential adaptive selection of the best of several normal means. Stat. Decis. Theory Relat. Top. III 2, 55–86 (1982)MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    Kaufmann, E., Cappé, O., Garivier, A.: On the complexity of best arm identification in multi-armed bandit models. J. Mach. Learn. Res. 17(1), 1–42 (2016)MathSciNetzbMATHGoogle Scholar
  25. 25.
    Kaufmann, E., Kalyanakrishnan, S.: Information complexity in bandit subset selection. In: Conference on Learning Theory, pp. 228–251 (2013)Google Scholar
  26. 26.
    Libin, P., Verstraeten, T., Theys, K., Roijers, D.M., Vrancx, P., Nowé, A.: Efficient evaluation of influenza mitigation strategies using preventive bandits. In: Sukthankar, G., Rodriguez-Aguilar, J.A. (eds.) AAMAS 2017. LNCS (LNAI), vol. 10643, pp. 67–85. Springer, Cham (2017). Scholar
  27. 27.
    Lloyd-Smith, J.O., Schreiber, S.J., Kopp, P.E., Getz, W.M.: Superspreading and the effect of individual variation on disease emergence. Nature 438(7066), 355–359 (2005)CrossRefGoogle Scholar
  28. 28.
    Medlock, J., Galvani, A.P.: Optimizing influenza vaccine distribution. Science 325(5948), 1705–1708 (2009)CrossRefGoogle Scholar
  29. 29.
    Paules, C., Subbarao, K.: Influenza. The Lancet (2017)CrossRefGoogle Scholar
  30. 30.
    Powell, W.B., Ryzhov, I.O.: Optimal Learning, vol. 841. Wiley, Hoboken (2012)CrossRefGoogle Scholar
  31. 31.
    Russo, D.: Simple Bayesian algorithms for best arm identification. In: Conference on Learning Theory, pp. 1417–1418 (2016)Google Scholar
  32. 32.
    Watts, D.J., Muhamad, R., Medina, D.C., Dodds, P.S.: Multiscale, resurgent epidemics in a hierarchical metapopulation model. Proc. Nat. Acad. Sci. U.S.A. 102(32), 11157–11162 (2005)CrossRefGoogle Scholar
  33. 33.
    WHO: WHO guidelines on the use of vaccines and antivirals during influenza pandemics (2004)Google Scholar
  34. 34.
    Willem, L., Stijven, S., Vladislavleva, E., Broeckhove, J., Beutels, P., Hens, N.: Active learning to understand infectious disease models and improve policy making. PLoS Comput. Biol. 10(4), e1003563 (2014)CrossRefGoogle Scholar
  35. 35.
    Wu, J.T., Riley, S., Fraser, C., Leung, G.M.: Reducing the impact of the next influenza pandemic using household-based public health interventions. PLoS Med. 3(9), e361 (2006)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Pieter J. K. Libin
    • 1
    • 2
    Email author
  • Timothy Verstraeten
    • 1
  • Diederik M. Roijers
    • 1
  • Jelena Grujic
    • 1
  • Kristof Theys
    • 2
  • Philippe Lemey
    • 2
  • Ann Nowé
    • 1
  1. 1.Artificial Intelligence Lab, Department of Computer ScienceVrije Universiteit BrusselBrusselsBelgium
  2. 2.Rega Institute for Medical Research, Clinical and Epidemiological VirologyKU Leuven - University of LeuvenLeuvenBelgium

Personalised recommendations