Partially Observable Reinforcement Learning for Sustainable Active Surveillance

  • Hechang Chen
  • Bo Yang
  • Jiming Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11062)


Active surveillance is the most effective strategy in the applications of infectious disease prevention, road network optimization, crime reconnaissance, etc. However, the incomplete data collected from partially monitored regions by active surveillance disables existing models to maintain a sustainable performance in the future. To address this issue, this article presents a sustainable active surveillance framework (SAS), which consists of a predictor, a classifier, and a planner, by developing a novel partially observable reinforcement learning algorithm. The predictor estimates priorities of candidate regions for monitoring. The classifier assigns candidate regions with similar features into the same groups, so that the data collected from monitored regions can be shared with unmonitored regions within the group. The planner determines where and when to allocate limited resources, considering the outcomes of available resources and model sustainability. An empirical case study on infectious disease prevention showed that the proposed SAS method significantly outperforms the state-of-the-art methods.


Sustainable active surveillance Resources allocation Reinforcement learning Neural networks 


  1. 1.
    Abekawa, T.: Framework of automatic text summarization using reinforcement learning. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Jeju Island, Korea, pp. 256–265, July 2012Google Scholar
  2. 2.
    Ceccato, P., Vancutsem, C., Klaver, R., Rowland, J., Connor, S.J.: A vectorial capacity product to monitor changing malaria transmission potential in epidemic regions of Africa. J. Trop. Med. 2012(2012), 595948 (2012)Google Scholar
  3. 3.
    Coppi, D., Calderara, S., Cucchiara, R.: Iterative active querying for surveillance data retrieval in crime detection and forensics. In: Proceedings of the International Conference on Imaging for Crime Detection and Prevention, London, UK, pp. 1–6, November 2012Google Scholar
  4. 4.
    Kiang, R., et al.: Meteorological, environmental remote sensing and neural network analysis of the epidemiology of malaria transmission in Thailand. Geospat. Health 1(1), 71–84 (2006)CrossRefGoogle Scholar
  5. 5.
    Knox, W.B., Stone, P.: Reinforcement learning from simultaneous human and MDP reward. In: International Conference on Autonomous Agents and Multiagent Systems, Minnesota, USA, pp. 4–8, June 2012Google Scholar
  6. 6.
    Kouadio, I.K., Aljunid, S., Kamigaki, T., Hammad, K., Oshitani, H.: Infectious diseases following natural disasters: prevention and control measures. Expert Rev. Anti-Infect. Ther. 10(1), 95–104 (2012)CrossRefGoogle Scholar
  7. 7.
    Lu, J., et al.: Continuing reassortment leads to the genetic diversity of influenza virus H7N9 in Guangdong, China. J. Virol. 88(15), 8297–8306 (2014)CrossRefGoogle Scholar
  8. 8.
    Pei, H., Yang, B., Liu, J., Dong, L.: Group sparse Bayesian learning for active surveillance on epidemic dynamics. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pp. 1–8. In Press, Louisiana, February 2018Google Scholar
  9. 9.
    Polydoros, A., Nalpantidis, L.: Survey of model-based reinforcement learning: applications on robotics. J. Intell. Robot. Syst. 86(2), 1–21 (2017)CrossRefGoogle Scholar
  10. 10.
    Ravishankar, N.R., Vijayakumar, M.V.: Reinforcement learning algorithms: survey and classification. Indian J. Sci. Technol. 10(1) (2017). ISSN(Print) 0974-6846, ISSN(Online) 0974-5645Google Scholar
  11. 11.
    Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Driessche, G.V.D.: Mastering the game of Go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)CrossRefGoogle Scholar
  12. 12.
    Simini, F., Gonzalez, M.C., Maritan, A., Barabasi, A.: A universal model for mobility and migration patterns. Nature 484(7392), 96–100 (2012)CrossRefGoogle Scholar
  13. 13.
    Smith, D.L., McKenzie, F.E.: Statics and dynamics of malaria infection in Anopheles mosquitoes. J. Malar. 3(1), 13 (2004)CrossRefGoogle Scholar
  14. 14.
    Sturrock, H.J.W., Hsiang, M.S., Cohen, J.M.: Targeting asymptomatic malaria infections: active surveillance in control and elimination. PLoS Med. 10(6), e1001467 (2013)CrossRefGoogle Scholar
  15. 15.
    Tay, E.L., Grant, K., Kirk, M., Mounts, A., Kelly, H.: Exploring a proposed who method to determine thresholds for seasonal influenza surveillance. PLoS ONE 8(10), e77244 (2013)CrossRefGoogle Scholar
  16. 16.
    Wang, Y., et al.: Causes of infection after earthquake, China, 2008. J. Emerg. Infect. Dis. 16(6), 974–975 (2010)CrossRefGoogle Scholar
  17. 17.
    WHO: Fact sheet: world malaria report 2016, December 2016.
  18. 18.
    WHO: Fact sheet: world malaria report 2017, November 2017.
  19. 19.
    Wu, J., Xu, X., Zhang, P., Liu, C.: A novel multi-agent reinforcement learning approach for job scheduling in grid computing. Future Gener. Comput. Syst. 27(5), 430–439 (2011)CrossRefGoogle Scholar
  20. 20.
    Xie, Z., Qin, Y., Wang, L.: Temporal-based risk forecasting approach for key areas on surveillance sensor networks of high-speed railway transport hub. In: Proceedings of the 22nd International Conference on Distributed Multimedia Systems, Florida, USA, pp. 190–193, November 2015Google Scholar
  21. 21.
    Yang, B., Guo, H., Yang, Y., Shi, B., Zhou, X., Liu, J.: Modeling and mining spatiotemporal patterns of infection risk from heterogeneous data for active surveillance planning. In: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, Quebec, Canada, pp. 493–499, January 2014Google Scholar
  22. 22.
    Zhou, W.G., Qu, Y., Wang, W.G., Tang, S.Y.: Application of health education of house-to-house visit in malaria prevention and control. Chin. J. Schistosomiasis Control 26(5), 517–521 (2014)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.College of Computer Science and TechnologyJilin UniversityChangchunChina
  2. 2.Key Laboratory of Symbolic Computation and Knowledge EngineeringMinistry of EducationChangchunChina
  3. 3.Department of Computer ScienceHong Kong Baptist UniversityHong KongChina

Personalised recommendations