Advertisement

Deep Reinforcement Learning for Task-Driven Discovery of Incomplete Networks

  • Peter MoralesEmail author
  • Rajmonda Sulo Caceres
  • Tina Eliassi-Rad
Conference paper
First Online:
Part of the Studies in Computational Intelligence book series (SCI, volume 881)

Abstract

Complex networks are often either too large for full exploration, partially accessible, or partially observed. Downstream learning tasks on these incomplete networks can produce low quality results. In addition, reducing the incompleteness of the network can be costly and nontrivial. As a result, network discovery algorithms optimized for specific downstream learning tasks given resource collection constraints are of great interest. In this paper, we formulate the task-specific network discovery problem in an incomplete network setting as a sequential decision making problem. Our downstream task is selective harvesting, the optimal collection of vertices with a particular attribute. We propose a framework, called Network Actor Critic (NAC), which learns a policy and notion of future reward in an offline setting via a deep reinforcement learning algorithm. A quantitative study is presented on several synthetic and real benchmarks. We show that offline models of reward and network discovery policies lead to significantly improved performance when compared to competitive online discovery algorithms.

Keywords

Network discovery Incomplete networks Deep reinforcement learning 
This is a preview of subscription content, log in to check access.

References

  1. 1.
    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015) CrossRefGoogle Scholar
  2. 2.
    Heess, N., Dhruva, T.B., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Ali Eslami, S.M., Riedmiller, M.A., Silver, D.: Emergence of locomotion behaviours in rich environments. CoRR, abs/1707.02286 (2017)Google Scholar
  3. 3.
    Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., Hassabis, D.: Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017)CrossRefGoogle Scholar
  4. 4.
    Wang, X., Garnett, R., Schneider, J.: Active search on graphs. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2013)Google Scholar
  5. 5.
    LaRock, T., Sakharov, T., Bhadra, S., Eliassi-Rad, T.: Reducing network incompleteness through online learning: a feasibility study. In: The 14th International Workshop on Mining and Learning with Graphs (2018)Google Scholar
  6. 6.
    Soundarajan, S., Eliassi-Rad, T., Gallagher, B., Pinar, A.: MaxOutProbe: an algorithm for increasing the size of partially observed networks. CoRR, abs/1511.06463 (2015)Google Scholar
  7. 7.
    Soundarajan, S., Eliassi-Rad, T., Gallagher, B., Pinar, A.: MaxReach: reducing network incompleteness through node probes. In: ASONAM, pp 152–157 (2016)Google Scholar
  8. 8.
    Avrachenkov, K., Basu, P., Neglia, G., Ribeiro, B., Towsley, D.: Pay few, influence most: online myopic network covering. In: IEEE Conference on Computer Communications Workshops, pp. 813–818 (2014)Google Scholar
  9. 9.
    Murai, F., Rennó, D., Ribeiro, B., Pappa, G.L., Towsley, D.F., Gile, K.: Selective harvesting over networks. Data Min. Knowl. Discov. 32(1), 187–217 (2017)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Zhang, Z., Cui, P., Zhu, W.: Deep learning on graphs: a survey. CoRR, abs/1812.04202 (2018)Google Scholar
  11. 11.
    You, J., Liu, B., Ying, R., Pande, V., Leskovec, J.: Graph convolutional policy network for goal-directed molecular graph generation. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6412–6422 (2018)Google Scholar
  12. 12.
    Mofrad, M.H., Melhem, R., Hammoud, M.: Partitioning graphs for the cloud using reinforcement learning. CoRR, abs/1907.06768 (2019)Google Scholar
  13. 13.
    De Cao, N., Kipf, T.: MolGAN: an implicit generative model for small molecular graphs, CoRR, abs/1805.11973 (2018)Google Scholar
  14. 14.
    Dai, H., Khalil, E.B., Zhang, Y., Dilkina, B., Song, L.: Learning combinatorial optimization algorithms over graphs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6351–6361 (2017)Google Scholar
  15. 15.
    Ho, C., Kochenderfer, M.J., Mehta, V., Caceres, R.S.: Control of epidemics on graphs. In: 54th IEEE Conference on Decision and Control, pp. 4202–4207 (2015)Google Scholar
  16. 16.
    Goindani, M., Neville, J.: Social reinforcement learning to combat fake news spread. In: Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence (2019)Google Scholar
  17. 17.
    Mittal, A., Dhawan, A., Medya, S., Ranu, S., Singh, A.K.: Learning heuristics over large graphs via deep reinforcement learning. CoRR, abs/1903.03332 (2019)Google Scholar
  18. 18.
    Haveliwala, T.H.: Topic-sensitive pagerank: a context-sensitive ranking algorithm for web search. IEEE Trans. Knowl. Data Eng. 15(4), 784–796 (2003)CrossRefGoogle Scholar
  19. 19.
    Kloumann, I.M., Ugander, J., Kleinberg, J.: Block models and personalized PageRank. Proc. Natl. Acad. Sci. 114(1), 33–38 (2017)CrossRefGoogle Scholar
  20. 20.
    Gleich, D.: PageRank beyond the web. SIAM Rev. 57 (2014).  https://doi.org/10.1137/140976649MathSciNetCrossRefGoogle Scholar
  21. 21.
    Lancichinetti, A., Fortunato, S., Radicchi, F.: Benchmark graphs for testing community detection algorithms. Phys. Rev. E 78, 046110 (2008) CrossRefGoogle Scholar
  22. 22.
    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. CoRR, abs/1707.06347 (2017)Google Scholar
  23. 23.
    Nadakuditi, R.R., Newman, M.E.J.: Graph spectra and the detectability of community structure in networks. CoRR, abs/1205.1813 (2012)Google Scholar
  24. 24.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR, abs/1412.6980 (2014)Google Scholar
  25. 25.
    Holland, P.W., Laskey, K.B., Leinhardt, S.: Stochastic blockmodels: first steps. Soc. Netw. 5(2), 109–137 (1983)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Erdös, P., Rényi, A.: On random graphs. Publicationes Mathematicae 6, 290–297 (1959)MathSciNetzbMATHGoogle Scholar
  27. 27.
    Rozemberczki, B., Davies, R., Sarkar, R., Sutton, C.A.: GEMSEC: graph embedding with self clustering. CoRR, abs/1802.03997 (2018)Google Scholar
  28. 28.
    Avrachenkov, K., Borkar, V.S., Kadavankandy, A., Sreedharan, J.K.: Comparison of random walk based techniques for estimating network averages. In: International Conference on Computational Social Networks, pp. 27–38 (2016)CrossRefGoogle Scholar
  29. 29.
    Avrachenkov, K., Borkar, V.S.,Kadavankandy, A., Sreedharan, J.K.: Revisiting random walk based sampling in networks: evasion of burn-in period and frequent regenerations. Comput. Soc. Netw. (2018)Google Scholar
  30. 30.
    Avrachenkov, K., Litvak, N., Nemirovsky, D., Smirnova, E., Sokol, M.: Quick detection of top-k personalized pagerank lists. In: International Workshop on Algorithms and Models for The Web-Graph, pp. 50–61 (2011)Google Scholar
  31. 31.
    Borkar, V.S., Mathkar, A.S.: Reinforcement learning for matrix computations: pagerank as an example. In: International Conference on Distributed Computing and Internet Technology, pp. 14–24 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Peter Morales
    • 1
    Email author
  • Rajmonda Sulo Caceres
    • 1
  • Tina Eliassi-Rad
    • 2
  1. 1.MIT Lincoln LaboratoryLexingtonUSA
  2. 2.Northeastern UniversityBostonUSA

Personalised recommendations

Cite paper

Buy options