Skip to main content
SpringerLink
Log in

Towards Reinforcement Learning-based Aggregate Computing

  • Conference paper
  • First Online:
Coordination Models and Languages (COORDINATION 2022)

Part of the book series: IFIP Advances in Information and Communication Technology ((LNCS,volume 13271))

Included in the following conference series:

Abstract

Recent trends in pervasive computing promote the vision of Collective Adaptive Systems (CASs): large-scale collections of relatively simple agents that act and coordinate with no central orchestrator to support distributed applications. Engineering global behaviour out of local activity and interaction, however, is a difficult task, typically addressed by try-and-error approaches in simulation environments. In the context of Aggregate Computing (AC), a prominent functional programming approach for CASs based on field-based coordination, this difficulty is reflected in the design of versatile algorithms preserving efficiency in a variety of environments. To deal with this complexity, in this work we propose to apply Machine Learning techniques to automatically devise local actions to improve over manually-defined AC algorithms specifications. Most specifically, we adopt a Reinforcement Learning-based approach to let a collective learn local policies to improve over the standard gradient algorithm—a cornerstone brick of several higher-level self-organisation algorithms. Our evaluation shows that the learned policies can speed up the self-stabilisation of the gradient to external perturbations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/cric96/experiment-2022-coordination.

References

  1. Aguzzi, G.: Research directions for aggregate computing with machine learning. In: IEEE International Conference on Autonomic Computing and Self-Organizing Systems, ACSOS 2021, Companion Volume, pp. 310–312. IEEE (2021). https://doi.org/10.1109/ACSOS-C52956.2021.00078

  2. Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: Deep reinforcement learning: a brief survey. IEEE Signal Process. Mag. 34(6), 26–38 (2017). https://doi.org/10.1109/MSP.2017.2743240

    Article  Google Scholar 

  3. Audrito, G., Casadei, R., Damiani, F., Pianini, D., Viroli, M.: Optimal resilient distributed data collection in mobile edge environments. Comput. Electr. Eng. 96(Part), 107580 (2021). https://doi.org/10.1016/j.compeleceng.2021.107580

  4. Audrito, G., Casadei, R., Damiani, F., Viroli, M.: Compositional blocks for optimal self-healing gradients. In: 11th IEEE International Conference on Self-Adaptive and Self-Organizing Systems, SASO 2017, pp. 91–100. IEEE Computer Society (2017). https://doi.org/10.1109/SASO.2017.18

  5. Barbalios, N., Tzionas, P.: A robust approach for multi-agent natural resource allocation based on stochastic optimization algorithms. Appl. Soft Comput. 18, 12–24 (2014). https://doi.org/10.1016/j.asoc.2014.01.004

  6. Beal, J., Bachrach, J., Vickery, D., Tobenkin, M.M.: Fast self-healing gradients. In: Proceedings of the 2008 ACM Symposium on Applied Computing (SAC), pp. 1969–1975. ACM (2008). https://doi.org/10.1145/1363686.1364163

  7. Beal, J., Dulman, S., Usbeck, K., Viroli, M., Correll, N.: Organizing the aggregate: languages for spatial computing. In: Formal and Practical Aspects of Domain-Specific Languages: Recent Developments, pp. 436–501. IGI Global (2013)

    Google Scholar 

  8. Beal, J., Pianini, D., Viroli, M.: Aggregate programming for the internet of things. Computer 48(9), 22–30 (2015). https://doi.org/10.1109/MC.2015.261

    Article  Google Scholar 

  9. Bucchiarone, A., et al.: On the social implications of collective adaptive systems. IEEE Technol. Soc. Mag. 39(3), 36–46 (2020). https://doi.org/10.1109/MTS.2020.3012324

    Article  Google Scholar 

  10. Casadei, R.: Macroprogramming: concepts, state of the art, and opportunities of macroscopic behaviour modelling. CoRR abs/2201.03473 (2022)

    Google Scholar 

  11. Casadei, R., Aguzzi, G., Viroli, M.: A programming approach to collective autonomy. J. Sens. Actuator Networks 10(2), 27 (2021). https://doi.org/10.3390/jsan10020027

    Article  Google Scholar 

  12. Casadei, R., Pianini, D., Placuzzi, A., Viroli, M., Weyns, D.: Pulverization in cyber-physical systems: engineering the self-organizing logic separated from deployment. Future Internet 12(11), 203 (2020). https://doi.org/10.3390/fi12110203

    Article  Google Scholar 

  13. Casadei, R., Pianini, D., Viroli, M., Natali, A.: Self-organising coordination regions: a pattern for edge computing. In: Riis Nielson, H., Tuosto, E. (eds.) COORDINATION 2019. LNCS, vol. 11533, pp. 182–199. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22397-7_11

    Chapter  Google Scholar 

  14. Casadei, R., Viroli, M., Audrito, G., Damiani, F.: FScaFi: a core calculus for collective adaptive systems programming. In: Margaria, T., Steffen, B. (eds.) ISoLA 2020. LNCS, vol. 12477, pp. 344–360. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61470-6_21

    Chapter  Google Scholar 

  15. Casadei, R., Viroli, M., Ricci, A., Audrito, G.: Tuple-based coordination in large-scale situated systems. In: Damiani, F., Dardha, O. (eds.) COORDINATION 2021. LNCS, vol. 12717, pp. 149–167. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-78142-2_10

    Chapter  Google Scholar 

  16. Foerster, J.N.: Deep multi-agent reinforcement learning. Ph.D. thesis, University of Oxford, UK (2018)

    Google Scholar 

  17. Jang, B., Kim, M., Harerimana, G., Kim, J.W.: Q-learning algorithms: a comprehensive classification and applications. IEEE Access 7, 133653–133667 (2019). https://doi.org/10.1109/ACCESS.2019.2941229

    Article  Google Scholar 

  18. Kephart, J.O., Chess, D.M.: The vision of autonomic computing. Computer 36(1), 41–50 (2003). https://doi.org/10.1109/MC.2003.1160055

    Article  MathSciNet  Google Scholar 

  19. Kernbach, S., Schmickl, T., Timmis, J.: Collective adaptive systems: Challenges beyond evolvability. CoRR abs/1108.5643 (2011)

    Google Scholar 

  20. Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32(11), 1238–1274 (2013). https://doi.org/10.1177/0278364913495721

    Article  Google Scholar 

  21. Lauer, M., Riedmiller, M.A.: An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In: Langley, P. (ed.) Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 - July 2, 2000, pp. 535–542. Morgan Kaufmann (2000)

    Google Scholar 

  22. Luong, N.C., et al.: Applications of deep reinforcement learning in communications and networking: a survey. IEEE Commun. Surv. Tutorials 21(4), 3133–3174 (2019). https://doi.org/10.1109/COMST.2019.2916583

    Article  Google Scholar 

  23. Mamei, M., Zambonelli, F.: Programming pervasive and mobile computing applications: the TOTA approach. ACM Trans. Softw. Eng. Methodol. 18(4), 15:1–15:56 (2009). https://doi.org/10.1145/1538942.1538945

  24. Mamei, M., Zambonelli, F., Leonardi, L.: Co-fields: a physically inspired approach to motion coordination. IEEE Pervasive Comput. 3(2), 52–61 (2004). https://doi.org/10.1109/MPRV.2004.1316820

    Article  Google Scholar 

  25. Matignon, L., Laurent, G.J., Fort-Piat, N.L.: Hysteretic q-learning : an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 64–69. IEEE (2007). https://doi.org/10.1109/IROS.2007.4399095

  26. Matignon, L., Laurent, G.J., Fort-Piat, N.L.: Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems. Knowl. Eng. Rev. 27(1), 1–31 (2012). https://doi.org/10.1017/S0269888912000057

    Article  Google Scholar 

  27. Mo, Y., Beal, J., Dasgupta, S.: An aggregate computing approach to self-stabilizing leader election. In: 2018 IEEE 3rd International Workshops on Foundations and Applications of Self* Systems (FAS*W), Trento, Italy, September 3–7, 2018, pp. 112–117. IEEE (2018). https://doi.org/10.1109/FAS-W.2018.00034

  28. Mousavi, S.S., Schukat, M., Howley, E.: Deep reinforcement learning: An overview. CoRR abs/1806.08894 (2018)

    Google Scholar 

  29. Müller-Schloer, C., Sick, B.: Controlled emergence and self-organization. In: Organic Computing. UCS, pp. 81–103. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-540-77657-4_4

    Chapter  Google Scholar 

  30. Nagpal, R., Shrobe, H., Bachrach, J.: Organizing a global coordinate system from local information on an Ad Hoc sensor network. In: Zhao, F., Guibas, L. (eds.) IPSN 2003. LNCS, vol. 2634, pp. 333–348. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-36978-3_22

    Chapter  MATH  Google Scholar 

  31. De Nicola, R., Jähnichen, S., Wirsing, M.: Rigorous engineering of collective adaptive systems: special section. Int. J. Softw. Tools Technol. Transf. 22(4), 389–397 (2020). https://doi.org/10.1007/s10009-020-00565-0

    Article  Google Scholar 

  32. Omicini, A.: Nature-inspired coordination for complex distributed systems. In: Intelligent Distributed Computing VI - Proceedings of the 6th International Symposium on Intelligent Distributed Computing - IDC 2012. Studies in Computational Intelligence, vol. 446, pp. 1–6. Springer, Cham (2012). https://doi.org/10.1007/978-3-642-32524-3_1

  33. Panait, L., Luke, S.: Cooperative multi-agent learning: the state of the art. Auton. Agents Multi Agent Syst. 11(3), 387–434 (2005). https://doi.org/10.1007/s10458-005-2631-2

    Article  Google Scholar 

  34. Pianini, D., Casadei, R., Viroli, M., Mariani, S., Zambonelli, F.: Time-fluid field-based coordination through programmable distributed schedulers. Logical Methods Comput. Sci. 17(4) (2021). https://doi.org/10.46298/lmcs-17(4:13)2021

  35. Pianini, D., Montagna, S., Viroli, M.: Chemical-oriented simulation of computational systems with ALCHEMIST. J. Simulation 7(3), 202–215 (2013). https://doi.org/10.1057/jos.2012.27

    Article  Google Scholar 

  36. Solar-Lezama, A.: Program Synthesis by Sketching. University of California, Berkeley (2008)

    Google Scholar 

  37. Sosic, A., KhudaBukhsh, W.R., Zoubir, A.M., Koeppl, H.: Inverse reinforcement learning in swarm systems. In: Larson, K., Winikoff, M., Das, S., Durfee, E.H. (eds.) Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2017, São Paulo, Brazil, May 8–12, 2017, pp. 1413–1421. ACM (2017)

    Google Scholar 

  38. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)

    Google Scholar 

  39. Tan, M.: Multi-agent reinforcement learning: independent versus cooperative agents. In: Utgoff, P.E. (ed.) Machine Learning, Proceedings of the Tenth International Conference, University of Massachusetts, Amherst, MA, USA, June 27–29, 1993, pp. 330–337. Morgan Kaufmann (1993). https://doi.org/10.1016/b978-1-55860-307-3.50049-6

  40. Trzec, K., Lovrek, I.: Field-based coordination of mobile intelligent agents: an evolutionary game theoretic analysis. In: Apolloni, B., Howlett, R.J., Jain, L. (eds.) KES 2007. LNCS (LNAI), vol. 4692, pp. 198–205. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74819-9_25

    Chapter  Google Scholar 

  41. Viroli, M., Beal, J., Damiani, F., Audrito, G., Casadei, R., Pianini, D.: From distributed coordination to field calculus and aggregate computing. J. Log. Algebraic Methods Program. 109, 100486 (2019). https://doi.org/10.1016/j.jlamp.2019.100486

  42. Watkins, C.J.C.H., Dayan, P.: Technical note q-learning. Mach. Learn. 8, 279–292 (1992). https://doi.org/10.1007/BF00992698

    Article  MATH  Google Scholar 

  43. De Wolf, T., Holvoet, T.: Emergence versus self-organisation: different concepts but promising when combined. In: Brueckner, S.A., Di Marzo Serugendo, G., Karageorgos, A., Nagpal, R. (eds.) ESOA 2004. LNCS (LNAI), vol. 3464, pp. 1–15. Springer, Heidelberg (2005). https://doi.org/10.1007/11494676_1

    Chapter  Google Scholar 

  44. Wolf, T.D., Holvoet, T.: Designing self-organising emergent systems based on information flows and feedback-loops. In: Proceedings of the First International Conference on Self-Adaptive and Self-Organizing Systems, SASO 2007, Boston, MA, USA, July 9–11, 2007, pp. 295–298. IEEE Computer Society (2007). https://doi.org/10.1109/SASO.2007.16

  45. Xiao, D., Hubbold, R.J.: Navigation guided by artificial force fields. In: Atwood, M.E., Karat, C., Lund, A.M., Coutaz, J., Karat, J. (eds.) Proceeding of the CHI 1998 Conference on Human Factors in Computing Systems, pp. 179–186. ACM (1998). https://doi.org/10.1145/274644.274671

  46. Xu, Y., Zhang, W., Liu, W., Ferrese, F.T.: Multiagent-based reinforcement learning for optimal reactive power dispatch. IEEE Trans. Syst. Man Cybern. Part C 42(6), 1742–1751 (2012). https://doi.org/10.1109/TSMCC.2012.2218596

Download references

Acknowledgements

This work has been supported by the MIUR FSE REACT-EU PON R&I 2014–2022 (CCI2014IT16M2OP005).

Author information

Authors and Affiliations

  1. Alma Mater Studiorum - Università di Bologna, Cesena, Italy

    Gianluca Aguzzi, Roberto Casadei & Mirko Viroli

Authors
  1. Gianluca Aguzzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roberto Casadei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mirko Viroli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gianluca Aguzzi .

Editor information

Editors and Affiliations

  1. Istituto di Scienza e Tecnologie dell'In, Pisa, Italy

    Maurice H. ter Beek

  2. Mälardalen University, Västerås, Sweden

    Marjan Sirjani

Rights and permissions

Reprints and permissions

Copyright information

© 2022 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Aguzzi, G., Casadei, R., Viroli, M. (2022). Towards Reinforcement Learning-based Aggregate Computing. In: ter Beek, M.H., Sirjani, M. (eds) Coordination Models and Languages. COORDINATION 2022. IFIP Advances in Information and Communication Technology, vol 13271. Springer, Cham. https://doi.org/10.1007/978-3-031-08143-9_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-08143-9_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-08145-3

  • Online ISBN: 978-3-031-08143-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation