Applied Intelligence

, Volume 48, Issue 5, pp 1251–1274 | Cite as

Agent systems verification : systematic literature review and mapping

Article
  • 133 Downloads

Abstract

Agent systems are distributed systems consist of agents that autonomously interact to each other in an environment to perform tasks and achieve goals. Performing verification is important to ensure correctness of agent properties and to detect faults. The objective of this review is to identify research gap and future research direction of agent systems verification. In this study, the surveys of existing techniques for checking agent properties and detecting faults during design, development and runtime phases of agent system life-cycle are presented. Search terms with relevant keywords were used to identify primary studies that relate to the topic of discussion. Next, the studies were classified based on the used techniques and the addressed properties. 231 primary studies were identified during the search process. From these studies, 49% were implemented for verification of agent systems during design, 27% during development and 25% during runtime. Model checking or model-based verification techniques are the highest proposed techniques (44%) followed by the testing and debugging during development (17%). The properties that are largely addressed by the selected studies are temporal properties (19%) and epistemic properties (9%). At the end of the review, the research gap and the future research direction are presented.

Keywords

Agent systems Multi-agent systems Verification Testing Debugging Fault management Violations detection 

Notes

Acknowledgements

This study is supported by the Fundamental Research Grant Scheme (FRGS Vot: 4F904) that sponsored by Ministry of Higher Education (MOHE). Authors would like to thank Research Management Center (RMC) Universiti Teknologi Malaysia under Research University Funding Vot: 02G31, for the research activities in making this study a success.

References

  1. 1.
    a Al N (2012) Conducting verification and validation of multi-agent systems. Int J Softw Eng Appl 3 (5):115–124. ISSN 09762221Google Scholar
  2. 2.
    Bakar NA, Selamat A (2011) Towards implementing dynamic multi-agent V&V framework. In: The third software engineering postgraduates workshop (SEPoW 2011), JB, MalaysiaGoogle Scholar
  3. 3.
    Abushark Y, Thangarajah J, Miller T, Harland J (2014) Checking consistency of agent designs against interaction protocols for early-phase defect location. In: Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems, AAMAS ’14. ISBN 978-1-4503-2738-1. International Foundation for Autonomous Agents and Multiagent Systems, Richland, pp 933–940Google Scholar
  4. 4.
    Philip A, Ali S, Roliana I, Mohd NM (2014) A systematic literature review of software requirements prioritization research. Inf Softw Technol 56(6):568–585. ISSN 09505849CrossRefGoogle Scholar
  5. 5.
    Nadeem A (2015) Requirements, formal verification and model transformations of an agent-based system: a case study. arXiv:1501.05120, 5(3)
  6. 6.
    Nadeem A, Anique A (2014) Approach for the formal modeling of requirements, verification, and architecture of a multi-agent robotic system. IJCSI Int J Comput Sci Issue 11(2):237–246Google Scholar
  7. 7.
    Nadeem A, Aisha SG, Nadeem S (2014) Requirement analysis, Architectural design and Formal verification of a multi-agent based University Information Management System. International Journal of Computer Science & Information Technology (IJCSIT) 6(5):9Google Scholar
  8. 8.
    Alberti M, Gavanelli M, Lamma E, Chesani F, Mello P, Torroni P (2004) A logic based approach to interaction design in open multi-agent systems. In: Proceedings of the 13th IEEE international workshops on enabling technologies: Infrastructures for collaborative enterprises (WETICE-2004), 2nd international workshop ”theory and practice of open computational systems (TAPOCS). IEEE Press, pp 387–392Google Scholar
  9. 9.
    Alberti M, Chesani F, Guerri A, Gavanelli M, Lamma E, Mello P, Milano M, Torroni P (2005) Expressing interaction in combinatorial auction through social integrity constraints. Proc of W(C)LP 2005-01:53–64Google Scholar
  10. 10.
    Natasha A, Mehdi D, Fahad K, Logan B, John J-J Ch, Meyer JCh (2010) Using theorem proving to verify properties of agent programs. In: Dastani M, Hindriks KV, Meyer J-JC (eds) Specification and verification of multi-agent systems. ISBN 9781441969835. Springer, US, pp 1–33Google Scholar
  11. 11.
    Natasha A, Logan B, HoangNga NN, Abdur R (2010) Automated verification of resource requirements in multi-agent systems using abstraction. In: Ron van der M, Jan-Georg S (eds) International workshop on model checking and artificial intelligence, volume 6572 of lecture notes in computer science. ISBN 978-3-642-20673-3. Springer, Berlin, pp 69–84Google Scholar
  12. 12.
    Alotaibi H, Zedan H (2010) Runtime verification of safety properties in multi-agents systems. In: 2010 10th International Conference on Intelligent Systems Design and Applications (ISDA). ISBN 9781424481354, pp 356–362Google Scholar
  13. 13.
    Aminof B, Murano A, Rubin S, Zuleger F (2016) Automatic verification of multi-agent systems in parameterised grid-environments. In: Proceedings of the 2016 international conference on autonomous agents & multiagent systems, pages 1190–1199. International Foundation for Autonomous Agents and Multiagent SystemsGoogle Scholar
  14. 14.
    Ammar B, Abdallah K (2011) Towards the formal specification and verification of multi-agent based systems. International Journal of Computer Science Issues (IJCSI) 8(4):200– 210Google Scholar
  15. 15.
    Ancona D, Briola D, El A, Seghrouchni F, Mascardi V, Taillibert P, Amal EFS, Viviana M, Davide A, Briola D, Taillibert P (2014) Efficient verification of MASs with projections. EMAS Pre-proceedings, pages 1–20Google Scholar
  16. 16.
    Anderson CA, Titler MG (2014) Development and verification of an agent-based model of opinion leadership. Implement Sci 9(1):136. ISSN 1748-5908CrossRefGoogle Scholar
  17. 17.
    Vinay G, Ashok K (2012) EARCT-environment for automated rank based continuous agent testing. Int J Comput 11(3):180–190Google Scholar
  18. 18.
    Astefanoaei L, Dastani M, Meyer J-J, de Boer FS (2009) On the semantics and verification of normative multi-agent systems. J Universal Comput Sci 15(13):2629–2652. ISSN 09486968Google Scholar
  19. 19.
    Houhamdi Z, Athamena B, Athamena B, Houhamdi Z (2012) A petri net based multi-agent system behavioral testing. Mod Appl Sci 6(3):46–57. ISSN 19131844.  https://doi.org/10.5539/mas.v6n3p46 Google Scholar
  20. 20.
    Ayed LJB, Siala F (2009) Event-b based verification of interaction properties in multi-agent systems. JSW 4(4):357–364CrossRefGoogle Scholar
  21. 21.
    Bagić M, Babac A, Kunštić M (2008) Verification of communication protocols in a multi-agent system. In: Proceedings of the 5th International Conference on Soft Computing As Transdisciplinary Science and Technology, CSTST ’08. ACM, New York, pp 286–291. ISBN 978-1-60558-046-3.  https://doi.org/10.1145/1456223.1456283
  22. 22.
    Bakar NA, Selamat A (2011) Analyzing model checking approach for multi agent system verification. In: 2011 5th Malaysian conference in software engineering, MySEC 2011. pp 95–100. ISBN 9781457715310.  https://doi.org/10.1109/MySEC.2011.6140650
  23. 23.
    Bakar NA, Selamat A (2012) Agent-based model checking verification framework. In: 2012 IEEE conference on open systems, ICOS 2012. ISBN 9781467310468Google Scholar
  24. 24.
    Bakar NA, Selamat A (2013) Runtime verification of multi-agent systems interaction quality. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). ISBN 9783642365454, vol 7802. LNAI, pp 435–444Google Scholar
  25. 25.
    Barbosa J, Leitao P (2011) Simulation of multi-agent manufacturing systems using agent-based modelling platforms. In: 2011 9th IEEE International Conference on Industrial Informatics (INDIN). pp 477–482.  https://doi.org/10.1109/INDIN.2011.6034926
  26. 26.
    Belardinelli F, Lomuscio A, Patrizi F (2014) Verification of agent-based artifact systems. J Artif Intell Res 51:333–376. ISSN 10769757.  https://doi.org/10.1613/jair.4424 MathSciNetMATHGoogle Scholar
  27. 27.
    Belardinelli F, Grossi D, Lomuscio A (2015) Finite abstractions for the verification of epistemic properties in open multi-agent systems. In: IJCAI International Joint Conference on Artificial Intelligence, 2015-Janua (IJCAI). ISSN 10450823, pp 854–860Google Scholar
  28. 28.
    Belardinelli F, Kouvaros P, Lomuscio A (2017) Parameterised verification of data-aware multi-agent systems. In: Proceedings of the twenty-sixth International Joint Conference on Artificial Intelligence, {IJCAI-17}, pp 98–104 https://doi.org/10.24963/ijcai.2017/15
  29. 29.
    Belardinelli F, Lomuscio A, Murano A, Rubin S (2017) Verification of multi-agent systems with imperfect information and public actions. In: Proceedings of the 16th Conference on Autonomous Agents and Multi-Agent Systems, AAMAS ’17. International Foundation for Autonomous Agents and Multiagent Systems, Richland, pp 1268–1276Google Scholar
  30. 30.
    Belardinelli F, Lomuscio A, Murano A, Rubin S (2017c) Verification of broadcasting multi-agent systems against an epistemic strategy logic. In: Proceedings of the twenty-sixth International Joint Conference on Artificial Intelligence, {IJCAI-17}.  https://doi.org/10.24963/ijcai.2017/14, pp 91–97
  31. 31.
    Bellifemine F, Caire G, Greenwood D (2007) Developing Multi-Agent Systems with JADE. ISBN 9780470057476Google Scholar
  32. 32.
    Benerecetti M, Cimatti A (2003) Validation of multiagent systems by symbolic model checking. Springer, Berlin, pp 32–46. ISBN 978-3-540-36540-2MATHGoogle Scholar
  33. 33.
    Bernon C, Gleizes M-P, Picard G (2007) Enhancing Self-organising emergent systems design with simulation. In: OâĂžÄôHare GMP, Ricci A, OâĂžÄôGrady M, Dikenelli O (eds) Engineering societies in the agents world vii, volume 4457 of lecture notes in computer science. Springer, Berlin, pp 284–299. ISBN 978-3-540-75522-7.  https://doi.org/10.1007/978-3-540-75524-1_16
  34. 34.
    Berrada M, Bounabat B, Harti M (2007) Qualitative verification of multi-agents reactive decisional system using business process modeling notation. In: Proceedings - 2006 IEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT 2006 Main Conference Proceedings), IAT’06. ISBN 9780769527482.  https://doi.org/10.1109/IAT.2006.105, pp 747–751
  35. 35.
    Bijani S, Robertson D (2012) A review of attacks and security approaches in open multi-agent systems. ISSN 02692821Google Scholar
  36. 36.
    Bordini RH, Fisher M, Wooldridge M, Visser W (2004) Model checking rational agents. IEEE Intell Syst 19(5):2004. ISSN 15411672.  https://doi.org/10.1109/MIS.2004.47 CrossRefGoogle Scholar
  37. 37.
    Bordini RH, Fisher M, Visser W, Wooldridge M (2006) Verifying multi-agent programs by model checking.  https://doi.org/10.1007/s10458-006-5955-7, vol 12, p 2006
  38. 38.
    Bordini RH, Hübner JF, Wooldridge M (2007) Programming Multi-Agent Systems in AgentSpeak using Jason. ISBN 9780470029008Google Scholar
  39. 39.
    Bordini RH, Dennis LA, Farwer B, Fisher M (2008) Automated verification of multi-agent programs. In: 23rd IEEE/ACM International Conference on Automated Software Engineering ASE 2008.  https://doi.org/10.1109/ASE.2008.17, pp 69–78
  40. 40.
    Bordini RH, Dennis LA, Farwer B, Fisher M (2010) Directions for agent model checking*. In: Dastani M, Hindriks K V, Meyer J-JC (eds) Specification and verification of multi-agent systems. ISBN 978-1-4419-6983-5. Springer, US, pp 103–123Google Scholar
  41. 41.
    Bordini R, Dastani M, Winikoff M (2007) Current issues in multi-agent systems developmentGoogle Scholar
  42. 42.
    Bosse T, Jonker CM, van der Meij L, Sharpanskykh A, Treur J (2006) Specification and verification of dynamics in cognitive agent models. In: IEEE/WIC/ACM International Conference on Intelligent Agent Technology, 2006. IAT ’06.  https://doi.org/10.1109/IAT.2006.112, pp 247–254
  43. 43.
    Bosse T, Lam DN, Barber KS (2008) Tools for analyzing intelligent agent systems. Web Intelli Agent Sys 6(4):355–371. ISSN 15701263.  https://doi.org/10.3233/WIA-2008-0145 Google Scholar
  44. 44.
    Bosse T, Jonker CM, van der Meij L, Sharpanskykh A, Treur J (2009) Specification and verification of dynamics in agent models. Int J Coop Inf Syst 18(01):167–193. ISSN 0218-8430.  https://doi.org/10.1142/S0218843009001987 CrossRefGoogle Scholar
  45. 45.
    Botia JA, Gomez-Sanz JJ, Pavon J (2006) Intelligent data analysis for the verification of multi-agent systems interactions. In: Corchado E, Yin H, Botti V, Fyfe C (eds) International Conference on Intelligent Data Engineering and Automated Learning, volume 4224 of Lecture Notes in Computer Science. ISBN 978-3-540- 45485-4. Springer, Berlin, pp 1207–1214Google Scholar
  46. 46.
    Bourahla M, Benmohamed M (2004) Formal specification and verification of multi-agent systems. Electronic notes in theoretical computer science. In: Proceedings of the 11th Workshop on Logic, Language, Information and Computation (WoLLIC 2004) Logic, Language, Information and Computation 2004, 123 (0): 5–17, 2005. ISSN 1571-0661.  https://doi.org/10.1016/j.entcs.2004.04.042
  47. 47.
    Brahimi S, Maamri R, Sahnoun Z (2014) Validation and verification of agent and multi-agent plans in dynamic environment. CEUR Workshop Proc 1294:73–82. ISSN 16130073Google Scholar
  48. 48.
    Brazier FMT, Cornelissen F, Gustavsson R, Jonker CM, Lindeberg O, Polak B, Treur J (2004) Compositional verification of a multi-agent system for one-to-many negotiation. Appl Intell 20(2):95–117. ISSN 0924669X.  https://doi.org/10.1023/B:APIN.0000013334.33853.0c MATHCrossRefGoogle Scholar
  49. 49.
    Bresciani P, Giorgini P, Mouratidis H, Manson G (2004) Multi-agent systems and security requirements analysis. pp. 1–13.  https://doi.org/10.1007/b96018
  50. 50.
    Bulling N, Hindriks KV (2009) Towards a verification framework for communicating rational agents. Springer, Berlin, pp 177–182. ISBN 978-3-642-04143-3.  https://doi.org/10.1007/978-3-642-04143-3_16 Google Scholar
  51. 51.
    Caire G, Cossentino M, Negri A, Poggi A, Turci P (2004) Multi-agent systems implementation and testing. In: Fourth international symposium: from agent theory to agent implementation, pp 14–16Google Scholar
  52. 52.
    ÇakÄśrlar Ä, Gürcan Ö, Dikenelli O, Bora Å (2015) RatKit: Repeatable automated testing toolkit for agent-based modeling and simulation. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol 9002, pp 17–27. ISBN 9783319146263.  https://doi.org/10.1007/978-3-319-14627-0_2
  53. 53.
    Carrera A, Iglesias CA, Garijo M (2014) Beast methodology: an agile testing methodology for multi-agent systems based on behaviour driven development. Inf Syst Front 16(2):169–182. ISSN 1387-3326.  https://doi.org/10.1007/s10796-013-9438-5 CrossRefGoogle Scholar
  54. 54.
    Casey W, Wright E, Morales JA, Appel M, Gennari J, Mishra B (2014) Agent-based trace learning in a recommendation-verification system for cybersecurity. In: Malicious and Unwanted Software: The Americas (MALWARE), 2014 9th International Conference.  https://doi.org/10.1109/MALWARE.2014.6999404, pp 135–143
  55. 55.
    Čermák P, Lomuscio A, Mogavero F, Murano A (2017) Practical Verification of Multi-Agent Systems against Slk Specifications. Information and Computation. ISSN 0890-5401Google Scholar
  56. 56.
    Chen J, Yang YL, Liu ZG (2013) Research on method of property verification of multi-agent system CGF architecture model. Appl Mech Mater 344:294–295. ISSN 16609336.  https://doi.org/10.4028/www.scientific.net/AMM.344.294 CrossRefGoogle Scholar
  57. 57.
    Choi J, Tsourdos A (2011) Verification of decision making behaviour for heterogeneous multi-agent system.  https://doi.org/10.2514/6.2011-6239
  58. 58.
    Choi J, Tsourdos A, White B (2010) Verification of multi-agent systems using formal approach. In: AIAA Guidance, Navigation, and Control Conference, pages —-. American Institute of Aeronautics and Astronautics. ISBN 978-1-60086-962-4.  https://doi.org/10.2514/6.2010-8042
  59. 59.
    Choi J, Kim S, Tsourdos A (2013) Verification of heterogeneous multi-agent system using MCMAS. Int J Syst Sci 0(0):1–18.  https://doi.org/10.1080/00207721.2013.793890 MATHGoogle Scholar
  60. 60.
    Choi J, Kim S, Tsourdos A (2015) Verification of heterogeneous multi-agent system using MCMAS. Int J Syst Sci 46(4):634–651.  https://doi.org/10.1080/00207721.2013.793890 MathSciNetMATHCrossRefGoogle Scholar
  61. 61.
    Cimatti A, Clarke E, Giunchiglia F, Roveri M (2000) NUSMV: a new symbolic model checker. Int J Softw Tools Technol Transfer 2(4):410–425. ISSN 14332779MATHCrossRefGoogle Scholar
  62. 62.
    Clarke EM, Grumberg O, Doron AP (1999) Model checking. The MIT Press, CambridgeGoogle Scholar
  63. 63.
    Coelho R, Kulesza U, von Staa A, Lucena C (2006) Unit testing in multi-agent systems using mock agents and aspects. In: Proceedings of the 2006 International Workshop on Software Engineering for Large-Scale Multi-Agent Systems - SELMAS ’06, SELMAS ’06. ACM, New York, pp 83–8. ISBN 1595933956.  https://doi.org/10.1145/1138063.1138079
  64. 64.
    Coelho R, Cirilo E, Kulesza U, Von Staa A, Rashid A, Lucena C (2007) JAT: a test automation framework for multi-agent systems. In: IEEE International Conference on Software Maintenance, ICSM. ISBN 1424412560.  https://doi.org/10.1109/ICSM.2007.4362655, pp 425–434
  65. 65.
    Collier R (2007) Debugging agents in agent factory. In: Bordini RH, Dastani M, Dix J, Seghrouchni A (eds) Programming multi-agent systems, volume 4411 of lecture notes in computer science. Springer, Berlin, pp 229–248. ISBN 978-3-540-71955-7.  https://doi.org/10.1007/978-3-540-71956-4_14
  66. 66.
    Costantini S, De Gasperis G (2014) Runtime Self-Checking via Temporal (Meta- ) Axioms for Assurance of Logical Agent Systems. pages 1–15Google Scholar
  67. 67.
    Da Silva P, De Melo ACV (2013) An approach for the verification of multi-agent systems by formally guided simulations. In: 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT). vol 2, pp 266–273.  https://doi.org/10.1109/WI-IAT.2013.119
  68. 68.
    Dastani M, John J-JCh, Meyer JCh (2010) Correctness of multi-agent programs: a hybrid approach. In: Dastani M, Hindriks KV, Meyer J-JC (eds) Specification and verification of multi-agent systems. ISBN 9781441969835. Springer, US, pp 161–194Google Scholar
  69. 69.
    Dastani M, Brandsema J, Dubel A, John-Jules Ch, Meyer Ch (2009) Debugging bdi-based multi-agent programs. In: Braubach L, Briot J-P, Thangarajah J (eds) International workshop on programming multi-agent systems, volume 5919 of lecture notes in computer science. Springer, Berlin, pp 151–169. ISBN 978-3-642-14842-2.  https://doi.org/10.1007/978-3-642-14843-9_10
  70. 70.
    Davoodi M, Meskin N, Khorasani K (2016) Simultaneous fault detection and consensus control design for a network of multi-agent systems. Automatica 66:185–194MathSciNetMATHCrossRefGoogle Scholar
  71. 71.
    Dekhtyar MI, Dikovsky AJ, Valiev M (2008) Temporal verification of probabilistic multi-agent systems. In: Avron A, Dershowitz N, Rabinovich A (eds) Pillars of computer science, volume 4800 of lecture notes in computer science. Springer, Berlin, pp 256–265. ISBN 978-3-540-78126-4.  https://doi.org/10.1007/978-3-540-78127-1_14
  72. 72.
    Delgado C, Benevides M (2009) Verification of epistemic properties in probabilistic multi-agent systems. In: Braubach L, van der Hoek W, Petta P, Pokahr A (eds) Multiagent system technologies, volume 5774 of lecture notes in computer science. Springer, Berlin, pp 16–28. ISBN 978-3-642-04142-6.  https://doi.org/10.1007/978-3-642-04143-3_3
  73. 73.
    Dennis LA, Farwer B, Bordini RHRH, Fisher M (2008) A flexible framework for verifying agent programs. In: Proceedings of the 7th International Conference on Autonomous Agent and Multiagent Systems, AAMAS ’08. ISBN 978-0-9817381-2-3. International Foundation for Autonomous Agents and Multiagent Systems, Richland, pp 1303–1306Google Scholar
  74. 74.
    Dennis LA, Fisher M, Lincoln NK, Lisitsa A, Veres SM (2014) Practical verification of decision-making in agent-based autonomous systems. Autom Softw Eng 23(3):305–359. ISSN 1573-7535.  https://doi.org/10.1007/s10515-014-0168-9 CrossRefGoogle Scholar
  75. 75.
    Earle CB, Fredlund LÅ, Moreno-Díaz R, Pichler F, Quesada-Arencibia A (2009) Debugging and verification of multi-agent systems. In: 12th International Conference Computer Aided Systems Theory - EUROCAST 2009, Las Palmas de Gran Canaria, Spain, February 15-20, 2009, Revised Selected Papers, volume 5717 of Lecture Notes in Computer Science. ISBN 978-3-642-04771-8.  https://doi.org/10.1007/978-3-642-04772-5, pp 263–270
  76. 76.
    Ekinci EE, Tiryaki AM, Çetin Ö, Dikenelli O, âĹžáetin â, Dikenelli O (2009) Goal-oriented agent testing revisited. In: Luck M, Gomez-Sanz J (eds) Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 5386 of Lecture Notes in Computer Science. Springer, Berlin, pp 46–59. ISBN 978-3-642-01337-9.  https://doi.org/10.1007/978-3-642-01338-6_13
  77. 77.
    El Fallah-Seghrouchni A, Gomez-Sanz JJ, Singh MP (2011) Formal methods in agent-oriented software engineering. Springer, Berlin, pp 213–228. ISBN 978-3-642-19208-1Google Scholar
  78. 78.
    El-Menshawy M, Bentahar J, Dssouli R (2010) Verifiable semantic model for agent interactions using social commitments. In: Dastani M, Segrouchni AEF, Leite J, Torroni P (eds) Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 6039 LNAI of Lecture Notes in Computer Science. Springer, Berlin, pp 128–152. ISBN 978-3-642-13337-4.  https://doi.org/10.1007/978-3-642-13338-1_8
  79. 79.
    Engelfriet J, Jonker CM, Treur J (1999) Compositional verification of multi-agent systems in temporal multi-epistemic logic. In: Intelligent Agents V: Agents Theories, Architectures, and Languages. ISBN 978-3-540-65713-2, vol 1555. Springer, pp 177–193Google Scholar
  80. 80.
    Esterline A, Gandluri B, Sundaresan M, Sankar J (2005) Verified models of multiagent systems for vehicle health management. In: Proceedings of SPIE, vol 5757, pp 602–613.  https://doi.org/10.1117/12.600017
  81. 81.
    Fern L, Rodrigues O, De Carvalho GR, De Barros R, José C, Lucena P, Rodrigues L, De Carvalho GR, Paes R, Lucena C (2005) Towards an integration test architecture for open MAS. In: Proceedings of the 1st Workshop on Software Engineering for Agent-Oriented Systems/SBESGoogle Scholar
  82. 82.
    Fisher M, Wooldridge M (1997) On the formal specification and verification of multi-agent systems. Int J Coop Infor Syst 06(01):37–65. 10.1142/S0218843097000057CrossRefGoogle Scholar
  83. 83.
    Flater D (2001) Debugging agent interactions: a case study. In: Proceedings of the 2001 ACM symposium on applied computing. ACM, ACM Press, pp 107–114Google Scholar
  84. 84.
    Fortino G, Garro A, Russo W, Caico R, Cossentino M, Termine F (2006) Simulation-driven development of multi-agent systems. In: Gentile A, Genco A, Sorce S (eds) Industrial simulation conference 2006 (ISC 2006), pages 17–24. EUROSIS (The European Simulation Society) & ETI (The European Technology Institute)Google Scholar
  85. 85.
    Franceschelli M, Martini S, Egerstedt M, Bicchi A, Giua A (2010) Observability and controllability verification in multi-agent systems through decentralized Laplacian spectrum estimation. In: 2010 49th IEEE Conference on Decision and Control (CDC), pp 5775–5780. ISBN 9781424477456.  https://doi.org/10.1109/CDC.2010.5717400
  86. 86.
    Raimondi F, Lomuscio A (2004) Automatic verification of deontic properties of multi-agent systems. In: Proceedings of the 7th International Workshop on Deontic Logic in Computer Science (DEON 2004), Madeira, Portugal, May 26-28, 2004, number 3065/2004. ISBN 3540221115 — 9783540221111. Springer, pp 228–242Google Scholar
  87. 87.
    Gammie P, van der Meyden R (2004) MCK: model checking the logic of knowledge. Springer, Berlin, pp 479–483. ISBN 978-3-540-27813-9MATHGoogle Scholar
  88. 88.
    Garanina NO, Bodin EV, Sidorova EA (2014) Using SPIN for verification of multi-agent data analysis. Model Anal Inform Sist 21(6):31–43. ISSN 1558-108X.  https://doi.org/10.3103/S014641161507007X CrossRefGoogle Scholar
  89. 89.
    Athanázio M, Gatti DC, Von Staa A (2006) Testing & Debugging Multi-Agent Systems: A State of the Art Report. p 28Google Scholar
  90. 90.
    German E, Sheremetov L (2008) An agent framework for processing fipa-acl messages based on interaction models. In: Luck M, Padgham L (eds) Agent-oriented software engineering VIII, volume 4951 of Lecture Notes in Computer Science. Springer, Berlin, pp 88–102. ISBN 978-3-540-79487-5.  https://doi.org/10.1007/978-3-540-79488-2_7
  91. 91.
    Giese H, Klein F (2007) Systematic verification of multi-agent systems based on rigorous executable specifications. Int J Agent-Oriented Softw Eng 1(1):28–62. ISSN 1746-1375.  https://doi.org/10.1504/IJAOSE.2007.013264 CrossRefGoogle Scholar
  92. 92.
    Giordano L, Martelli A, Schwind C (2003) Specifying and verifying systems of communicating agents in a temporal action logic. In: Proceedings of the 8th conference of AI*IA, volume 2829 of LNAI. Springer, pp 262–274Google Scholar
  93. 93.
    Giordano L, Martelli A, Schwind Cx (2004) Verifying communicating agents by model checking in a temporal action logic. In: Proceedings Logics in artificial intelligence, 9th european conference, JELIA 2004. Springer, pp 57–69Google Scholar
  94. 94.
    Giordano L, Martelli A, Schwind C (2007) Specifying and verifying interaction protocols in a temporal action logic. J Appl Logic 5(2):214–234. ISSN 1570-8683.  https://doi.org/10.1016/j.jal.2005.12.011 MathSciNetMATHCrossRefGoogle Scholar
  95. 95.
    Perelli G (2015) Logics for multi-agent systems verification. PhD thesis, Universit‘ a degli studi di Napoli ”Federico II”Google Scholar
  96. 96.
    Góxmez-Sanz J, Gervais M-P, Weiss G (2004) A survey on agent-oriented oriented software engineering research. In: Methodologies and software engineering for agent systems. ISBN 978-1-4020-8058-6, vol 11, pp 33–62Google Scholar
  97. 97.
    Gomez-Sanz JJ, Botia J, Serrano E, Pavón J (2008) Testing and debugging of MAS interactions with INGENIAS. In: International Workshop on Agent-Oriented Software Engineering. Springer, pp 199–212Google Scholar
  98. 98.
    Guerin F, Pitt J (2002) Agent communication frameworks and verification. In: AAMAS 2002 Workshop on Agent Communication Languages, pp 200–202Google Scholar
  99. 99.
    Gurcan O, Dikenelli O, Bernon C (2011) Towards a generic testing framework for agent-based simulation models. In: 2011 Federated Conference on Computer Science and Information Systems (fedCSIS), pp 635–642Google Scholar
  100. 100.
    Gutiérrez C, García-Magariño I, Serrano E, Botía JA (2013) Robust design of multi-agent system interactions: a testing approach based on pattern matching. Eng Appl Artif Intell 26(9):2093–2104. ISSN 09521976.  https://doi.org/10.1016/j.engappai.2013.06.006 CrossRefGoogle Scholar
  101. 101.
    Hamrouni N, Abderrahim Z (2012) Looking for verification and validation of a multi-agent system using new formalism: APN. In: 2012 6th International Conference on Sciences of Electronics, Technologies of Information and Telecommunications (SETIT).  https://doi.org/10.1109/SETIT.2012.6481885, pp 30–35
  102. 102.
    Herd B, Miles S, McBurney P, Luck M (2014) Verification and validation of agent-based simulations using approximate model checking. Multi-Agent-Based Simul XIV(8235):53–70CrossRefGoogle Scholar
  103. 103.
    Hnativ A, Ludwig SA (2009) Evaluation of trust in an ecommerce multi-agent system using fuzzy reasoning. In: IEEE International Conference on Fuzzy Systems, 2009. FUZZ-IEEE.  https://doi.org/10.1109/FUZZY.2009.5277344, pp 757–763
  104. 104.
    Holzmann GJ (1997) The model checker SPIN. IEEE Trans Softw Eng 23(5):279–295. ISSN 00985589CrossRefGoogle Scholar
  105. 105.
    Houhamdi Z (2011) Multi-agent system testing: a survey. Int J Advan Comput Sci Appl 2(6):135–141. ISSN 2158107X.  https://doi.org/10.14569/IJACSA.2011.020620 Google Scholar
  106. 106.
    Bing H, Ji G, Guo H (2009) Dynamic model of normative multi-agent system and its property verification mechanism. J Zhejiang University(engineering science) 43(6):1014–1019Google Scholar
  107. 107.
    Huang H-J, Wang X, Chen Q-C, Wang X-L (2005) Specification and verification of multi-agent systems with a property-preserving component-based methodology. In: Proceedings of 2005 International Conference on Machine Learning and Cybernetics, 2005, vol 1, pp 90–95 https://doi.org/10.1109/ICMLC.2005.1526925
  108. 108.
    Huget MP, Demazeau Y (2004) Evaluating multiagent systems: a record/replay approach. In: Proceedings - IEEE/WIC/ACM International Conference on Intelligent Agent Technology. IAT 2004, pp 536–539. ISBN 0769521010.  https://doi.org/10.1109/IAT.2004.1343013
  109. 109.
    Jamroga W, Penczek W (2012) Specification and verification of multi-agent systems. In: Bezhanishvili N, Goranko V (eds) Lectures on Logic and Computation, volume 7388 of Lecture Notes in Computer Science. Springer, Berlin, pp 210–263. ISBN 978-3-642-31484-1.  https://doi.org/10.1007/978-3-642-31485-8_6
  110. 110.
    Janssen MA (2003) Complexity and ecosystem management: the theory and practice of multi-agent approaches. Complex Ecosyst Manag 8(2):63–74Google Scholar
  111. 111.
    Jiang Mx, Ding Z, Zhou M, Zhou Y (2015) Formal modeling and verification of secure mobile agent systems. In: 2015 IEEE international Conference on Automation Science and Engineering (CASE), pp 545–550Google Scholar
  112. 112.
    Johansson C, Bucanac C (1998) The v model, CrosstalkGoogle Scholar
  113. 113.
    Jones AV, Lomuscio A (2010) Distributed BDD-based BMC for the verification of multi-agent systems. In: van der Hoek W, Kaminka GA, Lespérance Y, Luck M, Sen S (eds) Transition, number Aamas. ISBN 9781617387715. IFAAMAS, pp 675–682Google Scholar
  114. 114.
    Jonker CM, Treur J (2002) Compositional verification of multi-agent systems: a formal analysis of pro-activeness and reactiveness. Int J Coop Inf Syst 11:51–91. ISBN 3540654933.  https://doi.org/10.1142/S0218843002000480 CrossRefGoogle Scholar
  115. 115.
    Kacprzak M, Lomuscio A, Penczek W (2004) Verification of multiagent systems via unbounded model checking. In: Autonomous Agents and Multiagent Systems. ISBN 1581138644.  https://doi.org/10.1109/AAMAS.2004.296, pp 638–645
  116. 116.
    Kafali Ö, Torroni P (2012) Exception diagnosis in multiagent contract executions. Ann Math Artif Intell 64(1):73–107. ISSN 10122443.  https://doi.org/10.1007/s10472-012-9282-1 MathSciNetMATHCrossRefGoogle Scholar
  117. 117.
    Kahloul L, Grira M (2014) Formal specification and verification of mobile agent systems. Int J Comput Commun 9(3):292–304. ISSN 1841-9836CrossRefGoogle Scholar
  118. 118.
    Khamis MA, Nagi K (2013) Designing multi-agent unit tests using systematic test design patterns-(extended version). Eng Appl Artif Intell 26(9):2128–2142. ISSN 09521976.  https://doi.org/10.1016/j.engappai.2013.04.009 CrossRefGoogle Scholar
  119. 119.
    Khattak AS, Sikander M, Khiyal H, Rizvi SS, Khiyal MSH, Rizvi SS, Sikander M, Khiyal H, Rizvi SS (2014) Verification & validation of a multi-agent meeting scheduling simulation model. Electron J Comput Sci Inf Technol 2(1):47–64Google Scholar
  120. 120.
    Khattak AS, Khiyal MSH, Rizvi SS (2015) Verification and validation of agent-based model using E-VOMAS approach. International Journal Of Computer Science and Network Security (IJCSNS) 15(3):29–35Google Scholar
  121. 121.
    Kissoum Y, Sahnoun Z (2007) A formal approach for functional and structural test case generation in multi-agent systems. In: 2007 IEEE/ACS International conference on computer systems and applications.  https://doi.org/10.1109/AICCSA.2007.370867, pp 76–83
  122. 122.
    Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Engineering 2:1051. ISSN 00010782Google Scholar
  123. 123.
    Klimek R, Faber Ł, Kisiel-Dorohinicki M (2014) Deduction-based modelling and verification of agent-based systems for data integration. Man-Machine Interact 3(242):361–368CrossRefGoogle Scholar
  124. 124.
    Koleini M, Ryan M (2011) A knowledge-based verification method for dynamic access control policies. In: Qin S, Qiu Z (eds) Formal Methods and Software Engineering, volume 6991 of Lecture Notes in Computer Science. Springer, Berlin, pp 243–258. ISBN 978-3-642-24558-9.  https://doi.org/10.1007/978-3-642-24559-6_18.
  125. 125.
    Koleini M, Ritter E, Ryan M (2013) Model checking agent knowledge in dynamic access control policies. In: Piterman N, Smolka SA (eds) Tools and Algorithms for the Construction and Analysis of Systems, volume 7795 of Lecture Notes in Computer Science. Springer, Berlin, pp 448–462. ISBN 978-3-642-36741-0.  https://doi.org/10.1007/978-3-642-36742-7_31.
  126. 126.
    Koleini M, Ritter E, Ryan M (2014) Verification of agent knowledge in dynamic access control policies. CoRR, abs/1401.4: 448–462.  https://doi.org/10.1007/978-3-642-36742-7_31
  127. 127.
    Konur S, Fisher M, Schewe S (2009) Verification of multi-agent systems via combined model checkingGoogle Scholar
  128. 128.
    Kouvaros P (2013) Automatic verification of parameterised interleaved multi-agent systems. In: Proceedings of the 2013 international conference on autonomous agents and multi-agent systems, (Aamas), pp 861–868Google Scholar
  129. 129.
    Kouvaros P, Lomuscio A (2016) Parameterised verification for multi-agent systems. Artif Intell 234:152–189. ISSN 00043702.  https://doi.org/10.1016/j.artint.2016.01.008 MathSciNetMATHCrossRefGoogle Scholar
  130. 130.
    Kouvaros P, Lomuscio A (2017) Parameterised verification of infinite state multi-agent systems via predicate abstraction. Proceedings of the 31th Conference on Artificial Intelligence (AAAI) 2017:3013–3020Google Scholar
  131. 131.
    Kwiatkowska M, Norman G, Parker D (2011) PRISM 4.0: Verification of probabilistic real-time systems. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 6806 LNCS. ISBN 9783642221095, pp 585–591Google Scholar
  132. 132.
    Lacey T, Deloach SA (2000) Automatic verification of multiagent conversations. In: Proceedings of the Eleventh Annual Midwest Artificial Intelligence and Cognitive Science Conference. AAAI Press, pp 93–100Google Scholar
  133. 133.
    Lacey TH, Deloach SA (2000) Verification of agent behavioral models. In: Proceedings of the international conference on artificial intelligence (IC-AI’2000). CSREA Press, pp 557–564Google Scholar
  134. 134.
    Lam DN, Barber KS (2005) Debugging agent behavior in an implemented agent system. In: Bordini RH, Dastani M, Dix J, Seghrouchni AEF (eds) Programming multi-agent systems, volume 3346 of lecture notes in computer science. Springer, Berlin, pp 104–125. ISBN 978-3-540-24559-9.  https://doi.org/10.1007/978-3-540-32260-3_6.
  135. 135.
    Latif N, Hassan M, Hasan M (2011) Formal verification for interaction protocol in agent-based e-learning system using model checking toolkit - MCMAS. In: Zain JM, Wan Mohd WM, El-Qawasmeh E (eds) Software engineering and computer systems, volume 180 of communications in computer and information science. Springer, Berlin, pp 412–426. ISBN 978-3-642-22190-3.  https://doi.org/10.1007/978-3-642-22191-0_36.
  136. 136.
    Laureano-Cruces AL, Barceló-Aspeitia AA (2003) Formal verification of multi-agent systems behaviour emerging from cognitive task analysis. J Exp Theor Artif Intell 15(4):407–431. ISSN 0952-813X.  https://doi.org/10.1080/0952813031000119719 MATHCrossRefGoogle Scholar
  137. 137.
    Lee P, Saleh O, Alomair B, Bushnell L, Poovendran R (2014) Graph-based verification and misbehavior detection in multi-agent networks. In: Proceedings of the 3rd International Conference on High Confidence Networked Systems, HiCoNS’14. ACM, New York, pp 77–84. ISBN 978-1-4503-2652-0.  https://doi.org/10.1145/2566468.2566477
  138. 138.
    Li Y, Fang H, Chen J, Shang C (2016) Distributed fault detection and isolation for multi-agent systems using relative information. In: American control conference (ACC), 2016. American Automatic Control Council (AACC), pp 5939– 5944Google Scholar
  139. 139.
    Van Liedekerke MH, Avouris NM (1995) Debugging multi-agent systems. Inf Softw Technol 37:102–112Google Scholar
  140. 140.
    Lim YJ, Hong G, Shin D, Jee E, Bae D-H (2016) A runtime verification framework for dynamically adaptive multi-agent systems. In: 2016 International conference on big data and smart computing (bigcomp). IEEE, pp 509–512Google Scholar
  141. 141.
    Liu X, Gao X, Han J (2015) Fault detection for high-order multi-agent systems with disturbances. In: The 27th Chinese Control and Decision Conference (2015 CCDC), pp 3814–3819.  https://doi.org/10.1109/CCDC.2015.7162590
  142. 142.
    Lomuscio A, Michaliszyn J (2014) An abstraction technique for the verification of multi-agent systems against ATL specifications. In: Association for the advancement of artificial intelligence. ISBN 9781577356578Google Scholar
  143. 143.
    Lomuscio A, Michliszyn J (2016) Verification of multi-agent systems via predicate abstraction against ATLK specifications. In: Proceedings of the 2016 international conference on autonomous agents & multiagent systems. International Foundation for Autonomous Agents and Multiagent Systemsx, pp 662–670Google Scholar
  144. 144.
    Lomuscio A, Paquet H (2015) Verification of multi-agent systems via sdd-based model checking. In: Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’15, pages 1713–1714, Richland, SC. ISBN 978-1-4503-3413-6. International Foundation for Autonomous Agents and Multiagent SystemsGoogle Scholar
  145. 145.
    Lomuscio A, Penczek W (2007) Model checking security protocols: a multiagent systems approach. In: Proceedings of the 10th International Workshop on con-currency, specification and programming. Warsaw University Press, Poland, pp 400–412Google Scholar
  146. 146.
    Lomuscio A, Solanki M (2009) Mapping OWL-S processes to multi agent systems: a verification oriented approach. In: International Conference on Advanced Information Networking and Applications Workshops, 2009. WAINA ’09, pp 488–493. ISBN 9780769536392.  https://doi.org/10.1109/WAINA.2009.52
  147. 147.
    Lomuscio A, Qu H, Raimondi F (2009) MCMAS: a model checker for the verification of multi-agent systems. In: Bouajjani A, Maler O (eds) Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 5643 of Lecture Notes in Computer Science. ISBN 978-3-642-02657-7. Springer, Berlin, pp 682–688Google Scholar
  148. 148.
    Lomuscio A, Hongyang Q, Raimondi F (2015) MCMAS: an open-source model checker for the verification of multi-agent systems. Int J Softw Tools Technol Transfer:1–22Google Scholar
  149. 149.
    Low CKK, Chen TY, RâĹžâL’ěnnquist R, Rónnquist R (1999) Automated test case generation for BDI agents. Auton Agent Multi-Agent Syst 2(4):311–332. ISSN 13872532.  https://doi.org/10.1023/A:1010011219782 CrossRefGoogle Scholar
  150. 150.
    Passos LÍS, Rossetti RJF, Gabriel J (2013) Diagnosis of unwanted behaviours in multi-agent systems. In: 11Th european workshop on multi-agent systems (EUMAS 2013). France, ToulouseGoogle Scholar
  151. 151.
    Luck M, McBurney P, Preist C (2003) Agent technology: enabling next generation computing (A Roadmap for Agent Based Computing)Google Scholar
  152. 152.
    Luo X, Su K, Sattar A, Reynolds M (2006) Verification of multi-agent systems via bounded model checking. In: Sattar A, Kang B-h (eds) AI 2006: advances in artificial intelligence, volume 4304 of lecture notes in computer science. Springer, Berlin, pp 69–78. ISBN 978-3-540-49787-5.  https://doi.org/10.1007/11941439_11.
  153. 153.
    Luo Xiangyu, Zou Mengmeng, Luo Lingjie (2012) A modeling and verification method to multi-agent systems based on KQML Electrical electronics engineering (EEESYM), 2012 IEEE symposium on, pages 690–693, jun,  https://doi.org/10.1109/EEESym.2012.6258753
  154. 154.
    Mani N (2008) Testing and monitoring multi-agent systems for deadlock detection. PhD thesis. http://www.softqual.ucalgary.ca/pubs/theses/2009_Nariman_Mani_Thesis.pdf
  155. 155.
    Mani N, Garousi V, Far BH (2008) Testing multi-agent systems for deadlock detection based on UML models. In: Canadian Conference on Electrical and Computer Engineering. Boston, USA, pp 77–84. ISBN 9781424416431.  https://doi.org/10.1109/CCECE.2008.4564814
  156. 156.
    Mani N, Garousi V, Far BH (2010) Search-based testing of multi-agent manufacturing systems for deadlocks based on models. Int J Artif Intell Tools 19(04):417–437.  https://doi.org/10.1142/S0218213010000261 CrossRefGoogle Scholar
  157. 157.
    Marzougui B, Hassine K, Barkaoui K (2011) Method for verification of a multi agents system. In: 2011 Second International Conference on Intelligent Systems, Modelling and Simulation (ISMS), pp 62–65.  https://doi.org/10.1109/ISMS.2011.21
  158. 158.
    Mehmood A, Jawawi DNA (2013) Aspect-oriented model-driven code generation: a systematic mapping study. In: Information and software technology. ISBN 0950-5849, vol 55, pp 395–411Google Scholar
  159. 159.
    Meron D, Mermet B (2007) A tool architecture to verify properties of multiagent system at runtime. In: Bordini RH, Dastani M, Dix J, Seghrouchni AEF (eds) Programming multi-agent systems, volume 4411 of lecture notes in computer science. Springer, Berlin, pp 201–216. ISBN 978-3-540-71955-7.  https://doi.org/10.1007/978-3-540-71956-4_12
  160. 160.
    Mȩski A, Penczek W, Szreter M, Woźna-Szcześniak B, Zbrzezny A (2014) BDD-versus SAT-based bounded model checking for the existential fragment of linear temporal logic with knowledge: algorithms and their performance. Auton Agent Multi-Agent Syst 28(4):558–604. ISSN 13872532.  https://doi.org/10.1007/s10458-013-9232-2 CrossRefGoogle Scholar
  161. 161.
    Micalizio R, Torasso P, Torta G (2003) The architecture of the diagnostic agent in the ROBOCARE project: the long report. Technical report Technical Report RC-TR-1203Google Scholar
  162. 162.
    Micalizio R, Torasso P, Torta G (2004) On-line monitoring and diagnosis of multi-agent systems: a model based approach. In: ECAI, pp 848–852Google Scholar
  163. 163.
    Micalizio R, Torasso P, Torta G (2006) On-line monitoring and diagnosis of a team of service robots: a model-based approach. AI Commun 19(4):313–340. ISSN 0921-7126MathSciNetMATHGoogle Scholar
  164. 164.
    Micalizio R, Torasso P, Torta G (2006) Synthesizing diagnostic explanations from monitoring data in multi-robot systems. In: Proceedings of the IASTED International Conference on Artificial Intelligence and Applications, AIA 2006, AIA’06. ISBN 0-88986-556-6. ACTA Press, Anaheim, pp 279–286Google Scholar
  165. 165.
    Miller T, Padgham L, Thangarajah J (2011) Test coverage criteria for agent interaction testing. In: Weyns D, Gleizes M-P (eds) Agent-oriented software engineering XI, volume 6788 of lecture notes in computer science. ISBN 978-3-642-22635-9. Springer, Berlin, pp 91–105Google Scholar
  166. 166.
    Mireslami S, Far BH (2013) Automated verification of AUML based multi-agent system design. In: Canadian Conference on Electrical and Computer Engineering. pp 1–4. ISBN 9781479900329.  https://doi.org/10.1109/CCECE.2013.6567818
  167. 167.
    Mohammed A, Furbach U (2009) Multi-agent systems: modeling and verification using hybrid automata. In: Braubach L, Briot J-P, Thangarajah J (eds) International workshop on programming multi-agent systems, volume 5919 of lecture notes in computer science. Springer, Berlin, pp 49–66. ISBN 978-3-642-14842-2.  https://doi.org/10.1007/978-3-642-14843-9_4
  168. 168.
    Moreno M, Pavón J, Rosete A (2009) Testing in agent oriented methodologies. Lecture notes in computer science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5518 LNCS (PART 2):138–145. ISSN 03029743Google Scholar
  169. 169.
    Moscato F, Venticinque S, Aversa R, Di Martino B (2008) Formal modeling and verification of real-time multi-agent systems: the REMM framework. In: Badica C, Mangioni G, Carchiolo V, Burdescu D (eds) Intelligent distributed computing, systems and applications, volume 162 of studies in computational intelligence. Springer, Berlin, pp 187–196. ISBN 978-3-540-85256-8.  https://doi.org/10.1007/978-3-540-85257-5_19
  170. 170.
    Moshirpour M, Mousavi A, Far BH (2010) Model based detection of implied scenarios in multi agent systems. In: 2010 IEEE International Conference on Information Reuse and Integration (IRI), pp 63–68.  https://doi.org/10.1109/IRI.2010.5558962
  171. 171.
    Moshirpour M, Mousavi A, Far BH (2012) Detecting emergent behavior in distributed systems using scenario-based specifications. Int J Softw Eng Knowl Eng 22(6):729–746.  https://doi.org/10.1142/S0218194012400104 CrossRefGoogle Scholar
  172. 172.
    Moshirpour M, Mani N, Eberlein A, Far BH (2013) Model based approach to detect emergent behavior in multi-agent systems. In: 12th International Conference on Autonomous Agents and Multiagent Systems 2013, AAMAS 2013, volume 2 of AAMAS ’13. ISBN 978-1-4503-1993-5. International Foundation for Autonomous Agents and Multiagent Systems, Richland, pp 1285–1286Google Scholar
  173. 173.
    Murano A, Perelli G, Federico N (2015) Pushdown multi-agent system verification. Number Ijcai, 1090–1096Google Scholar
  174. 174.
    Vivekanandan K, Sivakumar N (2012) Agent oriented software testing – role oriented approach. Int J Adv Comput Sci Appl (IJACSA) 3(12):156–163Google Scholar
  175. 175.
    Ndumu DT, Nwana HS, Lee LC, Collis JC (1999) Visualising and debugging distributed multi-agent systems. In: Proceedings of the Third Annual Conference on Autonomous Agents, AGENTS ’99. ACM, New York, pp 326–333. ISBN 1-58113-066-X.  https://doi.org/10.1145/301136.301220
  176. 176.
    Cu D, Perini A, Tonella P, Kessler FB (2007) Nguyen automated continuous testing of multi-agent systems. In: Fifth European Workshop on Multi-Agent Systems (EUMAS)Google Scholar
  177. 177.
    Nguyen DC, Perini A, Tonella P (2008) A goal-oriented software testing methodology. In: Luck M, Padgham L (eds) Agent-oriented software engineering VIII, volume 4951 of lecture notes in computer science. Springer, Berlin, pp 58–72. ISBN 978-3-540-79487-5.  https://doi.org/10.1007/978-3-540-79488-2_5
  178. 178.
    Nguyen CDD, Perini A, Tonella P (2009) Experimental evaluation of ontology-based test generation for multi-agent systems. In: Luck M, Gomez-Sanz JJ (eds) Lecture notes in computer science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 5386 of lecture notes in computer science. Springer, Berlin, pp 187–198. ISBN 9783642013379.  https://doi.org/10.1007/978-3-642-01338-6_14
  179. 179.
    Nguyen CD, Perini A, Tonella P (2010) Goal-oriented testing for MASs. Int J Agent-Oriented Softw Eng 4(1):79. ISSN 1746-1375.  https://doi.org/10.1504/IJAOSE.2010.029810 CrossRefGoogle Scholar
  180. 180.
    Nguyen CD, Perini A, Tonella P (2010) Goal-oriented Testing for MASs. Int J Agent-Oriented Softw Eng 4(1):79–109. ISSN 1746-1375.  https://doi.org/10.1504/IJAOSE.2010.029810 CrossRefGoogle Scholar
  181. 181.
    Nguyen CD, Perini A, Bernon C, Pavón J, Thangarajah J (2011) Testing in multi-agent systems. In: Lecture notes in computer science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 6038 LNCS. ISBN 9783642192074, pp 180–190Google Scholar
  182. 182.
    Nugraheni CE (2011) Formal verification of parameterized multi-agent systems using predicate diagrams*. In: Computation tools 2011: the second international conference on computational logics, algebras, programming, tools, and benchmarking, RomeGoogle Scholar
  183. 183.
    Núñez M, Rodríguez I, Rubio F (2005) Specification and testing of autonomous agents in e-commerce systems. Softw Test Verification Reliab 15(4):211–233. ISSN 1099-1689.  https://doi.org/10.1002/stvr.323 CrossRefGoogle Scholar
  184. 184.
    Osman N, Robertson D, Walton C (2006) Run-time model checking of interaction and deontic models for multi-agent systems. In: Proceedings of the Fifth International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS ’06. ACM, New York, pp 238–240. ISBN 1-59593-303-4.  https://doi.org/10.1145/1160633.1160674
  185. 185.
    Osman NZ (2008) Runtime verification of deontic and trust models in multiagent interactions. PhD thesis, University of EdinburghGoogle Scholar
  186. 186.
    Ou-Yang C, Juan Y-C (2010) Applying process mining approach to support the verification of a multi-agent system. J Syst Sci Syst Eng 19(2):131–149. ISSN 1004-3756.  https://doi.org/10.1007/s11518-010-5132-z CrossRefGoogle Scholar
  187. 187.
    Padgham L, Winikoff M, Poutakidis D (2005) Adding debugging support to the Prometheus methodology. Eng Appl Artif Intell 18(2):173–190. ISSN 09521976.  https://doi.org/10.1016/j.engappai.2004.11.018 CrossRefGoogle Scholar
  188. 188.
    Padgham L, Zhang Z, Thangarajah J, Miller T (2013) Model-based test oracle generation for automated unit testing of agent systems. IEEE Trans Softw Eng 39(9):1230–1244. ISSN 00985589CrossRefGoogle Scholar
  189. 189.
    Pallottino L, Scordio VG, Frazzoli E, Bicchi A (2006) Probabilistic verification of a decentralized policy for conflict resolution in multi-agent systems. In: Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA. pp 2448–2453. ISBN 0780395069.  https://doi.org/10.1109/ROBOT.2006.1642069
  190. 190.
    Passos LS, Abreu R, Rossetti RJF (2015) Spectrum-based fault localisation for multi-agent systems. IJCAI International Joint Conference on Artificial Intelligence, 2015-Janua (Ijcai): 1134–1140. ISSN 10450823Google Scholar
  191. 191.
    Paurobally S, Cunningham J, Jennings NR (2004) Verifying the contract net protocol: a case study in interaction protocol and agent communication semantics. In: 2Nd International Workshop on Logic and Communication in Multi-Agent Systems, pp 98–117Google Scholar
  192. 192.
    Penczek W, Lomuscio A (2003) Verifying epistemic properties of multi-agent systems via bounded model checking. In: Proceedings of the second international joint conference on Autonomous agents and multiagent systems - AAMAS ’03. p 209. ISSN 01692968.  https://doi.org/10.1145/860575.860609
  193. 193.
    Peng W, Krueger W, Grushin A, Carlos P, Manikonda V, Santos M (2009) Graph-based methods for the analysis of large-scale multiagent systems. In: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 1, AAMAS ’09. ISBN 978-0-9817381-6-1. International Foundation for Autonomous Agents and Multiagent Systems, Richland, pp 545–552Google Scholar
  194. 194.
    Pilecki J, Bednarczyk MA, Jamroga W (2014) Synthesis and verification of uniform strategies for multi-agent systems. Comput Logic Multi-Agent Syst 8624:166–182MATHGoogle Scholar
  195. 195.
    Potiron K, Taillibert P, Seghrouchni AEF (2008) A step towards fault tolerance for multi-agent systems. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5118 LNAI: 156–172. ISSN 03029743Google Scholar
  196. 196.
    Poutakidis D, Padgham L, Winikoff M (2002) Debugging multi-agent systems using design artifacts: the case of interaction protocols. In: Proceedings of the first international joint conference on autonomous agents and multi-agent systems, pp 960–967Google Scholar
  197. 197.
    Poutakidis D, Padgham L, Winikoff M (2003) An exploration of bugs and debugging in multi-agent systems. In: International symposium on methodologies for intelligent systems. Springer, pp 628–632Google Scholar
  198. 198.
    Poutakidis D, Winikoff M, Padgham L, Zhang Z (2009) Debugging and testing of multi-agent systems using design artefacts. Springer, US. ISBN 9780387892986MATHCrossRefGoogle Scholar
  199. 199.
    Qasim A, Asad S, Kazmi R, Fakhir I (2015) Formal specification and verification of real-time multi-agent systems using timed-arc petri nets. Advan Electr Comput Eng 15(3):73–78CrossRefGoogle Scholar
  200. 200.
    Raimondi F, Pecheur C, Lomuscio A (2005) Applications of model checking for multi-agent systems: verification of diagnosability and recoverability. In: Proceedings of concurrency, specification & programming (CS&p), Warsaw University, pp 433–444Google Scholar
  201. 201.
    Röhl M, Uhrmacher A (2004) Controlled experimentation with agents - models and implementations. In: Gleizes MP, Omicini A, Zambonelli F (eds) ESAW, volume 3451 of Lecture Notes in Computer Science. ISBN 3-540-27330-1. Springer, pp 292–304Google Scholar
  202. 202.
    Rouff C (2002) A test agent for testing agents and their communities. In: IEEE Aerospace Conference Proceedings. vol 5, pp 2633–2638. ISBN 078037231X.  https://doi.org/10.1109/AERO.2002.1035446
  203. 203.
    Roungroongsom C, Pradubsuwun D (2015) Formal verification of multi-agent system based on JADE A semi-runtime approach. Recent Advan Inf Commun Technol 361:297–306Google Scholar
  204. 204.
    Sabri KE, Khedri R, Jaskolka J (2009) Verification of information flow in agent-based systems. In: Babin G, Kropf P, Weiss M (eds) Lecture notes in business information processing, volume 26 of lecture notes in business information processing. Springer, Berlin, pp 252–266. ISBN 9783642011863.  https://doi.org/10.1007/978-3-642-01187-0_22
  205. 205.
    Salamon T (2009) A three-layer approach to testing of multi-agent systems. In: Papadopoulos GA, Wojtkowski W, Wojtkowski G, Wrycza S, Zupancic J (eds) Information systems development. Springer, pp 393–401. ISBN 978-0-387-84809-9.  https://doi.org/10.1007/b137171_41
  206. 206.
    Selamat A, Dianah S, Bujang A (2008) The design of model checking agent for SMS management system. In: Nguyen NT, Jo GS, Howlett RJ, Jain LC (eds) Agent and multi-agent systems: technologies and applications, volume 4953 of lecture notes in computer science. ISBN 978-3-540-78581-1. Springer, Berlin, pp 813–821Google Scholar
  207. 207.
    Selamat A, Lockman MTT (2009) Multi-agent verification of RFID system. In: Nguyen NT, Katarzyniak RP, Janiak A (eds) New challenges in computational collective intelligence, volume 244 of studies in computational intelligence. ISBN 9783642039577. Springer, Berlin, pp 255–268Google Scholar
  208. 208.
    Serrano E, Muñoz A, Botia J (2012) An approach to debug interactions in multi-agent system software tests. Inf Sci 205:38–57. ISSN 0020-0255.  https://doi.org/10.1016/j.ins.2012.04.001 CrossRefGoogle Scholar
  209. 209.
    Serrano E, Rovatsos M, Botía JA, Botía JA (2013) Data mining agent conversations: a qualitative approach to multiagent systems analysis. Inf Sci 230(0):132–146. ISSN 0020-0255.  https://doi.org/10.1016/j.ins.2012.12.019 MathSciNetCrossRefGoogle Scholar
  210. 210.
    Shetty S, Kiran HSS, Namala MB, Singh S (2011) Logical modeling and verification of a strength based multi-agent argumentation scheme using NuSMV. In: Meghanathan N, Kaushik BK, Nagamalai D (eds) Advances in computer science and information technology, volume 131 of communications in computer and information science. Springer, Berlin, pp 378–387. ISBN 978-3-642-17856-6.  https://doi.org/10.1007/978-3-642-17857-3_38
  211. 211.
    Song S, Hao J, Liu Y, Sun J, Leung HF, Dong JS (2012) Analyzing multi-agent systems with probabilistic model checking approach. In: 2012 34th International Conference on Software Engineering (ICSE), pp 1337–1340. ISBN 9781467310673.  https://doi.org/10.1109/ICSE.2012.6227085
  212. 212.
    Srivastava PR, Karthik Anand V, Subba Reddy V, Raghurama G (2008) Regression testing techniques for agent oriented software. In: International Conference on Information Technology, 2008. ICIT ’08, pp 221–225. ISBN 978-0-7695-3513-5.  https://doi.org/10.1109/ICIT.2008.30
  213. 213.
    Stolzenburg F, Arai T (2003) From the specification of multiagent systems by statecharts to their formal analysis by model checking: towards safety-critical applications. In: Schillo M, Klusch M, Muller J, Tianfield H (eds) Multiagent System Technologies, volume 2831 of Lecture Notes in Computer Science. ISBN 978-3-540-20124-3. Springer, Berlin, pp 131–143Google Scholar
  214. 214.
    Sudeikat J, Renz W (2009) A systemic approach to the validation of SelfâĂžÄìOrganizing Dynamics within MAS. In: Luck M, Gomez-Sanz JJ (eds) Agent-oriented software engineering IX, volume 5386 of lecture notes in computer science. Springer, Berlin, pp 31–45. ISBN 978-3-642-01337-9.  https://doi.org/10.1007/978-3-642-01338-6_3
  215. 215.
    Sycara KP (1998) Multiagent systems. AI Mag 19(2):79. ISSN 0738-4602Google Scholar
  216. 216.
    Tao Z, Hong X, Shao-Bin H (2015) A formal verification method of obligation policy in multi-agent system international journal of u- and e- service. Sci Technol 8(11):113–124Google Scholar
  217. 217.
    Tekbacak F, Tuglular T, Dikenelli O (2009) An architecture for verification of access control policies with multi agent system ontologies. In: 33rd Annual IEEE International of Computer Software and Applications Conference, 2009. COMPSAC ’09. vol 2, pp 52–55.  https://doi.org/10.1109/COMPSAC.2009.114
  218. 218.
    Tiryaki AM, âĹžñztuna S, Dikenelli O, Erdur RC, Öztuna S (2007) SUNIT: a unit testing framework for test driven development of multi-agent systems. In: Padgham L, Zambonelli F (eds) Agent-oriented software engineering VII, volume 4405 of lecture notes in computer science. Springer, Berlin, pp 156–173. ISBN 978-3-540-70944-2.  https://doi.org/10.1007/978-3-540-70945-9_10
  219. 219.
    UPPAAL Team (2017) UPPAAL. http://www.uppaal.org/
  220. 220.
    Van Eijk RM, de Boer FS, van der Hoek W, Meyer JJ (2003) A verification framework for agent communication. Auton Agent Multi-Agent Syst V6(2):185–219.  https://doi.org/10.1023/A:1021836202093 CrossRefGoogle Scholar
  221. 221.
    Wan W, Bentahar J, Hamza AB (2011) Model checking epistemic and probabilistic properties of multi-agent systems. In: Mehrotra KG, Mohan CK, Oh JC, Varshney PK, Ali M (eds) Modern approaches in applied intelligence, volume 6704 of lecture notes in computer science. Springer, Berlin, pp 68–78. ISBN 978-3-642-21826-2.  https://doi.org/10.1007/978-3-642-21827-9_8
  222. 222.
    Wang S, Zhu H (2012) CATest: a test automation framework for multi-agent systems. In: 2012 IEEE 36th Annual of Computer Software and Applications Conference (COMPSAC). pp 148–157. ISBN 9780769547367.  https://doi.org/10.1109/COMPSAC.2012.24
  223. 223.
    Wooldridge M, Fisher M, Huget M-P, Parsons S (2002) Model checking multi-agent systems with MABLE. In: Proceedings of the First International Joint Conference on Autonomous Agents and Multiagent Systems: Part 2, AAMAS ’02. ISBN 1581134800. ACM, New York, p 952Google Scholar
  224. 224.
    Wright CJ, Mcminn P, Gallardo J (2012) Testing multi-agent based simulations using MASTER. In: Multi-agent based simulation 2012Google Scholar
  225. 225.
    Wright CJ, McMinn P, Gallardo J (2013) Towards the auto matic identification of faulty multi-agent based simulation runs using MASTER. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 7838 LNAI. pp 143–156. ISBN 9783642388583.  https://doi.org/10.1007/978-3-642-38859-0_11
  226. 226.
    Dianxiang X, Volz RA, Ioerger TR, Yen J (2002) Modeling and verifying multi-agent behaviors using predicate transition nets. In: Proceedings of the 14th international conference on software engineering and knowledge engineering. ACM Press, pp 193–200Google Scholar
  227. 227.
    Xu P, Deters R (2004) Using event-streams for fault-management in MAS. In: IEEE/WIC/ACM International Conference on Intelligent Agent Technology, 2004. (IAT 2004). pp 433–436. Proceedings. ISBN 0769521010.  https://doi.org/10.1109/IAT.2004.1342989
  228. 228.
    Xu P, Deters R (2004) MAS and fault-management. In: Proceedings of the 2004 International Symposium on Applications and the Internet. IEEE, pp 283–286. ISBN 0-7695-2068-5.  https://doi.org/10.1109/SAINT.2004.1266129
  229. 229.
    Peng X, Deters R (2005) Fault-management for multi-agent systems. In: Proceedings of the 2005 Symposium on Applications and the internet, pp 287–293.  https://doi.org/10.1109/SAINT.2005.29
  230. 230.
    Yeung WL (2011) Behavioral modeling and verification of multi-agent systems for manufacturing control. Expert Syst Appl 38(11):13555–13562. ISSN 09574174.  https://doi.org/10.1016/j.eswa.2011.04.067 Google Scholar
  231. 231.
    Yeung WL (2011) Formal verification of negotiation protocols for multi-agent manufacturing systems. Int J Prod Res 49(12):3669–3690. ISSN 00207543.  https://doi.org/10.1080/00207543.2010.492407 CrossRefGoogle Scholar
  232. 232.
    Yu E, Cysneiros LM (2003) Designing for privacy in a multi-agent world. Lect Note Artif Intell (Subseries of Lecture Notes in Computer Science) 2631:209–223. ISSN 03029743MATHGoogle Scholar
  233. 233.
    Zamani A (2011) Evaluation of the testing methods in agent-oriented software engineering. http://worldcomp-proceedings.com/proc/p2011/SER3075.pdf
  234. 234.
    Zhang Z, Thangarajah J, Padgham L (2007) Automated unit testing for agent systems. In: Proceedings of the 2nd international working conference on evaluation of novel approaches to software engineering (ENASE), Spain, pp 10–18Google Scholar
  235. 235.
    Zhang Z, Thangarajah J, Padgham L (2008) Automated unit testing intelligent agents in PDT. In: Proceedings of the 7th International Joint Conference on Autonomous Agents and multi-agent systems: Demo Papers, AAMAS ’08. International Foundation for Autonomous Agents and Multiagent Systems, Richland, pp 1673–1674Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Faculty of ComputingUniversiti Teknologi MalaysiaJohor BahruMalaysia
  2. 2.Malaysia & Media and Game Innovation Center of Excellence (MaGICX)Universiti Teknologi MalaysiaJohor BahruMalaysia
  3. 3.FIM, Center for Basic and Applied ResearchUniversity of Hradec KraloveHradec KraloveCzech Republic

Personalised recommendations