Formal Assurance for Cooperative Intelligent Autonomous Agents

  • Siddhartha BhattacharyyaEmail author
  • Thomas C. Eskridge
  • Natasha A. Neogi
  • Marco Carvalho
  • Milton Stafford
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10811)


Developing trust in intelligent agents requires understanding the full capabilities of the agent, including the boundaries beyond which the agent is not designed to operate. This paper focuses on applying formal verification methods to identify these boundary conditions in order to ensure the proper design for the effective operation of the human-agent team. The approach involves creating an executable specification of the human-machine interaction in a cognitive architecture, which incorporates the expression of learning behavior. The model is then translated into a formal language, where verification and validation activities can occur in an automated fashion. We illustrate our approach through the design of an intelligent copilot that teams with a human in a takeoff operation, while a contingency scenario involving an engine-out is potentially executed. The formal verification and counterexample generation enables increased confidence in the designed procedures and behavior of the intelligent copilot system.


Formal verification Intelligent agents Human-machine teams 


  1. 1.
    Newell, A., Shaw, J.C., Simon, H.A.: Report on a general problem-solving program. In: Proceedings of the International Conference on Information Processing, pp. 256–264 (1959)Google Scholar
  2. 2.
    Buchanan, B.G., Shortliffe, E.H.: Rule Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project. The Addison-Wesley Series in Artificial Intelligence. Addison-Wesley Longman Publishing Co., Inc., Boston (1984)Google Scholar
  3. 3.
    Anderson, J.R., Matessa, M., Lebiere, C.: ACT-R: a theory of higher level cognition and its relation to visual attention. Hum.-Comput. Interact. 12(4), 439–462 (1997)CrossRefGoogle Scholar
  4. 4.
    Laird, J.E.: The SOAR Cognitive Architecture. MIT Press, Cambridge (2012)Google Scholar
  5. 5.
    Sutton, R.L., Barto, B.: Reinforcement Learning. MIT Press, Cambridge (2008)Google Scholar
  6. 6.
    Mittal, S., Douglass, S.A.: Net-centric ACT-R based cognitive architecture with DEVS unified process. In: DEVS Symposium Spring Simulation Multiconference, Boston (2011)Google Scholar
  7. 7.
    Garlan, D., Cheng, S., Huang, A., Schmerl, B., Steenkiste, P.: Rainbow: architecture-based self adaptation with reusable infrastructure. Computer 37(10), 46–54 (2004)CrossRefGoogle Scholar
  8. 8.
    Wen, M., Ehlers, R., Topcu, U.: Correct-by-synthesis reinforcement learning with temporal logic constraints. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (2015)Google Scholar
  9. 9.
    Sharifloo, A.M., Spoletini, P.: LOVER: Light-weight fOrmal Verification of adaptivE systems at Run time. In: Păsăreanu, C.S., Salaün, G. (eds.) FACS 2012. LNCS, vol. 7684, pp. 170–187. Springer, Heidelberg (2013). CrossRefGoogle Scholar
  10. 10.
    Curzon, P., Ruknas, R., Blandford, A.: An approach to formal verification of human computer interaction. Form. Asp. Comput. 19, 513–550 (2007)CrossRefzbMATHGoogle Scholar
  11. 11.
    O’Conner, M., Tangirala, S., Kumar, R., Bhattacharyya, S., Sznaier, S., Holloway, L.: A bottom-up approach to verification of hybrid model-based hierarchical controllers with application to underwater vehicles. In: Proceedings of American Control Conference (2006)Google Scholar
  12. 12.
    Rocha, C., Cadavid, H., Muñoz, C., Siminiceanu, R.: A formal interactive verification environment for the plan execution interchange language. In: Derrick, J., Gnesi, S., Latella, D., Treharne, H. (eds.) IFM 2012. LNCS, vol. 7321, pp. 343–357. Springer, Heidelberg (2012). CrossRefGoogle Scholar
  13. 13.
    Dowek, G., Munoz, C., Pasareanu, C.: A small-step semantics of PLEXIL (2008)Google Scholar
  14. 14.
    Dowek, G., Munoz, C., Pasareanu, C.: A formal analysis framework for PLEXIL. In: Proceedings of 3rd Workshop on Planning and Plan Execution for Real-World Systems (2007)Google Scholar
  15. 15.
    Dowek, G., Munoz, C., Rocha, C.: Rewriting logic semantics of a plan execution language. In: EPTCS, vol. 18, pp. 77–91 (2009)Google Scholar
  16. 16.
    Strauss, P.J.: Executable semantics for PLEXIL: simulating a task-scheduling language in Haskell. Masters thesis (2009)Google Scholar
  17. 17.
    Balasubramanian, D., Pasareanu, C., Whalen, M.W., Karsai, G., Lowry, M.R.: Polyglot: modeling and analysis for multiple statechart formalisms. In: Dwyer, M.B., Tip, F. (eds.) ISSTA. ACM (2011)Google Scholar
  18. 18.
    Verdejo, A., Martí-Oliet, N.: Two case studies of semantics execution in Maude: CCS and LOTOS. Formal Methods Syst. Des. 27, 113–172 (2005)CrossRefzbMATHGoogle Scholar
  19. 19.
    Eskridge, T.C., Carvalho, M.M., Bhattacharyya, S., Vogl, T.: Verifiable autonomy final report. Technical report, Florida Institute of Technology and Rockwell Collins (2015)Google Scholar
  20. 20.
    Cimatti, A., Clarke, E., Giunchiglia, E., Giunchiglia, F., Pistore, M., Roveri, M., Sebastiani, R., Tacchella, A.: NuSMV 2: an opensource tool for symbolic model checking. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 359–364. Springer, Heidelberg (2002). CrossRefGoogle Scholar
  21. 21.
    Uppaal website (2010).
  22. 22.
    Owre, S., Rajan, S., Rushby, J.M., Shankar, N., Srivas, M.: PVS: combining specification, proof checking, and model checking. In: Alur, R., Henzinger, T.A. (eds.) CAV 1996. LNCS, vol. 1102, pp. 411–414. Springer, Heidelberg (1996). CrossRefGoogle Scholar
  23. 23.
    de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008). CrossRefGoogle Scholar
  24. 24.
    Larsen, K.G., Pettersson, P., Yi, W.: Model-checking for real-time systems. In: Reichel, H. (ed.) FCT 1995. LNCS, vol. 965, pp. 62–88. Springer, Heidelberg (1995). CrossRefGoogle Scholar
  25. 25.
    Bengtsson, J., Larsen, K., Larsson, F., Pettersson, P., Yi, W.: UPPAAL: a tool suite for automatic verification of real-time systems. Theor. Comput. Sci. (1996). RS-96-58Google Scholar
  26. 26.
    Alur, R., David, L.D.: A theory of timed automata. Theor. Comput. Sci. 126, 183–235 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Bozga, M., Daws, C., Maler, O., Olivero, A., Tripakis, S., Yovine, S.: Kronos: a model-checking tool for real-time systems. In: Ravn, A.P., Rischel, H. (eds.) FTRTFT 1998. LNCS, vol. 1486, pp. 298–302. Springer, Heidelberg (1998). CrossRefGoogle Scholar
  28. 28.
    Neogi, N.A.: Capturing safety requirements to enable effective task allocation between humans and automaton in increasingly autonomous systems. In: Proceedings of the AIAA Aviation Forum. 16th AIAA Aviation Technology, Integration, and Operations Conference (AIAA 2016-3594) (2016)Google Scholar
  29. 29.
    Code of Federal Regulations: Title 14 Aeronautics and Space. Federal Register, May 1962.
  30. 30.
    The Boeing Company: Boeing 737 pilots operating handbook. Continental Airlines, November 2002.
  31. 31.
    Official x-plane website (2016).

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Siddhartha Bhattacharyya
    • 1
    Email author
  • Thomas C. Eskridge
    • 1
  • Natasha A. Neogi
    • 2
  • Marco Carvalho
    • 1
  • Milton Stafford
    • 1
  1. 1.School of ComputingFlorida Institute of TechnologyMelbourneUSA
  2. 2.NASA Langley Research CenterHamptonUSA

Personalised recommendations