Skip to main content
SpringerLink
Log in

Recognizing intentions, interactions, and causes of plan failures

  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

Natural language systems for the description of image sequences have been developed (e.g. Neumann and Novak, 1986; Herzog et al., 1989). Even though these systems were used to verbalize the behaviour of human agents, they were limited in that they could only describe the purely visual, i.e. spatio-temporal, properties of the behaviour observed. For many applications of such systems (e.g. co-driver systems in traffic, expert systems in high-performance sports, tutorial systems that give “apprentices” instructions in construction tasks, etc.), it seems useful to extend their capabilities to cover a greater part of the performance of a human observer and thus make the system more helpful to the user. In particular, an interpretation process ought to be modelled that yields hypotheses about intentional entities from spatio-temporal information about agents. Its results should be verbalized in natural language.

This article presents an integrated approach for the recognition and natural language description of plans, intentions, interactions between multiple agents, plan failures, and causes of plan failures. The system described takes observations from image sequences as input. This type of input poses specific problems for the recognition process. After moving objects have been extracted from the image sequences by a vision system and spatio-temporal entities (such as spatial relations and events) have been recognized by an event-recognition system, a focussing process selects interesting agents to be concentrated on in the plan-recognition process. The set of plan hypotheses can be reduced by a hypothesis-selection component. Plan recognition serves as the basis for intention recognition, interaction recognition, and plan-failure analysis. The recognized intentional entities are described in natural language. The system is designed to extend the range of capabilities of the system SOCCER, which verbalizes real-world image sequences of soccer games in natural language.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Allen, J. and D. Litman: 1986, ‘Plans, Goals, and Language’.Proceedings of the IEEE,74(7):939–947.

    Google Scholar 

  • Allen, J. and C. Perrault: 1980, ‘Analyzing Intention in Utterances’.Artificial Intelligence,15:143–179.

    Google Scholar 

  • Allen, J.: 1983,Recognizing Intentions from Natural Language Utterances. In: M. Brady and R. Berwick (eds.),Computational Models of Discourse, Cambridge, MA: MIT Press, pp. 107–166.

    Google Scholar 

  • Allen, J.: 1984, ‘Towards a General Theory of Action and Time’.Artificial Intelligence,23:123–154.

    Google Scholar 

  • André, E., G. Herzog and T. Rist: 1988, ‘On the Simultaneous Interpretation of Real World Image Sequences and their Natural Language Description: The System SOCCER’. In:Proc. of the 8th European Conference on Artificial Intelligence, Munich, West Germany, pp. 449–454.

  • Anscombe, G.: 1957,Intention. Oxford: Basil Blackwell.

    Google Scholar 

  • Anscombe, G.: 1986, ‘Intention’. In: A. White (ed.),The Philosophy of Action, London: Oxford University Press, pp. 144–152.

    Google Scholar 

  • Appelt, D.: 1985,Planning English Sentences. Cambridge, England: Cambridge University Press.

    Google Scholar 

  • Audi, R.: 1973, ‘Intending’.Journal of Philosophy,70(13):387–403.

    Google Scholar 

  • Azarewicz, J. G. Fala, R. Fink and C. Heithecker: 1986, ‘Plan Recognition for Airborne Tactical Decision Making’. In:Proc. of the 5th National Conference of the American Association for Artificial Intelligence, Philadelphia, PA, pp. 805–811.

  • Azarewicz, J. G. Fala, I. Hayslip and C. Heithecker: 1987, ‘Multi-Agent Plan Recognition in an Adversarial Domain’. In:Proc. of the Expert Systems in Government (ESIG) Conference, pp. 188–193.

  • Baier, A.: 1970, ‘Act and Intent’.Journal of Philosophy,67(19):648–658.

    Google Scholar 

  • Baier, A.: 1976, ‘Intention, Practical Knowledge and Representation’. In: M. Brand and D. Walton (eds.),Action Theory, pp. 27–43, Dordrecht: Reidel.

    Google Scholar 

  • Balkanski, C.: 1990,Modeling Act-Type Relations in Collaborative Activity. Technical Report 23–90, Aiken Computation Laboratory, Center for Research in Computing Technology, Harvard University, Cambridge, MA.

    Google Scholar 

  • Benoit, J. E. Dombroski, P. Jordan and S. Laskowski: 1988, ‘Expert Systems for Multi-Agent Plan Recognition in Uncertain Environments (Extended Abstract)’. In:Proc. of the AAAI-88 Workshop on Plan Recognition, St. Paul, MN.

  • Bratman, M., D. Israel and M. Pollack: 1988, 'Plans and Resource-Bounded Practical Reasoning.Computational Intelligence,4(4):349–355.

    Google Scholar 

  • Bratman, M.: 1987,Intention, Plans, and Practical Reason. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Brennenstuhl, W.: 1975,Handlungstheorie und Handlungslogik. Vorbereitung zur Entwicklung einer sprachadäquaten Handlungslogik. Kronberg/Taunus: Skriptor.

    Google Scholar 

  • Brewer, W. and D. Dupree: 1983, ‘Use of Plan Schemata in the Recall and Recognition of Goal-Directed Actions’.Journal of Experimental Psychology: Learning, Memory, and Cognition,9:117–129.

    Google Scholar 

  • Broverman, C. K. Huff and V. Lesser: 1986, ‘The Role of Plan Recognition in Design of an Intelligent User Interface’. In:Proc. of the IEEE Conference on Man, Machine, and Cybernetics, pp. 863–868.

  • Bruce, B. and D. Newman: 1978, ‘Interacting Plans’.Cognitive Science,2:195–233.

    Google Scholar 

  • Bruce, B.: 1981, ‘Plans and Social Actions’. In: R. Spiro, B. Bruce, and W. Brewer (eds.),Theoretical Issues in Reading Comprehension, Chapter 15, pp. 367–384, Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Carberry, S.: 1989, ‘Plan Recognition and its Use in Understanding Dialogue’. In: A. Kobsa and W. Wahlster (eds.),User Models in Dialog Systems, Berlin: Springer, pp. 133–162.

    Google Scholar 

  • Carberry, S.: 1990a, ‘A Model of Plan Recognition that Facilitates Default Inferences’. In:Proc. of the Second International Workshop on User Modeling, Honolulu, HI.

  • Carberry, S.: 1990b, ‘Incorporating Default Inferences into Plan Recognition’. In:Proc. of the 8th National Conference of the American Association for Artificial Intelligence, Boston, MA, pp. 471–478.

  • Carbonell, J.: 1979,The Counterplanning Process: A Model of Decision-Making in Adverse Situations. Technical Report, Computer Science Department, Carnegie-Mellon University, Pittsburg, PA.

    Google Scholar 

  • Carver,, N. V. Lesser and D. McCue: 1984, ‘Focusing in Plan Recognition’. In:Proc. of the 4th National Conference of the American Association for Artificial Intelligence, Austin, TX, pp. 42–48.

  • Castelfranchi, C.: 1990, ‘No More Cooperation, Please! Controversial Points about the Social Structure of Verbal Interaction’. In:Proc. of the NATO Advanced Research Workshop on Computational Theories of Communication and their Applications: Problems and Prospects. Castel Ivano, Trentino, Italy.

  • Chapman, D.: 1989,A Framework for External Reference. Internal Report, MIT Artificial Intelligence Laboratory, Cambridge, MA.

    Google Scholar 

  • Charniak, E. and D. McDermott: 1985,Introduction to Artificial Intelligence. Reading, MA: Addison-Wesley.

    Google Scholar 

  • Cohen, P. and H. Levesque: 1987, ‘Intention = Choice + Commitment’. In:Proc. of the 6th National Conference of the American Association for Artificial Intelligence, Seattle, WA, pp. 410–415.

  • Cohen, P. C. Perrault and J. Allen: 1982, ‘Beyond Question Answering’. In: W. Lehnert and M. Ringle (eds.),Strategies for Natural Language Processing, Hillsdale, NJ: Erlbaum, pp. 245–274.

    Google Scholar 

  • Cohen, P.: 1986, ‘Communication as Rational Interaction’.CSLI Monthly,2(2).

  • Desmarais, M. S. Larochelle and L. Giroux: 1987, ‘A Text Editor Consultant’. In:Proc. of the IEEE MONTEX' 87 Conferences: COMBINT 87, New York, NY, pp. 217–219.

  • Dickman, H.: 1963, ‘The Perception of Behavioral Units’. In: R. Barker (ed.),The Stream of Behavior, New York, NY: Appleton-Century-Crofts, pp. 23–41.

    Google Scholar 

  • Finin, T.: 1983, ‘Providing Help and Advice in Task Oriented Systems’. In:Proc. of the 8th International Joint Conference on Artificial Intelligence, Karlsruhe, West Germany, pp. 176–178.

  • Fischer, G. A. Lemke and T. Schwab: 1985, ‘Knowledge-Based Help Systems’. In:Proc. of the CHI' 85 Conference on Human Factors in Computing Systems, San Francisco, CA. Special Issue of the SIGCHI Bulletin, pp. 161–167.

  • Foss, C. and G. Bower: 1986, ‘Understanding Actions in Relation to Goals’. In: N. Sharkey (ed.),Advances in Cognitive Science 1, Chichester: Ellis Horwood, pp. 94–124.

    Google Scholar 

  • Genesereth, M.: 1979, ‘The Role of Plans in Automated Consultation’. In:Proc. of the 6th International Joint Conference on Artificial Intelligence, Tokyo, Japan, pp. 311–319.

  • Georgeff, M.: 1983, ‘Communication and Interaction in Multi-Agent Planning’. In:Proc. of the 3rd National Conference of the American Association for Artificial Intelligence, Washington, DC, pp. 125–129.

  • Goldman, R. and E. Charniak: 1988, ‘A Probabilistic ATMS for Plan Recognition’. In;Proc. of the AAAI-88 Workshop on Plan Recognition, St. Paul, MN.

  • Goldman, A.: 1970,A Theory of Human Action. Englewood Cliffs, NJ: Prentice-Hall.

    Google Scholar 

  • Goodman, B. and D. Litman: 1989,Design Interfaces and Plan Recognition. Technical Report 7103, BBN Systems and Technologies Corp., Cambridge, MA.

    Google Scholar 

  • Greer, J. and G. McCalla: 1988,A Computational Framework for Granularity and its Application to Educational Diagnosis. Research Report 88–4, ARIES Laboratory, Department of Computational Science, University of Saskatchewan, Saskatoon, SK.

    Google Scholar 

  • Grosz, B. and C. Sidner: 1990, ‘Plans for Discourse’. In: P. Cohen, J. Morgan, and M. Pollack (eds.),Intentions in Communication, Cambridge, MA: MIT Press, pp. 417–444.

    Google Scholar 

  • Hecking, M.: 1987, ‘How to Use Plan Recognition to Improve the Abilities of the Intelligent Help System SINIX Consultant’. In:Proc. of the INTERACT ' 87.

  • Hecking, M.: 1990, ‘A Logical Treatment of Distributed and Multi-Level Plan Recognition’. In:Proc. of the AAAI-90 Workshop on Adversarial Reasoning.

  • Hertzberg, J.: 1989,Planen. Einführung in die Planerstellungsmethoden der Künstlichen Intelligenz. Mannheim: BI-Wissenschaftsverlag.

    Google Scholar 

  • Herzog, G. and G. Retz-Schmidt: 1990, ‘Das System SOCCER: Simultane Interpretation und natürlichsprachliche Beschreibung zeitveränderlicher Szenen’. In: J. Perl (ed.),Sport und Informatik, Schorndorf: Hofmann, pp. 95–119.

    Google Scholar 

  • Herzog, G. C.-K. Sung, E. André, W. Enkelmann, H.-H. Nagel, T. Rist and W. Wahlster: 1989, ‘Incremental Natural Language Description of Dynamic Imagery’. In: C. Freksa and W. Brauer (eds.),Wissensbasierte Systeme. 3. Internationaler GI-Kongreβ, Berlin: Springer, pp. 153–162.

    Google Scholar 

  • Huff, K. and V. Lesser: 1982,Knowledge-Based Command Understanding: An Example for the Software Development Environment. COINS Technical Report 82–6, Computer and Information Science Department, University of Massachusetts at Amherst, Amherst, MA.

    Google Scholar 

  • Kautz, H. and J. Allen: 1986, ‘Generalized Plan Recognition’. In:Proc. of the 5th National Conference of the American Association for Artificial Intelligence, Philadelphia, PA, pp. 32–37.

  • Kautz, H.: 1987,A Formal Theory of Plan Recognition. PhD thesis, Department of Computer Science, University of Rochester, Rochester, NY.

    Google Scholar 

  • Kautz, H.: 1990, ‘A Circumscriptive Theory of Plan Recognition’. In: P. Cohen, J. Morgan, and M. Pollack (eds.),Intentions in Communication, Cambridge, MA: MIT Press, pp. 105–133.

    Google Scholar 

  • Kenny, A.: 1963,Action, Emotion and Will. London: Routledge & Kegan Paul.

    Google Scholar 

  • Kim, J.: 1976, ‘Intention and Practical Inference’. In: J. Manninen and R. Tuomela (eds.),Essays on Explanation and Understanding, Dordrecht: Reidel, pp. 249–269.

    Google Scholar 

  • Konolige, K. and M. Pollack: 1989, ‘Ascribing Plans to Agents. Preliminary Report’. In:Proc. of the 11th International Joint Conference on Artificial Intelligence, Detroit, MI, pp. 924–930.

  • Lames, M. J. Perl, H.-J. Schröder, T. Uthman, and W.-D. Miethling: 1990, ‘Der Einsatz von Expertensystemen im Sport am Beispiel des TEnnis-Simulations-SYstems TESSY’. In: J. Perl (ed.),Sport und Informatik, Schorndorf: Hofmann, pp. 75–91.

    Google Scholar 

  • Levesque,, H. P. Cohen and J. Nunes: 1990, ‘On Acting Together’. In:Proc. of the 8th National Conference of the American Association for Artificial Intelligence, Boston, MA, pp. 94–99.

  • Lichtenstein, E. and W. Brewer: 1980, ‘Memory for Goal-Directed Actions’.Cognitive Psychology,12: 412–445.

    Google Scholar 

  • Litman, D. and J. Allen: 1987, ‘A Plan Recognition Model for Subdialogues in Conversations’.Cognitive Science,11: 163–200.

    Google Scholar 

  • Litman, D. and J. Allen: 1990, ‘Discourse Processing and Commonsense Plans’. In: P. Cohen, J. Morgan, and M. Pollack (eds.),Intentions in Communication, Cambridge, MA: MIT Press, pp. 365–388.

    Google Scholar 

  • Lochbaum,, K. B. Grosz and C. Sidner: 1990, ‘Models of Plans to Support Communication: An Initial Report’. In:Proc. of the 8th National Conference of the American Association for Artificial Intelligence, Boston, MA, pp. 485–490.

  • London, R. and W. Clancey: 1982, ‘Plan Recognition Strategies in Student Modeling: Prediction and Description’. In:Proc. of the 2nd National Conference of the American Association for Artificial Intelligence, Pittsburgh, PA, pp. 335–338.

  • London, R.: 1991, ‘Student Modeling to Support Multiple Instructional Approaches’.User Modeling and User-Adapted Interaction,1. Forthcoming.

  • Mahesh,, K. K. Eiselt and A. Rau: 1990, ‘A Theory of Representation for Understanding Stories in Multi-Agent Adversarial Domains’. In:Proc. of the AAAI-90 Workshop on Adversarial Reasoning.

  • McCue, D. and V. Lesser: 1983,Focusing and Constraint Management in Intelligent Interface Design. COINS Technical Report 83–36, Computer and Information Science Department, University of Massachusetts at Amherst, Amherst, MA.

    Google Scholar 

  • Miller, G. and P. Johnson-Laird: 1976,Language and Perception. Cambridge, England: Cambridge University Press.

    Google Scholar 

  • Miller, G. E. Galanter and K. Pribram: 1960,Plans and the Structure of Behaviour. New York, NY: Holt, Rinehart, and Winston.

    Google Scholar 

  • Nagel, H.-H.: 1988, ‘From Image Sequences Towards Conceptual Descriptions’.Image and Vision Computing,6(2):59–74.

    Google Scholar 

  • Neumann, B. and H.-J. Novak: 1986, ‘NAOS: Ein System zur natürlichsprachlichen Beschreibung zeitveränderlicher Szenen’.Informatik. Forschung und Entwicklung,1(1):83–92.

    Google Scholar 

  • Pleines, J.: 1975, ‘Kausale Relationen und Intentionalität’. In: V. Ehrich and P. Finke (eds.),Beiträge zur Grammatik und Pragmatik, Kronberg/Taunus: Scriptor, pp. 55–70.

    Google Scholar 

  • Pollack, M.: 1986, ‘A Model of Plan Inference that Distinguishes between the Beliefs of Actors and Observers’. In:Proc. of the 24th Annual Meeting of the ACL,Columbia University,New York, NY, pp. 207–214.

    Google Scholar 

  • Pollack, M.: 1989, Plan Recognition Beyond STRIPS. In:Proc. of the IJCAI' 89 Workshop on Plan Recognition.

  • Pollack, M.: 1990, ‘Plans as Complex Mental Attitudes’. In: P. Cohen, J. Morgan, and M. Pollack (eds.),Intentions in Communication, Cambridge, MA: MIT Press, pp. 77–103.

    Google Scholar 

  • Retz-Schmidt, G.: 1988, ‘A REPLAI of SOCCER: Recognizing Intentions in the Domain of Soccer Games’. In:Proc. of the 8th European Conference on Artificial Intelligence, Munich, West Germany, pp. 455–457.

  • Retz-Schmidt, G.: 1991,Die Interpretation des Verhaltens mehrerer Akteure in Szenenfolgen. PhD thesis, Fachbereich Informatik, Universität des Saarlandes, Saarbrücken.

    Google Scholar 

  • Rosenschein, J.: 1986,Rational Interaction: Cooperation among Intelligent Agents. PhD thesis, Department of Computer Science, Stanford University.

  • Ross, P. and J. Lewis: 1988, ‘Plan Recognition for Intelligent Tutoring Systems’. In: Ercoli and Lewis (eds.),Artificial Intelligence Tools in Education', New York, NY: Elsevier, pp. 29–37.

    Google Scholar 

  • Sandewall, E. G. Adorni, H.-H. Nagel and M. Thonnat: 1990, ‘PROMETHEUS Session’. In:Proc. of the 9th European Conference on Artificial Intelligence, Stockhom, Sweden, pp. 779–780.

  • Schank, R. and R. Abelson: 1977,Scripts, Plans, Goals and Understanding: An Inquiry into Human Knowledge Structures. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Schmidt, C. N. Sridharan and J. Goodson: 1978, ‘The Plan Recognition Problem: An Intersection of Psychology and Artificial Intelligence’.Artificial Intelligence,11:45–83.

    Google Scholar 

  • Searle, J.: 1990, ‘Collective Intentions and Actions’. In: P. Cohen, J. Morgan, and M. Pollack (eds.),Intentions in Communication, Cambridge, MA: MIT Press, pp. 401–415.

    Google Scholar 

  • Sidner, C.: 1985, ‘Plan Parsing for Intended Response Recognition in Discourse’.Computational Intelligence,1(1): 1–10.

    Google Scholar 

  • Strömberg, D. D. Driankov, N. Olander and G. Petterson: 1989, ‘Plan Recognition for a Fighter Aircraft’. In:Proc. of the 9th International Conference on Expert Systems of Applications, pp. 277–286.

  • Tuomela, R. and K. Miller: 1988, ‘We-Intentions’.Philosophical Studies,53:367–389.

    Google Scholar 

  • Werner, E.: 1988a, ‘Social Intentions’. In:Proc. of the 8th European Conference on Artificial Intelligence, Munich, West Germany, pp. 719–723.

  • Werner, E.: 1988b, ‘Toward a Theory of Communication and Cooperation for Multiagent Planning’. In: M. Vardi (ed.),Proc. of the 2nd Conference on Theoretical Aspects of Reasoning about Knowledge, Los Altos, CA: Morgan Kaufmann, pp. 129–143.

    Google Scholar 

  • Werner, E.: 1989, ‘Cooperating Agents: A Unified Theory of Communication and Social Structure’. In: M. Huhns and L. Gasser (eds.),Distributed Artificial Intelligence. Vol. 2, Los Altos, CA: Morgan Kaufmann.

    Google Scholar 

  • Werner, E.: 1990, ‘What can Agents do Together? A Semantics for Reasoning about Cooperative Ability’. In:Proc. of the 9th European Conference on Artificial Intelligence, Stockhom, Sweden, pp. 694–701.

  • Wilensky, R.: 1978,Understanding Goal-Based Stories. Technical Report 140, Department of Computer Science, Yale University, New Haven, CT.

    Google Scholar 

  • Wilensky, R.: 1983,Planning and Understanding: A Computational Approach to Human Reasoning. Reading, MA: Addison-Wesley.

    Google Scholar 

  • Wobcke, W: 1988, ‘A Logical Theory of Plan Recognition’. In:Proc. of the AAAI-88 Workshop on Plan Recognition, St. Paul, MN.

  • Woodroffe, M.: 1988, ‘Plan Recognition and Intelligent Tutoring Systems’. In: J. Self (ed.),Intelligent Computer-Aided Instruction, London: Chapman & Hall, pp. 212–225.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dept. of Computer Science, University of Hamburg, Bodenstedtstrasse 16, 50, Hamburg, Germany

    Gudula Retz-Schmidt

Authors
  1. Gudula Retz-Schmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

The work reported in this article was partially supported by the Special Collaborative Program on Artificial Intelligence and Knowledge-Based Systems (SFB 314) of the German Science Foundation (DFG), project N2: VITRA.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gudula, RS. Recognizing intentions, interactions, and causes of plan failures. User Model User-Adap Inter 1, 173–202 (1991). https://doi.org/10.1007/BF00154477

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00154477

Key words

Search

Navigation