Abstract
EDWARD is a system which is being developed to study multimodal human-computer interaction. It incorporates a graph-editor called Gr2 and a Dutch natural language dialogue system called DoNaLD. EDWARD is capable of realizing referring actions in three ways: it can utter unimodal referring expressions, it can generate pointing gestures and it can produce multimodal referring expressions which combine referring expressions with a pointing gesture. The system uses its knowledge base and a context model to decide the type and the conceptual content of its referring expressions. The context model used is based on Alshawi's notions of context factors and salience. Presently seven types of context factors are used. The decision tree and a set of rules used by EDWARD to guide the generation process are described.
This research was carried out within the framework of the research programme ‘Human-Computer Communication using natural language’ (MMC). The MMC-programme is sponsored by SPIN Stimuleringsprojectteam Informaticaonderzoek, BSO, Digital Equipment B.V.
Preview
Unable to display preview. Download preview PDF.
References
Allgayer, J., Jansen-Winkeln, R., Reddig, C., Reithinger, N.: Bidirectional use of knowledge in the multi-modal NL access system XTRA. Proceedings of the Eleventh International Joint Conference on Artificial Intelligence, Detroit, MI USA. 20–25 August 1989 (pp. 1492–1497).
Alshawi, H. (1987): Memory and Context for Language Interpretation. Cambridge (UK): Cambridge University Press.
Bos, E. (in press): A Graph-Editor. In L. Neal and G. Szwillus (Eds.). Syntax-Directed Editing. New York: Academic Press.
Brachman, R., Schmolze, J. (1985): An overview of the KL-ONE knowledge representation system. Cognitive Science, 9, 171–216.
Claassen, W., Huls, C. (1990): DoNaLD: A Dutch Natural Language Dialogue system (SPIN/MMC Research Report no. 11). Nijmegen, The Netherlands: NICI.
Claassen, W., Bos, E., Huls, C. (1990): The Pooh Way in Human-Computer Interaction: Towards Multimodal Interfaces (SPIN/MMC Research Report no. 5). Nijmegen, The Netherlands: NICI.
Dale, R., Haddock, N.: Generating Referring Expressions Involving Relations. Proceedings of the Fifth Meeting of the European Chapter of the Association for Computational Linguistics, Berlin, Germany, April 1991 (pp. 161–166).
De Smedt, K., Geurts, B., Desain, P.: Waiting for the gift of sound and vision: On naturallanguage sentence production in multimodal interfaces. ESPRIT Workshop on Natural Language Processing, Brussels, October 1987.
Grosz, B.J. (1978): Discourse Knowledge. In D. Walker (Ed.), Understanding Spoken Language. New York: North-Holland.
Grosz, B.J., Sidner, C.L. (1986): Attention, Intentions, and the Structure of Discourse. Computational Linguistics, 12(3):175–204, 1986.
Neal, J.G., Shapiro, S.C.: Intelligent Multi-media Interface Technology. Proceedings of the workshop on Architectures for Intelligent Interfaces: Elements and Prototypes, Lockhead AI Center, Monterrey, CA, 1988 (pp. 69–91).
Pattabhiraman, T., Cercone, N.: Salience in Natural Language Generation. Proceedings of the IJCAI-91 Workshop on Decision Making throughout the Generation Process, Sydney, Australia, August 1991 (pp. 34–41).
Reithinger, N.: The performance of an Incremental Generation Component for Multi-modal Dialogue Contributions. In: ????? (1992) ??????? Springer. These Proceedings.
Sidner, C.L. (1979): Towards a Computational Theory of Definite Anaphora Comprehension in English Discourse. Ph.D. Thesis, MIT, Cambridge, MA.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1992 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Claassen, W. (1992). Generating referring expressions in a multimodal environment. In: Dale, R., Hovy, E., Rösner, D., Stock, O. (eds) Aspects of Automated Natural Language Generation. IWNLG 1992. Lecture Notes in Computer Science, vol 587. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-55399-1_17
Download citation
DOI: https://doi.org/10.1007/3-540-55399-1_17
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-55399-1
Online ISBN: 978-3-540-47054-0
eBook Packages: Springer Book Archive