Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Multimodal Communication, Natural Language and Direct Manipulation (Gestures) in Human-Computer Interaction

  • Conference paper
Multimedia Interface Design in Education

Part of the book series: NATO ASI Series ((NATO ASI F,volume 76))

Abstract

Human-computer communication is quite impoverished compared to human-human communication, both in its form and its effectiveness. In many cases the aim of designing a multimedia interface is to produce an interaction which is more like that which takes place between people. Natural language is a highly developed form of (human-human) communication, one which is often supplemented by body language, one of the most important components of which is hand gesture. For example, people often use pointing (deixis) to avoid having to describe something that is complex, inexact or inaccessible to them. On the other hand, natural language can be much more refined at specifying what part or aspect of something is currently of interest. This is particularly true where quantification, abstraction, combination, negation or time are involved. This chapter describes how these two media of human-human interaction — natural language and gesture — can be integrated in a human-computer interface. It describes how these have been applied in a variety of different practical applications to combine the strengths of both modes of interaction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alsina, J.; Hanne, K.H. & Aninos, A.: Aircraft design activities in main phase 2. Esprit Loki Project, Final Report, July 1987

    Google Scholar 

  2. Bertin, J.: Graphische Semiologie, Diagramme, Netze, Karten, (Berlin 1974. Translated from Sémiologie Graphique 1967, English Version Semiology of Graphics, 1983)

    Google Scholar 

  3. Bijl, A. & Szalapaj, P.: Saying what you want with pictures and words. Proceedings 1st Conference on Human-Computer Interaction INTERACT 84. (Shackel, B. ed.) London 1984

    Google Scholar 

  4. Binot, J.L; Demoen, B; Hanne, K-H; Solomon, L; Vassiliou, Y; von Hahn, W. & Wachtel, T.: LOKI: A logic oriented approach to data and knowledge bases supporting natural language interaction. Proceedings Esprit Conference +88. pp. 562–577, Brussels, November 1988

    Google Scholar 

  5. Bolt, R.A.: ‘Put-that-there’: Voice and gesture at the graphics interface. Computer Graphics. 14, 3, pp. 262–270 (1980)

    Article  MathSciNet  Google Scholar 

  6. Brachman, R.J.: Research in natural language understanding. BBN Report No.4274, 1978–1979. Annual Report BBN, 1979

    Google Scholar 

  7. Brown, D. & Chandrasekaran, B.: Design considerations for picture production in a natural language graphics system. Department of Computer & Information Science, Ohio State University, 1983

    Google Scholar 

  8. Foley, D. & van Dam, A.: Fundamentals of interactive computer graphics. Addison-Wesley 1984

    Google Scholar 

  9. Friedell, M.: Automatic graphics environment synthesis. Case Western Reserve University 1983

    Google Scholar 

  10. Gnanamgari, S.: Information presentation through default displays, University of Pennsylvania 1981

    Google Scholar 

  11. Hanne, K. H; Hoepelman, J. P. & Faehnrich, K. P.: Combined graphics/natural language interfaces to knowledge-based systems. In: Proceedings Conference on Artificial Intelligence and Advanced Computer Technology, TCM, Liphook 1986

    Google Scholar 

  12. Hanne, K. & Hoepelman, J.: Combined graphic and natural language interaction, Proceedings Graphics Interfaces’88, pp. 105–111, Edmonton June, 1988

    Google Scholar 

  13. Hanne, K. H. & Wetzel, R.P.: DIS-QUE: Deictic Interaction System — Query Environment. Esprit — LOKI Report, KRGR 5.3 FhG/IAO. Jan. 1987

    Google Scholar 

  14. Hanne, K.H. & Graeble, A.: Design and implementation of direct manipulative and deictic user interfaces to knowledge based systems. In: Human Computer Interaction Interact’87 (H.J. Bullinger, & Shackel, B. eds.), pp. 1067–1073 September 1987

    Google Scholar 

  15. Hayes, P.: Steps towards integrating natural language and graphical interaction for knowledge-based systems. Proceedings European Conference on Artificial Intelligence ECAI’86, Vol.1, pp. 456–465.1986

    Google Scholar 

  16. Hoepelman, J.P.; Hanne, K. H. and Oellinger, W.: Classification of deictic phenomena in natural language. Esprit—LOKI Report KRGR 5.1A, FhG/IAO. January 1986

    Google Scholar 

  17. Kobsa, A. and Allgayer, J.; et al.: Combining deictic gestures and natural language for referent identification. Proceedings International Conference on Computational Linguistics, Bonn, pp. 356–561 1986

    Google Scholar 

  18. Krishnamurti, R. The MOLE picture book: On a logic for design. Design Computing 13, pp. 171–178. 1986

    Google Scholar 

  19. Lockwood, A.: Diagramms: A visual survey on graphs, maps, charts and diagrams for the graphical designer. New York 1969

    Google Scholar 

  20. Mackinlay, J. D.: Automatic design of graphic presentations. Stanford University, Report STAN-CS-86-1138, 1986

    Google Scholar 

  21. Moto-Oka, T.: The intelligent interface system. In: The fifth generation computer project: State of the art report (G.G. Scarrott ed.) pp. 101–114. University of Tokyo, Tokyo, Japan: Pergamon Infotech 1983

    Google Scholar 

  22. Newman, W. N. and Sproul, R.F.: Principles of interactive computer graphics. MacGraw Hill 1984

    Google Scholar 

  23. Ogawa, Y.; Shima, K.; Sugawara, T. & Takagi, S.: Knowledge representation and inference environment: KRINE, an approach to integration of frame, prolog and graphics. Proceedings Conference on Fifth Gen. Comp. Systems, pp. 643–649 1984

    Google Scholar 

  24. Schmauks, D.: Natural and simulated pointing. An interdisciplinary survey. University of Saarbruecken, Working Paper No. 16, XTRA Project, March 1987

    Google Scholar 

  25. Shneiderman, B.: The future of interactive systems and the emerge of direct manipulation. Behaviour and Information Technology, Vol. 1, 3, 237–256 (1982)

    Article  Google Scholar 

  26. Tilling, L.: Early experimental graphs. British Journal for the History of Science. 8, 30, pp. 193–213 (1975)

    Article  Google Scholar 

  27. Weitzman, L.: Designer: A knowledge based graphic design assistant. UCSD, La Jolla. ICS Report 8609 1986

    Google Scholar 

  28. Wetzel, P.: Erkennen von gesten im dialog blinder benutzer mit dem rechner, Diplomarbeit, Institut für Informatik, University Stuttgart, Stuttgart, 1988

    Google Scholar 

  29. Wilson, G.A. & Herot, C.F.: Semantics vs. graphics: To show or not to show. Proceedings Annual Conference on VLD, pp. 183–196 Montreal 1980

    Google Scholar 

  30. Yokota, M. et al.: Language-picture question: Answering through common semantic representation and its application to the world of weather report. In: Natural language communication with pictorial information systems (L. Bolc ed.) 1984

    Google Scholar 

  31. Zdybel, F.: An engine for intelligent graphics. In: Cooperative interfaces to information systems (L. Bolc & M. Jarke ed). Springer 1986

    Google Scholar 

  32. Zimmermann, T.G.; Lanier, J.; Blanchard, G; Byrson, S. & Harvill, Y.: A hand gesture interface device. Proceedings CHI’87 pp. 189–192 (1987)

    Google Scholar 

  33. The Personal Computer in the Year 2000, The Worksheet, (Personal Computer World, Oct 1988, pp. 155–162, 1988)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Fraunhofer-Institut für Arbeitswirtschaft und Organisation (FhG/IAO), Holzgartenstr. 17, D-7000, Stuttgart 1, Germany

    Karl-Heinz Hanne

Authors
  1. Karl-Heinz Hanne
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Human-Computer Interaction Research Group, Department of Computer Science, University of York, YO1 5DD, York, England

    Alistair D. N. Edwards

  2. Department of Computing, Faculty of Mathematics and Computing, The Open University, MK7 6AA, Milton Keynes, England

    Simon Holland

Rights and permissions

Reprints and permissions

Copyright information

© 1992 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Hanne, KH. (1992). Multimodal Communication, Natural Language and Direct Manipulation (Gestures) in Human-Computer Interaction. In: Edwards, A.D.N., Holland, S. (eds) Multimedia Interface Design in Education. NATO ASI Series, vol 76. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-58126-7_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-58126-7_11

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-55046-4

  • Online ISBN: 978-3-642-58126-7

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation