Skip to main content
SpringerLink
Log in

Do Oral Messages Help Visual Search?

  • Chapter
Advances in Natural Multimodal Dialogue Systems

Part of the book series: Text, Speech and Language Technology ((TLTB,volume 30))

Abstract

A preliminary experimental study is presented, that aims at eliciting the contribution of oral messages to facilitating visual search tasks on crowded displays. Results of quantitative and qualitative analyses suggest that appropriate verbal messages can improve both target selection time and accuracy. In particular, multimodal messages comprising a visual presentation of the isolated target together with absolute spatial oral information on its location in the displayed scene seem most effective. These messages also got top-ranking ratings from most subjects.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Ahlberg, C., Williamson, C., and Shneiderman, B. (1992). Dynamic Queries for Information Exploration: An Implementation and Evaluation. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 619–626, New York. ACM Press.

    Google Scholar 

  • André, E. (1997). WIP and PPP: A Comparison of two Multimedia Presentation Systems in Terms of the Standard Reference Model. Computer Standards and Interfaces, 18(6–7):555–564.

    Article  Google Scholar 

  • André, E. and Rist, T. (1993). The Design of Illustrated Documents as a Planning Task. In Maybury, M. T., editor, Intelligent Multimedia Interfaces, pages 96–116. Menlo Park, CA: AAAI/MIT Press.

    Google Scholar 

  • Baber, C. (2001). Computing in a Multimodal World. In Proceedings of the First International Conference on Universal Access in Human-Computer Interaction (UAHCI), pages 232–236. Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Bernsen, N. O. (1994). Foundations of Multimodal Representations, a Taxonomy of Representational Modalities. Interacting with Computers, 6:347–371.

    Article  Google Scholar 

  • Bieger, G. R. and Glock, M. D. (1986). Comprehending Spatial and Contextual Information in Picture-Text Instructions. Journal of Experimental Education, 54:181–188.

    Google Scholar 

  • Burhans, D. T., Chopra, R., and Srihari, R. K. (1995). Domain Specific Understanding of Spatial Expressions. In Proceedings of the IJCAI Workshop on the Representation and Processing of Spatial Expressions, pages 33–40, Montréal, Canada.

    Google Scholar 

  • Card, S. K. and Mackinlay, J. (1997). The Structure of the Information Visualization Design Space. In Proceedings of IEEE Symposium on Information Visualization (InfoVis), pages 92–99, Phoenix, AZ. IEEE Computer Society Press.

    Google Scholar 

  • Card, S. K., Mackinlay, J., and Shneiderman, B. (1999). Readings in Information Visualization — Using Vision to Think. San Francisco (CA): Morgan Kaufmann Publishers.

    Google Scholar 

  • Chelazzi, L. (1999). Serial Attention Mechanisms in Visual Search: A Critical Look at the Evidence. Psychological Research, 62(2–3):195–219.

    Article  Google Scholar 

  • Coutaz, J. and Caelen, J. (1991). A Taxonomy for Multimedia and Multimodal User Interfaces. In Proceedings of the First ERCIM Workshop on Multimodal HCI, pages 143–148, INESC, Lisbon.

    Google Scholar 

  • Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., and Young, R. (1995). Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The CARE Properties. In Proceedings of INTERACT, pages 115–120, Lillehammer, Norway.

    Google Scholar 

  • Cribbin, T. and Chen, C. (2001a). A Study of Navigation Strategies in Spatial-Semantic Visualizations. In Proceedings of the Ninth International Conference on Human-Computer Interaction (HCI International), pages 948–952. Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Cribbin, T. and Chen, C. (2001b). Exploring Cognitive Issues in Visual Information Retrieval. In Proceedings of INTERACT, pages 166–173, Tokyo, Japan. Amsterdam: IOS Press.

    Google Scholar 

  • de Vries, G. and Johnson, G.I. (1997). Spoken Help for a Car Stereo: An Exploratory Study. Behaviour and Information Technology, 16(2):79–87.

    Article  Google Scholar 

  • Engelkamp, J. (1992). Modality and Modularity of the Mind. In Actes du 5ème Colloque de l’ARC ‘Percevoir, Raisonner, Agir — Articulation de Modèles Cognitifs’, pages 321–343, Nancy.

    Google Scholar 

  • Faraday, P. and Sutcliffe, A. (1997). Designing Effective Multimedia Presentations. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 272–278. New York: ACM Press & Addison Wesley.

    Google Scholar 

  • Fekete, J.-D. and Plaisant, C. (2002). Interactive Information Visualization of a Million Items. In Proceedings of IEEE Symposium on Information Visualization (InfoVis), pages 117–124. Boston, MA: IEEE Press.

    Google Scholar 

  • Findlay, J. M. and Gilchrist, D. (1998). Eye Guidance and Visual Search. In Underwood, G., editor, Eye Guidance in Reading and Scene Perception, chapter 13, pages 295–312. Amsterdam: Elsevier.

    Google Scholar 

  • Frank, A. (1998). Formal Models for Cognition — Taxonomy of Spatial Location Description and Frames of Reference. In Freksa, C., Habel, C., and Wender, K. F., editors, Spatial Cognition — An Interdisciplinary Approach to Representing and Processing Spatial Knowledge, volume 1, pages 293–312. Berlin: Springer Verlag.

    Google Scholar 

  • Grabowski, N. A. and Barner, K. E. (1998). Data Visualisation Methods for the Blind Using Force Feedback and Sonification. In Proceedings of the SPIE Conference on Telemanipulator and Telepresence Technologies, volume 3524, pages 131–139. Bellingham, WA: International Society for Optical Engineering (SPIE).

    Google Scholar 

  • Gramopadhye, A. K. and Madhani, K. (2001). Visual Search and Visual Lobe Size. In Arcelli, C., editor, Proceedings of Fourth International Conference on Visual Form, Lecture Notes in Computer Science, volume 2059, pages 525–531. Berlin Heidelberg: Springer Verlag.

    Google Scholar 

  • Henderson, J. M. and Hollingworth, A. (1998). Eye Movements during Scene Viewing: an Overview. In Underwood, G., editor, Eye Guidance in Reading and Scene Perception, chapter 12, pages 269–293. Amsterdam: Elsevier.

    Google Scholar 

  • Lamping, J., Rao, R., and Pirolli, P. (1995). A Focus + Context Technique Based on Hyperbolic Geometry for Visualizing Large Hierarchies. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 401–408. New York: ACM Press & Addison Wesley.

    Google Scholar 

  • Massaro, D. (2002). Personal communication.

    Google Scholar 

  • Maybury, M. T. (2001). Universal Multimedia Information Access. In Proceedings of the First International Conference on Universal Access in Human-Computer Interaction (UAHCI), pages 382–386. Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Maybury, M.T. (1993). Planning Multimedia Explanations Using Communicative Acts. In Maybury, M.T., editor, Intelligent Multimedia Interfaces, pages 59–74. Menlo Park, CA: AAAI/MIT Press.

    Google Scholar 

  • Nigay, L. and Coutaz, J. (1993). A Design Space for Multimodal Systems: Concurrent Processing and Data Fusion. In Proceedings of INTERCHI, pages 172–178. New York: ACM Press & Addison Wesley.

    Google Scholar 

  • Oviatt, S., DeAngeli, A., and Kuhn, K. (1997). Integration and Synchronisation of Input Modes during Multimodal Human-Computer Interaction. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 415–422. New York: ACM Press & Addison Wesley.

    Google Scholar 

  • Rasmussen, J. (1986). Information Processing and Human-Machine Interaction: An Approach to Cognitive Engineering. Amsterdam: North-Holland.

    Google Scholar 

  • Robbe, S., Carbonell, N., and Dauchy, P. (2000). Expression Constraints in Multimodal Human-Computer Interaction. In Proceedings of the International Conference on Intelligent User Interfaces (IUI), pages 225–229. New York: ACM Press.

    Google Scholar 

  • Shneiderman, B. (1983). Direct Manipulation: A Step Beyond Programming Languages. IEEE Computer, 16:57–69.

    Google Scholar 

  • Shneiderman, B. (1996). The Eyes Have It: A task by Data Type Taxonomy for Information Visualizations. In Proceedings of IEEE Workshop on Visual Languages (VL), pages 336–343, Boulder, CO.

    Google Scholar 

  • Sidner, C. L. and Dzikovska, M. (2002). Hosting Activities: Experience with and Future Directions for a Robot Agent Host. In Proceedings of the International Conference on Intelligent User Interfaces (IUI), pages 143–150. New York: ACM Press.

    Google Scholar 

  • Stock, O., Strappavara, C., and Zancanaro, M. (1997). Explorations in an Environment for Natural Language Multimodal Information Access. In Maybury, M. T., editor, Intelligent Multimedia Information Retrieval, pages 381–398. Menlo Park, CA: AAAI/MIT Press.

    Google Scholar 

  • van Diepen, M. J., Wampers, M., and d’Ydewall, G. (1998). Functional Division of the Visual Field: Moving Masks and Moving Windows. In Underwood, G., editor, Eye Guidance in Reading and Scene Perception, chapter 15, pages 337–355. Amsterdam: Elsevier.

    Google Scholar 

  • van Mulken, S., André, E., and Muller, J. (1999). An Empirical Study on the Trustworthiness of Life-Like Interface Agents. In Proceedings of the Eighth International Conference on Human-Computer Interaction (HCI International), pages 152–156. Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Yu, W., Ramloll, R., and Brewster, S. A. (2001). Haptic Graphs for Blind Computer Users. Haptic Human-Computer Interaction, 2058:41–51.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LORIA, 615 rue du Jardin Botanique, 54600, Villers-lès-Nancy, France

    Noëlle Carbonell & Suzanne Kieffer

Authors
  1. Noëlle Carbonell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Suzanne Kieffer
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Waalre, The Netherlands

    Jan C. J. van Kuppevelt

  2. University of Southern Denmark, Odense, Denmark

    Laila Dybkjær  & Niels Ole Bernsen  & 

Rights and permissions

Reprints and permissions

Copyright information

© 2005 Springer

About this chapter

Cite this chapter

Carbonell, N., Kieffer, S. (2005). Do Oral Messages Help Visual Search?. In: van Kuppevelt, J.C.J., Dybkjær, L., Bernsen, N.O. (eds) Advances in Natural Multimodal Dialogue Systems. Text, Speech and Language Technology, vol 30. Springer, Dordrecht. https://doi.org/10.1007/1-4020-3933-6_7

Download citation

Publish with us

Policies and ethics

Search

Navigation