Towards Designing Audio Assistance for Comprehending Haptic Graphs: A Multimodal Perspective

  • Özge Alaçam
  • Christopher Habel
  • Cengiz Acartürk
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8009)


Statistical graphs, such as line graphs are widely used in multimodal communication settings. Language accompanies graphs and humans produce gestures during the course of communication. For visually impaired people, haptic-audio interfaces provide perceptual access to graphical representations. The local and sequential character of haptic perception introduces limitations in haptic perception of hard-to-encode information, which can be resolved by providing audio assistance. In this article we first present a review of multimodal interactions between gesture, language and graphical representations. We then focus on methodologies for investigating hard-to-encode information in graph comprehension. Finally, we present a case study to provide insight for designing audio assistance.


Haptic Graph Comprehension Audio-Verbal Assistance 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Acartürk, C.: Multimodal Comprehension of Graph-text Constellations: An Information Processing Perspective. University of Hamburg Dissertation, Hamburg (2010)Google Scholar
  2. 2.
    Habel, C., Acartürk, C.: Towards Comprehension of Haptic Multiple-line Graphs. In: Goncu, C., Marriott, K. (eds.) Proceedings of the Diagrams Workshop on Accessible Graphics: Graphics for Vision Impaired People, Kent, UK (2012)Google Scholar
  3. 3.
    Hegarty, M.: The Cognitive Science of Visual-spatial Displays: Implications for Design. Topics in Cognitive Science 3, 446–474 (2011)CrossRefGoogle Scholar
  4. 4.
    Kosslyn, S.M.: Understanding Charts and Graphs. Applied Cognitive Psychology 3, 185–226 (1989)CrossRefGoogle Scholar
  5. 5.
    Tufte, E.R.: The Visual Display of Quantitative Information. Graphic Press, Cheshire CT (1983)Google Scholar
  6. 6.
    Zacks, J., Levy, E., Tversky, B.: &Schiano, D.: Graphs in Print. In: Anderson, M., Meyer, B., Olivier, P. (eds.) Diagrammatic Representation and Reasoning, pp. 187–206. Springer, London (2002)CrossRefGoogle Scholar
  7. 7.
    Abu Doush, I., Pontelli, E., Simon, D., Son, T.C., Ma, O.: Making Microsoft ExcelTM Accessible: Multimodal Presentation of Charts. In: Proceedings Eleventh International ACM SIGACCESS Conference on Computers and Accessibility, Pittsburg, PA, USA, pp. 147–154 (2009)Google Scholar
  8. 8.
    Yu, W., Brewster, S.A.: Evaluation of Multimodal Graphs for Blind People. Journal of Universal Access in the Information Society 2, 105–124 (2003)CrossRefGoogle Scholar
  9. 9.
    Sjöström, C., Danielsson, H., Magnusson, C., Rassmus-Gröhn, K.: Phantom-based Haptic Line Graphics for Blind Persons. Visual Impairment Research 5, 13–32 (2003)CrossRefGoogle Scholar
  10. 10.
    Yu, J., Habel, C.: A Haptic-Audio Interface for Acquiring Spatial Knowledge about Apartments. In: Magnusson, C., Szymczak, D., Brewster, S. (eds.) HAID 2012. LNCS, vol. 7468, pp. 21–30. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  11. 11.
    Loomis, J., Klatzky, R., Lederman, S.: Similarity of Tactual and Visual Picture Recognition with Limited Field of View. Perception 20, 167–177 (1991)CrossRefGoogle Scholar
  12. 12.
    Mackinlay, J.D.: Automating the Design of Graphical Presentations of Relational Information. ACM Transactions on Graphics 5, 110–141 (1986)CrossRefGoogle Scholar
  13. 13.
    Pearce, E.A., Smith, C.G.: Fodor’s World Weather Guide. Random House, New York (1998)Google Scholar
  14. 14.
    Habel, C., Acarturk, C.: Causal Inference in Graph-Text Constellations: Designing Verbally Annotated Graphs. Tsinghua Science & Technology 16, 7–12 (2011)CrossRefGoogle Scholar
  15. 15.
    Acartürk, C.: Points, Lines and Arrows in Statistical Graphs. In: Cox, P., Plimmer, B., Rodgers, P. (eds.) Diagrams 2012. LNCS, vol. 7352, pp. 95–101. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  16. 16.
    Acartürk, C., Alacam, O.: Gestures in Communication through Line Graphs. In: Miyake, N., Peebles, D., Cooper, R.P. (eds.) Proceedings of the 34th Annual Conference of the Cognitive Science Society, Austin, TX, pp. 66–71. Cognitive Science Society (2012)Google Scholar
  17. 17.
    Habel, C., Acartürk, C.: On Reciprocal Improvement in Multimodal Generation: Co-reference by Text and Information Graphics. In: van der Sluis, I., Theune, M., Reiter, E., Krahmer, E. (eds.) Proceedings of the Workshop on Multimodal Output Generation, MOG 2007, pp. 69–80 (2007)Google Scholar
  18. 18.
    Acartürk, C., Taboada, M., Habel, C.: Cohesion in Multi-modal Documents: Effects of Cross-referencing. Information Design Journal (in press)Google Scholar
  19. 19.
    Lowe, R., Boucheix, J.M.: Cueing Complex Animations: Does Direction of Attention Foster Learning Processes? Learning and Instruction 21, 650–663 (2011)CrossRefGoogle Scholar
  20. 20.
    Acarturk, C., Habel, C., Cagiltay, K., Alacam, O.: Multimodal Comprehension of Language and Graphics: Graphs with and without Annotations. Journal of Eye Movement Research 1, 2 (2008)Google Scholar
  21. 21.
    Habel, C., Acartürk, C.: Eye-tracking Evidence for Multimodal Language-graphics Comprehension: The Role of Integrated Conceptual Representations. In: Proceedings of NODALIDA 2009: The Workshop on Multimodal Communication- From Human Behavior to Computational Models, vol. 6 (2009)Google Scholar
  22. 22.
    McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992)Google Scholar
  23. 23.
    McNeill, D.: Gesture and Thought. University of Chicago Press, Chicago (2005)CrossRefGoogle Scholar
  24. 24.
    Goldin-Meadow, S.: Hearing Gesture: How Our Hands Help Us Think. Harvard University Press, Cambridge (2003)Google Scholar
  25. 25.
    Hostetter, A.B., Alibali, M.W.: Visible Embodiment: Gestures as Simulated Sction. Psychonomic Bulletin and Review 15, 495–514 (2008)CrossRefGoogle Scholar
  26. 26.
    Barsalou, L.W.: Perceptual Symbol Systems. Behavioral and Brain Sciences 22, 577–660 (1999)Google Scholar
  27. 27.
    Garbarini, F., Adenzato, M.: At the Root of Embodied Cognition: Cognitive Science Meets Neurophysiology. Brain and Cognition 56, 100–106 (2004)CrossRefGoogle Scholar
  28. 28.
    Kita, S., Özyürek, A.: What Does Cross-linguistic Variation in Semantic Coordination of Speech and Gesture Reveal? Evidence for an Interface Representation of Spatial Thinking and Speaking. J. Memory and Language 48, 16–32 (2003)CrossRefGoogle Scholar
  29. 29.
    Goldin-Meadow, S., Beilock, S.L.: Action’s Influence on Thought: The Case of Gesture. Perspectives on Psychological Science 5, 664–674 (2010)CrossRefGoogle Scholar
  30. 30.
    Tversky, B.: Visualizing Thought. Topics in Cognitive Science 3, 499–535 (2011)CrossRefGoogle Scholar
  31. 31.
    Alaçam, Ö., Habel, C., Acartürk, C.: Investigation of Haptic Line-Graph Comprehension Through Co-Production of Gesture and Language. Tilburg Gesture Research Meeting, Tilburg (2013)Google Scholar
  32. 32.
    Quek, F., McNeill, D., Bryll, R., Duncan, S., Ma, X.F., Kirbas, C., Ansari, R.: Multimodal Human Discourse: Gesture and Speech. ACM Transactions on Computer-Human Interaction (TOCHI) 9, 171–193 (2002)CrossRefGoogle Scholar
  33. 33.
    Acartürk, C., Habel, C.: Eye Tracking in Multimodal Comprehension of Graphs. In: Proceedings of the Workshop on Technology-Enhanced Diagrams Research, vol. 887, pp. 11–25 (2012),

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Özge Alaçam
    • 1
  • Christopher Habel
    • 1
  • Cengiz Acartürk
    • 2
  1. 1.Department of InformaticsUniversity of HamburgHamburgGermany
  2. 2.Informatics InstituteMiddle East Technical UniversityAnkaraTurkey

Personalised recommendations