Gestural Attributions as Semantics in User Interface Sound Design

  • Kai Tuuri
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5934)


This paper proposes a gesture-based approach to user interface sound design, which utilises projections of body movements in sounds as meaningful attributions. The approach is founded on embodied conceptualisation of human cognition and it is justified through a literature review on the subject of interpersonal action understanding. According to the resulting hypothesis, stereotypical gestural cues, which correlate with, e.g., a certain communicative intention, represent specific non-linguistic meanings. Based on this theoretical framework, a model of a process is also outlined where stereotypical gestural cues are implemented in sound design.


gestures user interfaces sound design semantics 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Mott, R.L.: Sound effects: Radio, TV and Film. Focal Press, Boston (1990)Google Scholar
  2. 2.
    Leman, M.: Embodied Music Cognition and Mediation Technology. MIT Press, Cambridge (2008)Google Scholar
  3. 3.
    Shannon, C.E., Weaver, W.: The Mathematical Theory of Communication. The University of Illinois Press, Urbana (1949)zbMATHGoogle Scholar
  4. 4.
    Czaja, S.J.: Systems design and evaluation. In: Salvendy, G. (ed.) Handbook of human factors and ergonomics, 2nd edn., pp. 17–40. Wiley, New York (1997)Google Scholar
  5. 5.
    Bannon, L.J.: A human-centred perspective on interaction design. In: Pirhonen, A., Isomäki, H., Roast, C., Saariluoma, P. (eds.) Future interaction design, pp. 31–51. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  6. 6.
    Brewster, S., Wright, P., Edwards, A.: A detailed investigation into the effectiveness of earcons. In: Kramer, G. (ed.) Auditory display, pp. 471–498. Addison-Wesley, Reading (1994)Google Scholar
  7. 7.
    Rocchesso, D., Serafin, S., Behrendt, F., Bernardini, N., Bresin, R., Eckel, G., Franinovic, K., Hermann, T., Pauletto, S., Susini, P., Visell, Y.: Sonic interaction design: sound, information and experience. In: CHI 2008 extended abstracts on Human factors in computing systems. ACM Press, New York (2008)Google Scholar
  8. 8.
    Rizzolatti, G., Arbib, M.A.: Language within our grasp. Trends in Neurosciences 21, 188–194 (1998)CrossRefGoogle Scholar
  9. 9.
    Svanaes, D.: Kinaesthetic thinking. Computers in Human Behavior 13(4), 443–463 (1997)CrossRefGoogle Scholar
  10. 10.
    Gibson, J.J.: The ecological approach to visual perception. Houghton Mifflin, Boston (1979)Google Scholar
  11. 11.
    Varela, F., Thompson, E., Rosch, E.: The Embodied Mind. MIT Press, Cambridge (1991)Google Scholar
  12. 12.
    Rizzolatti, G., Craighero, L.: The mirror-neuron system. Annual Review of Neuroscience 27, 169–192 (2004)CrossRefGoogle Scholar
  13. 13.
    Gallese, V., Eagle, M.N., Migone, P.: Intentional Attunement: Mirror Neurons and the Neural Underpinnings of Interpersonal Relations. J. Am. Psychoanal. Assoc. 55(1), 131–175 (2007)Google Scholar
  14. 14.
    Kohler, E., Keysers, C., Umiltá, M.A., Fogassi, L., Gallese, V., Rizzolatti, G.: Hearing sounds, understanding actions: action representation in mirror neurons. Science 297, 846–848 (2002)CrossRefGoogle Scholar
  15. 15.
    Merleau-Ponty, M.: Phenomenology of Perception. Routledge, London (1945/1996)Google Scholar
  16. 16.
    Johnson, M.: The Body in the Mind: The Bodily Basis of Meaning, Imagination, and Reason. University of Chicago, Chicago (1987)Google Scholar
  17. 17.
    Almäng, J.: Intentionality and intersubjectivity. Ph.D. thesis, Goteborg. Goteborg Universitet, Sweden (2007)Google Scholar
  18. 18.
    Scherer, K.R., Bänziger, T.: Emotional expression in prosody: a review and an agenda for future research. In: Proc. of SP 2004, Nara, Japan, pp. 359–366 (2004)Google Scholar
  19. 19.
    Brunswik, E.: Perception and the representative design of psychological experiments, 2nd edn. University of California Press, Berkeley (1956)Google Scholar
  20. 20.
    Searle, J.R.: Mind, language and society. Basic Books, New York (1998)Google Scholar
  21. 21.
    Juslin, P.N., Laukka, P.: Communication of emotions in vocal expression and music performance: Different channels, same code? Psych. Bull. 129(5), 770–814 (2003)CrossRefGoogle Scholar
  22. 22.
    Tuuri, K., Pirhonen, A., Välkkynen, P.: Bodily Engagement in Multimodal Interaction: A Basis for a New Design Paradigm? In: Kurkovsky, S. (ed.) Multimodality in Mobile Computing and Mobile Devices. IGI Global, Hershey (2009)Google Scholar
  23. 23.
    Godøy, R.I.: Gestural-sonorous objects: embodied extensions of Schaeffer’s conceptual apparatus. Organised Sound 11(02), 149–157 (2006)CrossRefGoogle Scholar
  24. 24.
    Avanzini, F., Serafin, S., Rocchesso, D.: Friction sounds for sensory substitution. In: Proc. of ICAD 2004, Sydney, Australia (2004)Google Scholar
  25. 25.
    Vitale, R., Bresin, R.: Emotional Cues in Knocking Sounds. In: Proc. of ICMPC 10 (abstract only), Sapporo, Japan, p. 276 (2008)Google Scholar
  26. 26.
    Fernald, A.: Human maternal vocalizations to infants as biologically relevant signals: An evolutionary perspective. In: Barkow, J., Cosmides, L., Tooby, J. (eds.) The Adapted Mind: Evolutionary Psychology and the Generation of Culture, pp. 391–428. Oxford University Press, Oxford (1992)Google Scholar
  27. 27.
    Tuuri, K., Eerola, T.: Could function-specific prosodic cues be used as a basis for non-speech user interface sound design? In: Proc. of ICAD 2008, Paris, France (2008)Google Scholar
  28. 28.
    Feyereisen, P., de Lannoy, J.-D.: Gestures and speech: Psychological investigations. Cambridge University Press, New York (1991)Google Scholar
  29. 29.
    Tuuri, K., Pirhonen, A., Eerola, T.: Design and Evaluation of Prosody Based Non-Speech Audio Feedback for Physical Training Application (journal submission) Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Kai Tuuri
    • 1
  1. 1.Department of Computer Science and Information SystemsUniversity of JyväskyläFinland

Personalised recommendations