Interactive Multisensory Data Representation

  • Patricia SearchEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9187)


Large data sets require new forms of data representation that reduce the complexity of the information, and help users identify trends and communicate the meaning of the data to diverse audiences. With multisensory data design, it is possible to increase the number of variables and relationships that can be represented simultaneously. Sound, touch, gesture, and movement can enhance the perception of data relationships. Complex data sets also require new ways of organizing databases that encourage the development of new perspectives and facilitate collaboration. Audiovisual metadata is an alternative to text-based metadata that supports data exploration by providing a flexible format for database organization. With these new approaches to data representation, it is important to understand the semiotics of multisensory data design.


Data visualization Multisensory data representation Data sonification Audiovisual metadata 


  1. 1.
    Wall, M.: Big Data: Are You Ready for the Blast-off? BBC NEWS Business, 3 March 2014. Retrieved from
  2. 2.
    King, G.: Why Big Data is a Big Deal. Harvard Magazine, November-December 2014. Retrieved from
  3. 3.
    Tak, S., Toet, L.: Towards interactive multisensory data representations. In: Proceedings of the International Conference on Computer Graphics Theory and Applications and International Conference Visualization Theory and Applications, pp. 558–561. SciTePress – Science and Technology Publications, Lisbon (2013)Google Scholar
  4. 4.
    Loftin, R.B.: Multisensory perception: beyond the visual in visualization. Comput. Sci. Eng. 5(4), 56–58 (2003)CrossRefGoogle Scholar
  5. 5.
    Search, P.: The metastructural dynamics of interactive electronic design. Vis. Lang. Cult. Dimensions Vis. Commun. 37(2), 146–165 (2003)Google Scholar
  6. 6.
    Downie, M., Carothers, C., Goebel, J.: A New Paradigm for Interactive Exploration of Data with Live Coding (2013). Retrieved from
  7. 7.
    Stanford University: HANA Immersive Visualization Environment (HIVE) (2014). Retrieved from
  8. 8.
    Manovich, L.: Media Visualization: Visual Techniques for Exploring Large Media Collections (2012). Retrieved from
  9. 9.
    Software Studies Initiative: Cultural Analytics 2014 (2014). Retrieved from
  10. 10.
    Ngo, M.K., Spence, C.: Auditory, tactile, and multisensory cues facilitate search for dynamic visual stimuli. Attention, Percept. Psychophysics 72(6), 1654–1664 (2010)CrossRefGoogle Scholar
  11. 11.
    Search, P.: Kaleidoscope: the dynamic discourse of visual literacy in experience design. In: Avgerinou, M., Griffin, R., Giesen, J., Search, P., Spinillo. C. (eds.) Visual Literacy Beyond Frontiers: Information, Culture and Diversity, pp. 185–192. International Visual Literacy Association, Loretto (2008)Google Scholar
  12. 12.
    Macken-Horarik, M.: Interacting with the multimodal text: reflections on image and verbiage. Vis. Commun. 3(1), 5–26 (2004)CrossRefGoogle Scholar
  13. 13.
    Search, P.: Defining a sense of place in interactive multimedia design. In: Avgerinou, A., Search, P., Chandler, S. (eds.) Visual Literacy in the 21st Century: Trends, Demands, and Capacities, pp. 143–148. International Visual Literacy Association, Chicago (2011)Google Scholar
  14. 14.
    Search, P.: The dynamic aesthetics of experience design. In: Proceedings of the International Association of Empirical Aesthetics Congress, pp. 142–145. International Association of Empirical Aesthetics, Chicago (2008)Google Scholar
  15. 15.
    Search, P.: The spatial grammar of interaction design: weaving a tapestry of space and time in multimedia computing. In: Griffin, R., Cowden, B., Avgerinou, M. (eds.) Animating the Mind’s Eye, pp. 185–190. International Visual Literacy Association, Loretto (2006)Google Scholar
  16. 16.
    Berkeley, G.: A New Theory of Vision and Other Writings. E. P. Dutton, New York (1922)Google Scholar
  17. 17.
    Piaget, J., Inhelder, B.: The Child’s Concept of Space. Routledge and Kegan Paul, London (1956)Google Scholar
  18. 18.
    Gibson, J.: The Perception of the Visual World. Houghton Mifflin, Boston (1950)Google Scholar
  19. 19.
    Rock, I.: The perception of disoriented figures. In: Held, R. (ed.) Image, Object, and Illusion, pp. 71–78. W. H. Freeman and Company, San Francisco (1974)Google Scholar
  20. 20.
    Gaines, E.: Communication and the semiotics of space. J. Creative Commun. 1(2), 173–181 (2006)CrossRefGoogle Scholar
  21. 21.
    Djajadiningrat, J., Matthews, B., Stienstra, M.: Easy doesn’t do it: skill and expression in tangible aesthetics. Pers. Ubiquit. Comput. 11(8), 657–676 (2007)CrossRefGoogle Scholar
  22. 22.
    Palmerius, K.L., Forsell, C.: The impact of feedback design in haptic volume visualization. In: Third Joint EuroHaptics Conference 2009 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, World Haptics 2009, pp. 154–159. IEEE Press, New York (2009)Google Scholar
  23. 23.
    Christie, M.: Aboriginal knowledge traditions in digital environments. J. Indigenous Educ. 34, 61–66 (2005)Google Scholar
  24. 24.
    Verran, H., Christie, M.: Using/designing digital technologies of representation in aboriginal australian knowledge practices. Hum. Technol. 3(2), 214–227 (2007)Google Scholar
  25. 25.
    Srinivasan, R., Huang, J.: Fluid ontologies for digital museums. Int. J. Digit. Libr. 5(3), 193–204 (2005)CrossRefGoogle Scholar
  26. 26.
    Dunsire, G., Hillmann, D., Phipps, J., Coyle, K.: A reconsideration of mapping in a semantic world. In: Proceedings of the International Conference on Dublin Core and Metadata Applications, pp. 26–36. Dublin Core Metadata Initiative (2011)Google Scholar
  27. 27.
    School of Australian Indigenous Knowledge Systems (Charles Darwin University): Indigenous Knowledge and Resource Management in Northern Australia (IKRMNA) (2005). Retrieved from
  28. 28.
    Hunter, J.: Next generation metadata tools: supporting dynamic knowledge spaces. In: Kapitzke, C., Bruce, B.C. (eds.) Libr@ries: Changing Information Space and Practice, pp. 91–122. Lawrence Erlbaum Associates, Mahwah (2006)Google Scholar
  29. 29.
    Biocca, F.: The space of cognitive technology: the design medium and cognitive properties of virtual space. In: Beynon, M., Nehaniv, C.L., Dautenhahn, K. (eds.) CT 2001. LNCS (LNAI), vol. 2117, pp. 55–56. Springer, Heidelberg (2001)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Rensselaer Polytechnic InstituteTroyUSA

Personalised recommendations