Visuo-haptic Perception of Objects and Scenes



Although many high-level perceptual tasks can be achieved on the basis of information encoded through one sensory modality, it is increasingly evident that the maintenance of a robust, coherent perception of the objects that surround us depends on multisensory integration. Consequently, multisensory representations of object information in memory, particularly those based on vision and touch, result in more efficient object recognition and spatial localisation. The following chapter reviews evidence on how multisensory object information can, for example, resolve problems often associated with unisensory processing such as maintaining shape constancy with changes in object viewpoint or motion, and updating spatial representations with changes in observer position. Further evidence from neuroimaging studies suggests that the perceptual processes involved in object and spatial recognition are underpinned by shared neural resources. Taken together, these studies suggest that the traditional view of sensory systems processing object information in an independent manner is breaking down such that, conversely, the wealth of evidence now lies firmly in favour of sensory systems which are highly interactive all along the information processing hierarchy, and which can modulate and affect high-level perceptual outcomes.


Object Recognition Familiar Object Observer Motion Egocentric Reference Frame Object Constancy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Amedi A, Malach R, Hendler T, Peled S, Zohary E (2001) Visuo-haptic, object-related activation in the ventral visual pathway. Nat Neurosci 4:324–330PubMedCrossRefGoogle Scholar
  2. Andresen DR, Vinberg J, Grill-Spector K (2009) The representation of object viewpoint in human visual cortex. Neuroimage 45(2):522–536PubMedCrossRefGoogle Scholar
  3. Berkeley G (1709) An essay towards a new theory of vision. Jeremy Pepyat Booksellers, Dublin, IrelandGoogle Scholar
  4. Biederman I (1987) Recognition-by-components: a theory of human image understanding. Psychol Rev 94:115–147PubMedCrossRefGoogle Scholar
  5. Biederman I, Gerhardstein PC (1993) Recognizing depth-view, rotated objects: evidence and conditions for three-dimensional viewpoint invariance. J Exp Psychol Hum Percept Perform 19:1162–1182PubMedCrossRefGoogle Scholar
  6. Biederman I, Ju G (1988) Surface versus edge-based determinants of visual recognition. Cogn Psychol 20(1):38–64PubMedCrossRefGoogle Scholar
  7. Biederman I, Rabinowitz JC, Glass AL, Stacy EW Jr (1974) On the information extracted from a glance at a scene. J Exp Psychol 103(3):597–600PubMedCrossRefGoogle Scholar
  8. Bülthoff HH, Edelman S (1992) Psychophysical support for a two-dimensional view interpolation theory of object recognition. Proc Natl Acad Sci USA 89:60–64PubMedCrossRefGoogle Scholar
  9. Chan JS, Newell FN (2008) Behavioral evidence for task-dependent “what” versus “where” processing within and across modalities. Percept Psychophys 70(1):36–49PubMedCrossRefGoogle Scholar
  10. Chan JS, Whitaker TA, Simões-Franklin C, Garavan H, Newell FN (2010) Implied haptic object motion activates visual area MT/MST. Neuroimage, 49(2):1708–1716Google Scholar
  11. Diwadkar VA, McNamara TP (1997) Viewpoint dependence in scene recognition. Psycholog Sci 8:302–307CrossRefGoogle Scholar
  12. Easton RD, Srinivas K, Greene AJ (1997) Do vision and haptics share common representations? Implicit and explicit memory within and between modalities. J Exp Psychol Learn Mem Cogn 23(1):153–163PubMedCrossRefGoogle Scholar
  13. Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433PubMedCrossRefGoogle Scholar
  14. Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8:162–169PubMedCrossRefGoogle Scholar
  15. Ernst MO, Lange C, Newell FN (2007) Multisensory recognition of actively explored objects. Can J Exp Psychol 61(3):242–253PubMedCrossRefGoogle Scholar
  16. Gibson JJ (1962) Observations on active touch. Psycholog Rev 69:477–491CrossRefGoogle Scholar
  17. Goodale MA, Milner AD (1992) Separate visual pathways for perception and action. Trends Neurosci 15:20–25PubMedCrossRefGoogle Scholar
  18. Gori M, Del Viva M, Sandini G, Burr DC (2008) Young children do not integrate visual and haptic form information. Curr Biol 18(9):694–698PubMedCrossRefGoogle Scholar
  19. Grill-Spector K, Kushnir T, Edelman S, Avidan G, Itzchak Y, Malach R (1999) Differential processing of objects under various viewing conditions in the human lateral occipital complex. Neuron 24:187–203PubMedCrossRefGoogle Scholar
  20. Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P (2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293:2425–2430PubMedCrossRefGoogle Scholar
  21. Hayward WG, Williams P (2000) Viewpoint dependence and object discriminability. Psychol Sci 11:7–12PubMedCrossRefGoogle Scholar
  22. James TW Humphrey GK Gati JS Menon RS, Goodale MA (2002) Differential effects of viewpoint on object-driven activation in dorsal and ventral streams. Neuron 35:793–801PubMedCrossRefGoogle Scholar
  23. Kaas AL, van Mier HI, Lataster J, Fingal M, Sack AT (2007) The effect of visuo-haptic congruency on haptic spatial matching. Exp Brain Res 183(1):75–85PubMedCrossRefGoogle Scholar
  24. Klatzky RL, Lederman SJ (1995) Identifying objects from a haptic glance. Percept Psychophys 57(8):1111–1123PubMedCrossRefGoogle Scholar
  25. Klatzky RL, Lederman SJ, Metzger V (1985) Identifying objects by touch: an expert system. Percept Psychophys 37:299–302PubMedCrossRefGoogle Scholar
  26. Kourtzi Z, Kanwisher N (2000) Activation in human MT/MST by static images with implied motion. J Cogn Neurosci 12(1):48–55PubMedCrossRefGoogle Scholar
  27. Lacey S, Peters A, Sathian K (2007) Cross-modal object recognition is viewpoint-independent. PLoS ONE 2(9):e890PubMedCrossRefGoogle Scholar
  28. Lacey S, Tal N, Amedi A, Sathian K (2009) A putative model of multisensory object representation. Brain Topogr 21(3–4):269–274Google Scholar
  29. Lewis TL, Maurer D (2005) Multiple sensitive periods in human visual development: evidence from visually deprived children. Dev Psychobiol 46(3):163–183PubMedCrossRefGoogle Scholar
  30. Logothetis NK, Pauls J (1995) Psychophysical and physiological evidence for viewer-centered object representations in the primate. Cereb Cortex 5(3):270–288PubMedCrossRefGoogle Scholar
  31. Malach R, Reppas JB, Benson RR, Kwong KK, Jiang H, Kennedy WA, Ledden PJ, Brady TJ, Rosen BR, Tootell RBH (1995) Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proc Natl Acad Sci USA 92:8135–8139PubMedCrossRefGoogle Scholar
  32. Marr D (1982) Vision. Freeman Publishers, San FranciscoGoogle Scholar
  33. Meltzoff AN, Borton RW (1979) Intermodal matching by human neonates. Nature 282:403–404PubMedCrossRefGoogle Scholar
  34. Mishkin M, Ungerleider LG, Macko KA (1983) Object vision and spatial vision: two cortical pathways. Trends Neurosci 6:414–417CrossRefGoogle Scholar
  35. Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 4:452–465CrossRefGoogle Scholar
  36. Newell FN (1998) Stimulus context and view dependence in object recognition. Perception 27(1):47–68PubMedCrossRefGoogle Scholar
  37. Newell FN, Ernst MO, Tjan BS, Bülthoff HH (2001) Viewpoint dependence in visual and haptic object recognition. Psychol Sci 12(1):37–42PubMedCrossRefGoogle Scholar
  38. Newell FN, Findlay JM (1997) The effect of depth rotation on object identification. Perception 26(10):1231–1257PubMedCrossRefGoogle Scholar
  39. Newell FN, Finucane CM, Lisiecka D, Pasqualotto A, Vendrell I (2010a) Active perception allows for spatial updating of object locations across modalities. SubmittedGoogle Scholar
  40. Newell, Hansen, Steven, Calvert (2010b) Tactile discrimination and localisation of novel object features activates both ventral and dorsal streams. Manuscript in preparationGoogle Scholar
  41. Newell FN, Sheppard DM, Edelman S, Shapiro KL (2005) The interaction of shape- and location-based priming in object categorisation: evidence for a hybrid "what + where" representation stage. Vision Res 45(16):2065–2080PubMedCrossRefGoogle Scholar
  42. Newell FN, Wallraven C, Huber S (2004) The role of characteristic motion in object categorization. J Vis 4(2):118–129PubMedCrossRefGoogle Scholar
  43. Newell FN, Woods AT, Mernagh M, Bülthoff HH (2005) Visual, haptic and crossmodal recognition of scenes. Exp Brain Res 161(2):233–242PubMedCrossRefGoogle Scholar
  44. Newport R, Rabb B, Jackson SR (2002) Noninformative vision improves haptic spatial perception. Curr Biol 12(19):1661–1664PubMedCrossRefGoogle Scholar
  45. Norman JF, Norman HF, Clayton AM, Lianekhammy J, Zielke G (2004) The visual and haptic perception of natural object shape. Percept Psychophys 66(2):342–351PubMedCrossRefGoogle Scholar
  46. Palmer S, Rosch E, Chase P (1981) Canonical perspective and the perception of objects. In: Long JB, Baddeley AD (eds) Attention and performance, vol. IX. Erlbaum, Hillsdale, NJ, pp 135–151Google Scholar
  47. Pasqualotto A, Finucane CM, Newell FN (2005) Visual and haptic representations of scenes are updated with observer movement. Exp Brain Res 166(3–4):481–488PubMedCrossRefGoogle Scholar
  48. Pasqualotto A, Newell FN (2007) The role of visual experience on the representation and updating of novel haptic scenes. Brain Cogn 65(2):184–194PubMedCrossRefGoogle Scholar
  49. Postma A, Zuidhoek S, Noordzij ML, Kappers AM (2008) Keep an eye on your hands: on the role of visual mechanisms in processing of haptic space. Cogn Proc 9(1):63–68CrossRefGoogle Scholar
  50. Postma A, Zuidhoek S, Noordzij ML, Kappers AM (2008) Haptic orientation perception benefits from visual experience: evidence from early-blind, late-blind, and sighted people. Percept Psychophys 70(7):1197–1206PubMedCrossRefGoogle Scholar
  51. Quiroga RQ, Reddy L, Kreiman G, Koch C, Fried I (2005) Invariant visual representation by single neurons in the human brain. Nature 435:1102–1107PubMedCrossRefGoogle Scholar
  52. Romanski LM, Tian B, Fritz J, Mishkin M, Goldman-Rakic PS, Rauschecker JP (1999) Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex. Nat Neurosci 2:1131–1136PubMedCrossRefGoogle Scholar
  53. Reales JM, Ballestersos S (1999) Implicit and explicit memory for visual and haptic objects: cross-modal priming depends on structural descriptions. J Exp Psychol Learn Mem Cogn 25_(3)644–663CrossRefGoogle Scholar
  54. Reddy L, Kanwisher N (2006) Coding of visual objects in the ventral stream. Curr Opin Neurobiol 16:408–414PubMedCrossRefGoogle Scholar
  55. Reed CL, Klatzky RL, Halgren E (2005) What vs. where in touch: an fMRI study. Neuroimage 25:718–726PubMedCrossRefGoogle Scholar
  56. Rousselet GA, Fabre-Thorpe M, Thorpe SJ (2002) Parallel processing in high-level categorization of natural images. Nat Neurosci 5(7):629–630PubMedGoogle Scholar
  57. Sann C, Streri A (2007) Perception of object shape and texture in human newborns: evidence from cross-modal transfer tasks. Dev Sci 10(3):399–410PubMedCrossRefGoogle Scholar
  58. Schendan HE, Stern CE (2007) Mental rotation and object categorization share a common network of prefrontal and dorsal and ventral regions of posterior cortex. Neuroimage 35(3):1264–1277PubMedCrossRefGoogle Scholar
  59. Schendan HE, Stern CE (2008) Where vision meets memory: prefrontal-posterior networks for visual object constancy during categorization and recognition. Cereb Cortex 18(7):1695–1711PubMedCrossRefGoogle Scholar
  60. Setti A, Newell FN (2009) The effect of body and part-based motion on the recognition of unfamiliar objects. Vis Cogn 18(3):456–480Google Scholar
  61. Simons DJ, Wang RF, Roddenberry D (2002) Object recognition is mediated by extraretinal information. Percept Psychophys 64(4):521–530PubMedCrossRefGoogle Scholar
  62. Stone JV (1999) Object recognition: view-specificity and motion-specificity. Vis Res 39(24):4032–4044PubMedCrossRefGoogle Scholar
  63. Stone JV (1998) Object recognition using spatiotemporal signatures. Vis Res 38(7):947–951PubMedCrossRefGoogle Scholar
  64. Tarr MJ (1995) Rotating objects to recognize them: a case study of the role of viewpoint dependency in the recognition of three-dimensional objects. Psychon Bull Rev 2:55–82CrossRefGoogle Scholar
  65. Tarr MJ, Bülthoff HH (1998) Image-based object recognition in man, monkey and machine. Cognition 67:1–20PubMedCrossRefGoogle Scholar
  66. Thinus-Blanc C, Gaunet F (1997) Representation of space in blind persons: vision as a spatial sense? Psychol Bull 121(1):20–42PubMedCrossRefGoogle Scholar
  67. Tootell RB, Reppas JB, Kwong KK, Malach R, Born RT, Brady TJ, Rosen BR, Belliveau JW (1995) Functional analysis of human MT and related visual cortical areas using magnetic resonance imaging. J Neurosci 15(4):3215–3230PubMedGoogle Scholar
  68. Ungerleider LG, Haxby JV (1994) ‘What’ and ‘where’ in the human brain. Curr Opin Neurobiol 4, 157–165PubMedCrossRefGoogle Scholar
  69. Ullman S (1998) Three-dimensional object recognition based on the combination of views. Cogn 67:21–44CrossRefGoogle Scholar
  70. Van Boven RW, Ingeholm JE, Beauchamp MS, Bikle PC, Ungerleider LG (2005) Tactile form and location processing in the human brain. Proc Natl Acad Sci U S A 102(35):12601–12605PubMedCrossRefGoogle Scholar
  71. Volcic R, van Rheede JJ, Postma A, Kappers AM (2008) Differential effects of non-informative vision and visual interference on haptic spatial processing. Exp Brain Res 190(1):31–41PubMedCrossRefGoogle Scholar
  72. Vuong QC, Tarr MJ (2004) Rotation direction affects object recognition. Vis Res 44(14):1717–1730PubMedCrossRefGoogle Scholar
  73. Welch RB, Warren DH (1986) Intersensory interactions. In: Boff KR, Kaufman L, Thomas JP (eds) Handbook of perception and performance, vol. 1. Sensory processes and perception. John Wiley and Sons, New York, pp 25–1–25–36Google Scholar
  74. Whitaker TA, Chan J, Newell FN (2010) The role of characteristic motion in haptic and visuo-haptic object recognition. Manuscript in preparationGoogle Scholar
  75. Wang RF, Simons DJ (1999) Active and passive scene recognition across views. Cognition 70(2):191–210PubMedCrossRefGoogle Scholar
  76. Woods AT, Moore A, Newell FN (2009) Canonical views in haptic object perception. Perception 37(12):1867–1878CrossRefGoogle Scholar
  77. Young MP (1992) Objective analysis of the topological organization of the primate cortical visual system. Nature 358(6382):152–155PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science + Business Media, LLC 2010

Authors and Affiliations

  1. 1.School of Psychology and Institute of Neuroscience, Lloyd Building, Trinity CollegeDublin 2Ireland

Personalised recommendations