Encyclopedia of Computational Neuroscience

Living Edition
| Editors: Dieter Jaeger, Ranu Jung

3D Shape Perception, Models of

  • Benjamin KunsbergEmail author
  • Steven W. Zucker
Living reference work entry
DOI: https://doi.org/10.1007/978-1-4614-7320-6_100661-1


A 3D shape percept is the psychophysical response induced from a visual stimulus. The percept is the result of (partially) unknown neural processes within the visual system. Psychophysical tests of performance have mainly involved humans, and most relevant neurophysiological research (to date) has been performed on primates. The challenges are that (i) many different visual areas can be involved in producing a shape percept; and (ii) a single image rarely constrains the potential shape possibilities to a unique percept. Therefore, the visual system (and our computational models) must exploit assumptions to restrict the range and type of solutions. Since most models seek to return a single and, to some extent, veridical 3D shape percept, understanding both the assumptions needed and the mechanisms of computation are key components of research.

Detailed Description

An Ill-posed Inverse Problem

When a 3D object must be recovered from a single image of the object, the inverse...
This is a preview of subscription content, log in to check access.


  1. Adato Y, Vasilyev Y, Zickler T, Ben-Shahar O (2010) Shape from specular flow. IEEE Trans Pattern Anal Mach Intell 32(11):2054–2070CrossRefGoogle Scholar
  2. Agin GJ, Binford TO (1976) Computer description of curved objects. IEEE Trans Comput 4:439–449CrossRefGoogle Scholar
  3. Barron JT, Malik J (2012) Shape, albedo, and illumination from a single image of an unknown object. In: 2012 IEEE conference on computer vision and pattern recognition (CVPR), Providence, pp 334–341.  https://doi.org/10.1109/CVPR.2012.6247693
  4. Barron JL, Fleet DJ, Beauchemin SS (1994) Performance of optical flow techniques. Int J Comput Vis 12(1):43–77CrossRefGoogle Scholar
  5. Belhumeur P, Kriegman D (1998) What is the set of images of an object under all possible illumination conditions? Int J Comput Vis 28:1–16CrossRefGoogle Scholar
  6. Biederman I (1987) Recognition-by-components: a theory of human image understanding. Psychol Rev 94:115–147CrossRefGoogle Scholar
  7. Binford TO (1971) Visual perception by computer. In: Proceedings of the IEEE conference on systems and control, MiamiGoogle Scholar
  8. Blum H (1973) Biological shape and visual science (part I). J Theor Biol 38(2):205–287CrossRefGoogle Scholar
  9. Breton P, Zucker SW (1996) Shadows and shading flow fields. In: IEEE computer society conference on computer vision and pattern recognition, 1996. Proceedings CVPR’96, San Francisco, 1996, IEEE, pp 782–789Google Scholar
  10. Britten KH, Shadlen MN, Newsome WT, Movshon JA (1992) The analysis of visual motion: a comparison of neuronal and psychophysical performance. J Neurosci 12(12):4745–4765CrossRefGoogle Scholar
  11. Brooks FP Jr (1995) The mythical man-month (anniversary ed.). Addison-Wesley Longman Publishing Co., BostonGoogle Scholar
  12. Bülthoff HH, Edelman S (1992) Psychophysical support for a two-dimensional view interpolation theory of object recognition. Proc Natl Acad Sci 89(1):60–64CrossRefGoogle Scholar
  13. Collett M (2010) How desert ants use a visual landmark for guidance along a habitual route. Proc Natl Acad Sci 107(25):11638–11643CrossRefGoogle Scholar
  14. Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 6:681–685CrossRefGoogle Scholar
  15. Cormack LK, Czuba TB, Knll J, Huk AC (2017) Binocular mechanisms of 3d motion processing. Annu Rev Vis Sci 3(1):297–318. PMID: 28746813CrossRefGoogle Scholar
  16. Cutting JE (1986) Perception with an eye for motion, vol 1. Mit Press, Cambridge, MAGoogle Scholar
  17. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Proceedings of the 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), vol 1. IEEE Computer Society, Washington, DC, pp 886–893. ISBN:0-7695-2372-2.  https://doi.org/10.1109/CVPR.2005.177
  18. Deift P, Sylvester J (1981) Some remarks on the shape-from-shading problem in computer vision. J Math Anal Appl 84(1):235–248CrossRefGoogle Scholar
  19. Dickinson SJ, Pizlo Z (2013) Shape perception in human and computer vision: an interdisciplinary perspective. Springer Publishing Company, Incorporated. ISBN:1447151941Google Scholar
  20. Dickinson SJ, Pentland A, Rosenfeld A (1992) 3-d shape recovery using distributed aspect matching. IEEE Trans Pattern Anal Mach Intell 14(2):174–198CrossRefGoogle Scholar
  21. Eigen D, Puhrsch C, Fergus R (2014) Depth map prediction from a single image using a multi-scale deep network. In: Proceedings of the 27th international conference on neural information processing systems – vol 2, NIPS’14. MIT Press, Cambridge, MA, pp 2366–2374Google Scholar
  22. Faisman A, Langer MS (2013) Qualitative shape from shading, highlights, and mirror reflections. J Vis 13(5):10CrossRefGoogle Scholar
  23. Farley Norman J, Todd JT, Phillips F (1995) The perception of surface orientation from multiple sources of optical information. Percept Psychophys 57(5):629–636CrossRefGoogle Scholar
  24. Fleming RW, Torralba A, Adelson EH (2004) Specular reflections and the perception of shape. J Vis 4(9):10CrossRefGoogle Scholar
  25. Fleming R, Rice D-H, Bulthoff H (2011) Estimation of 3D shape from image orientations. Proc Natl Acad Sci 108(51):20438–20443.  https://doi.org/10.1073/pnas.1114619109
  26. Geman S, Graffigne C (1986) Markov random field image models and their applications to computer vision. In: Proceedings of the international congress of mathematicians, BerkeleyGoogle Scholar
  27. Gibson JJ (1950) The perception of the visual world. Houghton Mifflin, OxfordGoogle Scholar
  28. Gibson JJ (1966) The senses considered as perceptual systems. Houghton Mifflin, BostonGoogle Scholar
  29. Gibson JJ (1979) The ecological approach to visual perception. Houghton Mifflin, BostonGoogle Scholar
  30. Gibson JJ, Cornsweet J (1952) The perceived slant of visual surfaces: optical and geographical. J Exp Psychol 44(1):11CrossRefGoogle Scholar
  31. Guzmán A (1968) Decomposition of a visual scene into three-dimensional bodies. In: Proceedings of the December 9–11, 1968, fall joint computer conference, part I, New York, AFIPS ‘68 (Fall, part I). ACM, pp 291–304Google Scholar
  32. Hoffman DD, Singh M, Prakash C (2015) The interface theory of perception. Psychon Bull Rev 22(6):1480–1506CrossRefGoogle Scholar
  33. Horn BKP (1970) Shape from shading: a method for obtaining a smooth opaque object from one view. PhD thesisGoogle Scholar
  34. Horn B (1986) Robot vision. The MIT Press, Cambridge, MAGoogle Scholar
  35. Horn B, Brooks M (1989) Shape from shading. The MIT Press, Cambridge, MAGoogle Scholar
  36. Howard IP, Rogers BJ (1995) Binocular vision and stereopsis. Oxford University Press, OxfordGoogle Scholar
  37. Hubel DH (1988) Eye, brain, and vision, vol 22. Scientific American Library, distributed by W. H. Freeman & Co., New York, 240ppGoogle Scholar
  38. Johansson G (1973) Visual perception of biological motion and a model for its analysis. Percept Psychophys 14(2):201–211CrossRefGoogle Scholar
  39. Khang B-G, Koenderink JJ, Kappers AML (2007) Shape from shading from images rendered with various surface types and light fields. Perception 36(8):1191–1213. PMID: 17972483CrossRefGoogle Scholar
  40. Kim J, Marlow PJ, Anderson BL (2014) Texture-shading flow interactions and perceived reflectance. J Vis 14(7):1CrossRefGoogle Scholar
  41. Kimia BB (2003) On the role of medial geometry in human vision. J Physiol Paris 97(2–3):155–190CrossRefGoogle Scholar
  42. Knill DC, Richards W (1996) Perception as Bayesian inference. Cambridge University Press, New YorkGoogle Scholar
  43. Koenderink JJ (1984) What does the occluding contour tell us about solid shape? Perception 13(3):321–330CrossRefGoogle Scholar
  44. Koenderink J (2011) Vision as a user interfaceGoogle Scholar
  45. Koenderink JJ, Van Doorn AJ (1980a) Photometric invariants related to solid shape. J Mod Opt 27(7):981–996Google Scholar
  46. Koenderink JJ, Van Doorn AJ (1980b) Photometric invariants related to solid shape. Opt Acta: Int J Optics 27(7):981–996CrossRefGoogle Scholar
  47. Koenderink JJ, Van Doorn AJ, Kappers AM (1992) Surface perception in pictures. Percept Psychophys 52(5):487–496CrossRefGoogle Scholar
  48. Koenderink J, van Doorn A, Wagemans J (2015) Part and whole in pictorial relief. Iperception 6(6):2041669515615713.  https://doi.org/10.1177/2041669515615713
  49. Koffka K (1955) Principles of Gestalt psychology. International library of psychology, philosophy, and scientific method. Routledge & K. Paul. https://books.google.com/books?id=KJkAAAAAMAAJ
  50. Kolers PA (2013) Aspects of motion perception: international series of monographs in experimental psychology, vol 16. Elsevier. https://www.elsevier.com/books/aspects-of-motion-perception/kolers/978-0-08-016843-2
  51. Kunsberg B, Zucker SW (2018) Critical contours: an invariant linking image flow with salient surface organization. SIAM J Imag Sci 11(3):1849–1877CrossRefGoogle Scholar
  52. Langer MS, Zucker SW (1994) Shape-from-shading on a cloudy day. J Opt Soc Am A 11(2):467–478CrossRefGoogle Scholar
  53. Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436CrossRefGoogle Scholar
  54. Li G, Zucker S (2010) Differential geometric inference in surface stereo. IEEE Trans Pattern Anal Mach Intell 32:72–86CrossRefGoogle Scholar
  55. Liu F, Shen C, Lin G, Reid ID (2015) Learning depth from single monocular images using deep convolutional neural fields. CoRR abs/150207411. http://arxiv.org/abs/1502.07411, https://dblp.org/rec/bib/journals/corr/LiuSLR15
  56. Logothetis NK, Sheinberg DL (1996) Visual object recognition. Annu Rev Neurosci 19(1):577–621. PMID: 8833455CrossRefGoogle Scholar
  57. Logothetis NK, Pauls J, Poggio T (1995) Shape representation in the inferior temporal cortex of monkeys. Curr Biol 5(5):552–563CrossRefGoogle Scholar
  58. Longuet-Higgins HC (1981) A computer algorithm for reconstructing a scene from two projections. Nat Neurosci 293:133–135Google Scholar
  59. Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the international conference on computer vision, vol 2. ICCV'99. IEEE Computer Society, Washington, DC, p 1150. ISBN:0-7695-0164-8, http://dl.acm.org/citation.cfm?id=850924.851523
  60. Mach E (1965) On the physiological effect of spatially distributed light stimuli. Translation in Ratliff F, Mach Bands: Quantitative studies on neural networks in the retinaGoogle Scholar
  61. Marr D (1982) Vision: a computational investigation into the human representation and processing of visual information. Henry Holt and Co., New YorkGoogle Scholar
  62. Metzger W, Senckenbergische Naturforschende Gesellschaft (1936) Gesetze des Sehens. Frankfurt am Main: W. Kramer & Co.Google Scholar
  63. Mooney S, Anderson B (2014) Specular image structure modulates the perception of three-dimensional shape. Curr Biol 24(22):2737–2742CrossRefGoogle Scholar
  64. Norman D (2002) The design of everyday things. Basic Books, Inc., New York. ISBN:9780465067107Google Scholar
  65. Pizlo Z (2008) 3D shape: its unique place in visual perception. The MIT Press. http://mitpress.mit.edu/catalog/item/default.asp?ttype=2\&\#38;tid=12396Google Scholar
  66. Pizlo Z, Li Y, Sawada T, Steinman RM (2014) Making a machine that sees like us. Oxford University PressGoogle Scholar
  67. Poggio T, Edelman S (1990) A network that learns to recognize three-dimensional objects. Nature 343:236–266CrossRefGoogle Scholar
  68. Poggio T, Torre V, Koch C (1985) Computational vision and regularization theory. Nature 317:314–319CrossRefGoogle Scholar
  69. Pylyshyn ZW (1986) Computation and cognition: toward a foundation for cognitive science. The MIT Press, Cambridge, MAGoogle Scholar
  70. Ramachandran VS (1988) Perception of shape from shading. Nature 331(6152):163CrossRefGoogle Scholar
  71. Reichardt W (1987) Evaluation of optical motion information by movement detectors. J Comp Physiol A 161(4):533–547CrossRefGoogle Scholar
  72. Rosenfeld A, Kak AC (1982) Digital image processing, vol 2, 2nd edn. Academic, New YorkGoogle Scholar
  73. Saxena A, Chung SH, Ng AY (2005) Learning depth from single monocular images. In: NIPS 18. MIT PressGoogle Scholar
  74. Scharstein D, Szeliski R (2002) A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int J Comput Vis 47(1):7–42CrossRefGoogle Scholar
  75. Seitz SM, Curless B, Diebel J, Scharstein D, Szeliski R (2006) A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proceedings of the 2006 I.E. computer society conference on computer vision and pattern recognition – vol 1, CVPR ‘06. IEEE Computer Society, Washington, DC, pp 519–528Google Scholar
  76. Serre T, Wolf L, Bileschi S, Riesenhuber M, Poggio T (2007) Robust object recognition with cortex-like mechanisms. IEEE Trans Pattern Anal Mach Intell 3:411–426CrossRefGoogle Scholar
  77. Shepard RN, Metzler J (1971) Mental rotation of three-dimensional objects. Science 171(3972):701–703CrossRefGoogle Scholar
  78. Siddiqi K, Pizer S (2008) Medial representations: mathematics, algorithms and applications, 1st edn. Springer Publishing Company, Incorporated. ISBN:1402086571Google Scholar
  79. Stevens KA (1983) Slant-tilt: the visual encoding of surface orientation. Biol Cybern 46(3):183–195CrossRefGoogle Scholar
  80. Sun P, Schofield AJ (2012) Two operational modes in the perception of shape from shading revealed by the effects of edge information in slant settings. J Vis 12(1):12CrossRefGoogle Scholar
  81. Tanaka K (1996) Inferotemporal cortex and object vision. Annu Rev Neurosci 19(1):109–139CrossRefGoogle Scholar
  82. Todd JT, Norman JF (2018) The visual perception of metal. J Vis 18:3–9CrossRefGoogle Scholar
  83. Tomasi C, Kanade T (1992) Shape and motion from image streams under orthography: a factorization method. Int J Comput Vis 9(2):137–154CrossRefGoogle Scholar
  84. Ullman S (1979) The interpretation of structure from motion. Proc R Soc Lond B: Biol Sci 203(1153):405–426.  https://doi.org/10.1098/rspb.1979.0006
  85. Ullman S (1980) Against direct perception. Behav Brain Sci 3(3):373–381CrossRefGoogle Scholar
  86. Ullman S (1989) Aligning pictorial descriptions: an approach to object recognition. Cognition 32(3):193–254CrossRefGoogle Scholar
  87. Vergne R, Barla P, Fleming R, Granier X (2012) Surface flows for image-based shading design. ACM Trans Graph 31(3):1–9CrossRefGoogle Scholar
  88. von Helmholtz H (1962) Helmholtz’s treatise on physiological optics. Dover Publications, New York. http://www.worldcat.org/title/helmholtzs-treatise-on-physiological-optics/oclc/8110065
  89. Waltz D (1975) Understanding line drawings of scenes with shadows. In: The psychology of computer vision. McGraw-HillGoogle Scholar
  90. Warren WH (2012) Does this computational theory solve the right problem? Marr, Gibson, and the goal of vision. Perception 41(9):1053–1060. PMID: 23409371CrossRefGoogle Scholar
  91. Witkin A (1980) Shape from contour. PhD thesisGoogle Scholar
  92. Witkin A (1981) Recovering surface shape and orientation from texture. Artif Intell 17(1–3):17–45CrossRefGoogle Scholar
  93. Wu J, Wang Y, Xue T, Sun X, Freeman WT, Tenenbaum JB (2017) MarrNet: 3D shape reconstruction via 2.5D sketches. CoRR. http://arxiv.org/abs/1711.03129, https://dblp.org/rec/bib/journals/corr/abs-1711-03129
  94. Yamane Y, Carlson ET, Bowman KC, Wang Z, Connor CE (2008) A neural code for three-dimensional object shape in macaque inferotemporal cortex. Nat Neurosci 11:1352–1360.  https://doi.org/10.1038/nn.2202
  95. Yamins DL, Dicarlo JJ (2016) Using goal-driven deep learning models to understand sensory cortex. Nat Neurosci 19(3):356CrossRefGoogle Scholar
  96. Yuille A, Kersten D (2006) Vision as bayesian inference: analysis by synthesis? Trends Cogn Sci 10(7):301–308CrossRefGoogle Scholar
  97. Zhang R, Tsai P, Cryer JE, Shah M (1999) Shape from shading: a survey. IEEE Trans Pattern Anal Mach Intell 21:690–706CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Division of Applied MathematicsBrown UniversityProvidenceUSA
  2. 2.Department of Computer ScienceYale UniversityNew HavenUSA

Section editors and affiliations

  • Thomas Serre
    • 1
  1. 1.Institute for Brain Sciences, Brown UniversityProvidenceUSA