Abstract
I have been asked to review the progress that computational neuroscience has made over the past 20 years in understanding how vision works. In reflecting on this question, I come to the conclusion that perhaps the most important advance we have made is in gaining a deeper appreciation of the magnitude of the problem before us. While there has been steady progress in our understanding—and I will review some highlights here—we are still confronted with profound mysteries about how visual systems work. These are not just mysteries about biology, but also about the general principles that enable vision in any system whether it be biological or machine. I devote much of this chapter to examining these open questions, as they are crucial in guiding and motivating current efforts. Finally, I shall argue that the biggest mysteries are likely to be ones we are not currently aware of, and that bearing this in mind is important as it encourages a more exploratory, as opposed to strictly hypothesis-driven, approach.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adelson EH (1993) Perceptual organization and the judgment of brightness. Science 262(5142):2042–2044
Adelson EH (2000) Lightness perception and lightness illusions. In: Gazzaniga M (ed) The new cognitive neurosciences, 2nd edn. MIT, Cambridge, MA, pp 339–351
Andolina IM, Jones HE et al (2007) Corticothalamic feedback enhances stimulus response precision in the visual system. Proc Natl Acad Sci USA 104(5):1685–1690
Angelucci A, Bullier J (2003) Reaching beyond the classical receptive field of V1 neurons: horizontal or feedback axons? J Physiol Paris 97(2–3):141–154
Arathorn DW (2005) Computation in the higher visual cortices: map-seeking circuit theory and application to machine vision. In: Proceedings of the 33rd applied imagery pattern recognition workshop (AIPR 2004), Washington, DC, 1–6
Arathorn DW, Yang Q et al (2007) Retinally stabilized cone-targeted stimulus delivery. Opt Express 15(21):13731–13744
Atick J, Redlich A (1992) What does the retina know about natural scenes? Neural Comput 4:196–210
Barlow H (1961) Possible principles underlying the transformation of sensory messages. In: Rosenblith WA (ed) Sensory communications. MIT, Cambridge, MA, pp 217–234
Barron JT, Malik J (2012) Shape, albedo, and illumination from a single image of an unknown object. In: Conference on computer vision and pattern recognition, Washington, DC, 1–8
BBC (1973) Controversy. http://www.aiai.ed.ac.uk/events/lighthill1973/1973-BBC-Lighthill-Controversy.mov
Blakemore C, Campbell FW (1969) On the existence of neurones in the human visual system selectively sensitive to the orientation and size of retinal images. J Physiol 203(1):237–260
Boyaci H, Fang F, Murray SO, Kersten D (2007) Responses to lightness variations in early human visual cortex. Curr Biol 17:989–993
Brainard DH, Williams DR et al (2008) Trichromatic reconstruction from the interleaved cone mosaic: Bayesian model and the color appearance of small spots. J Vis 8(5):15
Briggs F, Usrey WM (2007) A fast, reciprocal pathway between the lateral geniculate nucleus and visual cortex in the macaque monkey. J Neurosci 27(20):5431–5436
Brown M, Lowe DG (2005) Unsupervised 3D object recognition and reconstruction in unordered datasets. In: Fifth international conference on 3-D digital imaging and modeling, 2005, 3DIM 2005, Ottawa, 56–63
Cadieu CF, Olshausen BA (2012) Learning intermediate-level representations of form and motion from natural movies. Neural Comput 24(4):827–866
Carandini M, Demb JB et al (2005) Do we know what the early visual system does? J Neurosci 25(46):10577–10597
Clark D, Uetz G (1990) Video image recognition by the jumping spider, Maevia inclemens(Araneae: Salticidae). Anim Behav 40:884–890
Dan Y, Atick JJ et al (1996) Efficient coding of natural scenes in the lateral geniculate nucleus: experimental test of a computational theory. J Neurosci 16(10):3351–3362
Daugman J (1985) Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. J Opt Soc Am 2:1160–1169
David SV, Vinje WE et al (2004) Natural stimulus statistics alter the receptive field structure of v1 neurons. J Neurosci 24(31):6991–7006
Dayan P, Hinton GE et al (1995) The Helmholtz machine. Neural Comput 7(5):889–904
De Valois RL, Albrecht DG et al (1982) Spatial frequency selectivity of cells in macaque visual cortex. Vision Res 22(5):545–559
Douglas RJ, Martin KAC (2004) Neuronal circuits of the neocortex. Annu Rev Neurosci 27(1):419–451
Douglas RJ, Martin KAC et al (1989) A canonical microcircuit for neocortex. Neural Comput 1(4):480–488
Drees O (1952) Untersuchungen über die angeborenen verhaltensweisen bei springspinnen (salticidae). Z Tierpsychol 9(2):169–207
Dreyfus HL, Dreyfus SE (1988) Making a mind versus modeling the brain: artificial intelligence back at a branchpoint. Daedalus 117(1):15–43
Felleman DJ, Van Essen DC (1991) Distributed hierarchical processing in the primate cerebral cortex. Cereb Cortex 1(1):1–47
Field D (1987) Relations between the statistics of natural images and the response properties of cortical-cells. J Opt Soc Am A 4:2379–2394
Frégnac Y, Baudot P, Levy M, Marre O (2005) An intracellular view of time coding and sparseness in V1 during virtual oculomotor exploration of natural scenes. In: 2nd International Cosyne conference in computational and systems neuroscience, Salt Lake City, UT, 17
Freiwald WA, Tsao DY et al (2009) A face feature space in the macaque temporal lobe. Nat Neurosci 12(9):1187–1196
Fukushima K (1980) Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybern 36(4):193–202
Gauthier JL, Field GD et al (2009a) Receptive fields in primate retina are coordinated to sample visual space more uniformly. PLoS Biol 7(4):e1000063
Gauthier JL, Field GD et al (2009b) Uniform signal redundancy of parasol and midget ganglion cells in primate retina. J Neurosci 29(14):4675–4680
Geisler WS, Perry JS et al (2001) Edge co-occurrence in natural images predicts contour grouping performance. Vision Res 41(6):711–724
Gibson J (1986) The ecological approach to visual perception—James Jerome Gibson—Google books. Erlbaum, Hillsdale, NJ
Gordon G, Kaplan DM, Lankow B, Little DY, Sherwin J, Suter BA, Thaler L (2011) Toward an integrated approach to perception and action: conference report and future directions. Front Syst Neurosci 5:20
Gray CM, Singer W (1989) Stimulus-specific neuronal oscillations in orientation columns of cat visual cortex. Proc Natl Acad Sci USA 86:1698–1702
Grill-Spector K, Kushnir T et al (2000) The dynamics of object-selective activation correlate with recognition performance in humans. Nat Neurosci 3(8):837–843
Hartley R, Zisserman A (2003) Multiple view geometry in computer vision. Cambridge University Press, Cambridge, MA
Heeger DJ (1999) Linking visual perception with human brain activity. Curr Opin Neurobiol 9(4):474–479
Hofer H, Williams D (2005) Different sensations from cones with the same photopigment. J Vis 5(5):444–454
Hofer H, Carroll J et al (2005) Organization of the human trichromatic cone mosaic. J Neurosci 25(42):9669–9679
Hubel DH (1982) Exploration of the primary visual cortex, 1955–78. Nature 299(5883):515–524
Hubel DH, Wiesel TN (1962) Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J Physiol 160:106–154
Hubel DH, Wiesel TN (1965) Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. J Neurophysiol 28:229–289
Hung CP, Kreiman G et al (2005) Fast readout of object identity from macaque inferior temporal cortex. Science 310(5749):863–866
Hupé JM, James AC et al (2001) Feedback connections act on the early part of the responses in monkey visual cortex. J Neurophysiol 85(1):134–145
Hyvarinen A, Hoyer PO (2001) A two-layer sparse coding model learns simple and complex cell receptive fields and topography from natural images. Vision Res 41(18):2413–2423
Hyvarinen A, Gutmann M et al (2005) Statistical model of natural stimuli predicts edge-like pooling of spatial frequency channels in V2. BMC Neurosci 6(1):12
Julesz B (1981) Textons, the elements of texture perception, and their interactions. Nature 290(5802):91–97
Karklin Y, Lewicki M (2003) Learning higher-order structures in natural images. Network 14(3):483–499
Karklin Y, Lewicki M (2005) A hierarchical Bayesian model for learning nonlinear statistical regularities in nonstationary natural signals. Neural Comput 17(2):397–423
Karklin Y, Lewicki MS (2009) Emergence of complex cell properties by learning to generalize in natural scenes. Nature 457(7225):83–85
Kersten D, Mamassian P et al (2004) Object perception as Bayesian inference. Annu Rev Psychol 55:271–304
Khosrowshahi A, Baker J et al (2007) Predicting responses of V1 neurons to natural movies. Society for Neuroscience, San Diego, CA, p 33
Knill D, Richards W (1996) Perception as Bayesian inference. Cambridge University Press, Cambridge, MA
Knill DC, Saunders JA (2003) Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Res 43(24):2539–2558
Koepsell K, Wei Y, Wang Q, Rathbun DL, Usrey WM, Hirsch JA, Sommer FT (2009) Retinal oscillations carry visual information to cortex. Front Syst Neurosci 3
Koepsell K, Wang X et al (2010) Exploring the function of neural oscillations in early sensory systems. Front Neurosci 4:53
Kourtzi Z, Kanwisher N (2001) Representation of perceived object shape by the human lateral occipital complex. Science 293(5534):1506–1509
Land MF (1969) Movements of the retinae of jumping spiders (Salticidae: dendryphantinae) in response to visual stimuli. J Exp Biol 51(2):471–493
Land MF (1971) Orientation by jumping spiders in the absence of visual feedback. J Exp Biol 54(1):119–139
Land MF (1985) Fields of view of the eyes of primitive jumping spiders. J Exp Biol 119:381–384
Lee TS, Mumford D (2003) Hierarchical Bayesian inference in the visual cortex. J Opt Soc Am A Opt Image Sci Vis 20(7):1434–1448
Lee TS, Yang CF et al (2002) Neural activity in early visual cortex reflects behavioral experience and higher-order perceptual saliency. Nat Neurosci 5(6):589–597
Lennie P (1998) Single units and visual cortical organization. Perception 27(8):889–935
Litke AM, Bezayiff N et al (2004) What does the eye tell the brain? Development of a system for the large-scale recording of retinal output activity. IEEE Trans Nucl Sci 51(4):1434–1440
Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110
Ma WJ, Beck JM et al (2006) Bayesian inference with probabilistic population codes. Nat Neurosci 9(11):1432–1438
Mamassian P, Knill DC et al (1998) The perception of cast shadows. Trends Cogn Sci 2(8):288–295
Mancilla JG, Lewis TJ et al (2007) Synchronization of electrically coupled pairs of inhibitory interneurons in neocortex. J Neurosci 27(8):2058–2073
Marcelja S (1980) Mathematical description of the responses of simple cortical cells. J Opt Soc Am 70(11):1297–1300
Marr D (1982) Vision: a computational investigation into the human representation and processing of visual information. WH Freeman, San Francisco, CA
Mumford D (1994) Neuronal architectures for pattern-theoretic problems. Large-scale neuronal theories of the brain. MIT, Cambridge, MA
Murray SO, Kersten D et al (2002) Shape perception reduces activity in human primary visual cortex. Proc Natl Acad Sci USA 99(23):15164–15169
Nakayama K, He Z et al (1995) Visual surface representation: a critical link between lower-level and higher-level vision. In: Kosslyn SM, Osherson DN (eds) An invitation to cognitive science: visual cognition, vol 2. MIT, Cambridge, MA, pp 1–70
Naselaris T, Prenger RJ et al (2009) Bayesian reconstruction of natural images from human brain activity. Neuron 63(6):902–915
Naselaris T, Kay KN et al (2011) Encoding and decoding in fMRI. Neuroimage 56(2):400–410
Newcombe RA, Davison AJ (2010) Live dense reconstruction with a single moving camera. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR), San Francisco, CA, 1498–1505
Nishimoto S, Vu AT et al (2011) Reconstructing visual experiences from brain activity evoked by natural movies. Curr Biol 21(19):1641–1646
O’Rourke NA, Weiler NC, Micheva KD, Smith SJ (2012) Deep molecular diversity of mammalian synapses: why it matters and how to measure it. Nat Rev Neurosci 13:365–379
Olshausen BA, Field DJ (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381(6583):607–609
Oram MW, Perrett DI (1992) Time course of neural responses discriminating different views of the face and head. J Neurophysiol 68(1):70–84
Papert S (1966) The summer vision project. MIT Artificial Intelligence Group, Vision Memo No. 100, 1–6
Pearl J (1988) Probabilistic reasoning in intelligent systems: networks of plausible inference—Judea Pearl—Google books. Morgan Kaufmann Publishers, San Francisco, CA
Poirazi P, Brannon T et al (2003) Pyramidal neuron as two-layer neural network. Neuron 37(6):989–999
Polsky A, Mel BW et al (2004) Computational subunits in thin dendrites of pyramidal cells. Nat Neurosci 7(6):621–627
Qiu FT, von der Heydt R (2005) Figure and ground in the visual cortex: v2 combines stereoscopic cues with gestalt rules. Neuron 47(1):155–166
Quiroga RQ, Reddy L et al (2005) Invariant visual representation by single neurons in the human brain. Nature 435(7045):1102–1107
Rao RP, Ballard DH (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci 2(1):79–87
Rao RPN, Olshausen BA et al (2002) Probabilistic models of the brain: perception and neural function. MIT, Cambridge, MA
Ress D, Heeger DJ (2003) Neuronal correlates of perception in early visual cortex. Nat Neurosci 6(4):414–420
Riesenhuber M, Poggio T (2004) How the visual cortex recognizes objects: the tale of the standard model. In: Chalupa L, Werner J (eds) The visual neurosciences. MIT, Cambridge, MA, pp 1–14
Roorda A (2011) Adaptive optics for studying visual function: a comprehensive review. J Vis 11(7)
Roorda A, Williams DR (1999) The arrangement of the three cone classes in the living human eye. Nature 397(6719):520–522
Rossi AF, Rittenhouse CD et al (1996) The representation of brightness in primary visual cortex. Science 273(5278):1104–1107
Rust N, Movshon J (2005) In praise of artifice. Nat Neurosci 8(12):1647–1650
Schwartz O, Simoncelli EP (2001) Natural signal statistics and sensory gain control. Nat Neurosci 4:819–825
Sincich LC, Zhang Y et al (2009) Resolving single cone inputs to visual receptive fields. Nat Neurosci 12(8):967–969
Smith E, Lewicki M (2006) Efficient auditory coding. Nature 439(7079):978–982
Snavely N, Seitz SM et al (2006) Photo tourism: exploring photo collections in 3D. ACM, New York, NY
Srinivasan MV, Laughlin SB et al (1982) Predictive coding: a fresh view of inhibition in the retina. Proc R Soc Lond B Biol Sci 216(1205):427–459
Tappen MF, Freeman WT et al (2005) Recovering intrinsic images from a single image. IEEE Trans Pattern Anal Mach Intell 27(9):1459–1472
Tarsitano M, Andrew R (1999) Scanning and route selection in the jumping spider Portia labiata. Anim Behav 58(2):255–265
Tarsitano M, Jackson RR (1997) Araneophagic jumping spiders discriminate between detour routes that do and do not lead to prey. Anim Behav 53:257–266
Thomson AM, Bannister AP (2003) Interlaminar connections in the neocortex. Cereb Cortex 13(1):5–14
Thorpe SJ, Imbert M (1989) Biological constraints on connectionist models. In: Pfeifer R, Schreter Z, Fogelman-Soulie F, Steels L (eds) Connectionism in perspective. Elsevier Inc., Amsterdam, pp 63–92
Thorpe S, Fize D et al (1996) Speed of processing in the human visual system. Nature 381:520–522
Thrun S, Montemerlo M et al (2006) Stanley: the robot that won the DARPA grand challenge. J Field Robot 23(9):661–692
Tinbergen N (1974) Curious naturalists (revised edition). University of Massachusetts Press, Amherst, MA
Treisman AM, Gelade G (1980) A feature-integration theory of attention. Cogn Psychol 12(1):97–136
Tsao DY, Freiwald WA et al (2006) A cortical region consisting entirely of face-selective cells. Science 311(5761):670–674
Tsao DY, Moeller S et al (2008) Comparing face patch systems in macaques and humans. Proc Natl Acad Sci 105(49):19514–19519
Van Essen DC, Anderson CH (1995) Information processing strategies and pathways in the primate visual system. In: Zornetzer SF, Davis JL, Lau C, McKenna T (eds) An introduction to neural and electronic networks. Academic, San Diego, CA, pp 45–76
Vinje WE, Gallant JL (2000) Sparse coding and decorrelation in primary visual cortex during natural vision. Science 287(5456):1273–1276
Vogel CR, Arathorn DW et al (2006) Retinal motion estimation in adaptive optics scanning laser ophthalmoscopy. Opt Express 14(2):487–497
Wandell BA, Dumoulin SO et al (2007) Visual field maps in human cortex. Neuron 56(2):366–383
Zhou H, Friedman HS et al (2000) Coding of border ownership in monkey visual cortex. J Neurosci 20(17):6594–6611
Zhu M, Rozell C (2011) Population characteristics and interpretations of nCRF effects emerging from sparse coding. In: Computational and Systems Neuroscience (COSYNE), Salt Lake City, UT
Acknowledgments
I thank Jim Bower for encouraging me to write this article and for his patience in giving me the time to complete it, and Jim DiCarlo for providing the MIT AI Memo. Supported by funding from NSF (IIS-1111654), NIH (EY019965), NGA (HM1582-08-1-0007), and the Canadian Institute for Advanced Research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer Science+Business Media New York
About this chapter
Cite this chapter
Olshausen, B.A. (2013). 20 Years of Learning About Vision: Questions Answered, Questions Unanswered, and Questions Not Yet Asked. In: Bower, J. (eds) 20 Years of Computational Neuroscience. Springer Series in Computational Neuroscience, vol 9. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-1424-7_12
Download citation
DOI: https://doi.org/10.1007/978-1-4614-1424-7_12
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-1423-0
Online ISBN: 978-1-4614-1424-7
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)