Skip to main content
SpringerLink
Log in

Vision-Based Grasping, Where Robotics Meets Neuroscience

  • Chapter
  • First Online:
The Visual Neuroscience of Robotic Grasping

Part of the book series: Cognitive Systems Monographs ((COSMOS,volume 28))

Abstract

This chapter describes how we model the integration of on-line, action-oriented visual information (dorsal pathway) with knowledge about the target object and memories of previous grasping experiences and object characteristics (ventral pathway). Previous models of vision-based grasping have built so far mainly, when not exclusively, on monkey data. Recent neuropsychological and neuroimaging research has shed a new light on how visuomotor coordination is organized and performed in the human brain. Thanks to such research, a model of vision-based grasping which integrates knowledge coming from single-cell monkey studies with human data can be developed. The basic framework of the proposed model is outlined in this chapter. Final goal of the proposal is to mimic, in a robotic setup, the coordination between sensory, associative and motor cortex of the human brain in vision-based grasping actions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Ansuini C (2008) Reaching beyond grasp. Ph.D. thesis, Università degli Studi di Padova

    Google Scholar 

  • Ansuini C, Santello M, Massaccesi S, Castiello U (2006) Effects of end-goal on hand shaping. J Neurophysiol 95(4):2456–2465. doi:10.1152/jn.01107.2005

    Article  Google Scholar 

  • Ansuini C, Giosa L, Turella L, Alto G, Castiello U (2008) An object for an action, the same object for other actions: effects on hand shaping. Exp Brain Res 185(1):111–119. doi:10.1007/s00221-007-1136-4

    Article  Google Scholar 

  • Ansuini C, Santello M, Tubaldi F, Massaccesi S, Castiello U (2007a) Control of hand shaping in response to object shape perturbation. Exp Brain Res 180(1):85–96, 006–0840-9

    Google Scholar 

  • Arbib MA, Iberall T, Lyons D (1985) Coordinated control programs for control of the hands. In: Goodwin AW, Darian-Smith I (eds) Hand function and the neocortex. Experimental brain research supplemental, vol 10. Springer, Berlin, pp 111–129

    Google Scholar 

  • Arbib MA (2010) Mirror system activity for action and language is embedded in the integration of dorsal and ventral pathways. Brain Lang 112(1):12–24. doi:10.1016/j.bandl.2009.10.001

    Article  MathSciNet  Google Scholar 

  • Baud-Bovy G, Soechting JF (2001) Two virtual fingers in the control of the tripod grasp. J Neurophysiol 86(2):604–615

    Google Scholar 

  • Begliomini C, Wall MB, Smith AT, Castiello U (2007) Differential cortical activity for precision and whole-hand visually guided grasping in humans. Eur J Neurosci 25(4):1245–1252. doi:10.1111/j.1460-9568.2007.05365.x

    Article  Google Scholar 

  • Bingham GP, Muchisky MM (1993a) Center of mass perception and inertial frames of reference. Percept Psychophysi 54(5):617–632

    Google Scholar 

  • Binkofski F, Buccino G, Posse S, Seitz RJ, Rizzolatti G, Freund H (1999) A fronto-parietal circuit for object manipulation in man: evidence from an fMRI-study. Eur J Neurosci 11(9):3276–3286

    Article  Google Scholar 

  • Bryson J, Stein LA (2001) Modularity and specialized learning: mapping between agent architectures and brain organization. Emergent Neural Comput Architectures Based Neurosci LNCS 2036:98–113

    Article  Google Scholar 

  • Cadieu C, Kouh M, Pasupathy A, Connor CE, Riesenhuber M, Poggio T (2007) A model of V4 shape selectivity and invariance. J Neurophysiol 98(3):1733–1750. doi:10.1152/jn.01265.2006

    Article  Google Scholar 

  • Castiello U, Bennetta KM, Egan GF, Tochon-Danguy HJ, Kritikos A, Dunai J (2000) Human inferior parietal cortex programs the action class of grasping. Cogn Syst Res 1(2):89–97. doi:10.1016/S1389-0417(99)00011-X

    Article  Google Scholar 

  • Castiello U (2005) The neuroscience of grasping. Nat Rev Neurosci 6(9):726–736. doi:10.1038/nrn1744

    Article  Google Scholar 

  • Cattaneo L, Voss M, Brochier T, Prabhu G, Wolpert DM, Lemon RN (2005) A cortico-cortical mechanism mediating object-driven grasp in humans. Proc Nat Acad Sci USA 102(3):898–903. doi:10.1073/pnas.0409182102

    Article  Google Scholar 

  • Cavina-Pratesi C, Monaco S, McAdam T, Milner D, Schenk T, Culham JC (2007b) Which aspects of hand-preshaping does human AIP compute during visually guided actions? Evidence from event-related fMRI. In: Annual meeting of the society for neuroscience

    Google Scholar 

  • Chinellato E, Fisher RB, Morales A, del Pobil AP (2003a) Ranking planar grasp configurations for a three-finger hand. In: IEEE International conference on robotics and automation, Taipei, Taiwan

    Google Scholar 

  • Cisek P (2005) A computational model of reach decisions in the primate cerebral cortex. In: Modeling natural action selection

    Google Scholar 

  • Clower DM, Dum RP, Strick PL (2005) Basal ganglia and cerebellar inputs to ‘AIP’. Cereb Cortex 15(7):913–920. doi:10.1093/cercor/bhh190

    Article  Google Scholar 

  • Cohen NR, Cross ES, Tunik E, Grafton ST, Culham JC (2009) Ventral and dorsal stream contributions to the online control of immediate and delayed grasping: a tms approach. Neuropsychologia 47(6):1553–1562. doi:10.1016/j.neuropsychologia.2008.12.034

    Article  Google Scholar 

  • Cuijpers RH, Smeets JBJ, Brenner E (2004) On the relation between object shape and grasping kinematics. J Neurophysiol 91(6):2598–2606. doi:10.1152/jn.00644.2003

    Article  Google Scholar 

  • Cutkosky M, Howe R (1990) Human grasp choice and robotic grasp analysis. In: Venkataraman S, Iberall T (eds) Dextrous robot hands, Springer, Berlin, Chap. 1, pp 5–31

    Google Scholar 

  • Eastough D, Edwards MG (2007) Movement kinematics in prehension are affected by grasping objects of different mass. Exp Brain Res 176(1):193–198. doi:10.1007/s00221-006-0749-3

    Article  Google Scholar 

  • Ehrsson HH, Fagergren A, Jonsson T, Westling G, Johansson RS, Forssberg H (2000) Cortical activity in precision- versus power-grip tasks: an fMRI study. J Neurophysiol 83(1):528–536

    Google Scholar 

  • Ehrsson HH, Fagergren A, Johansson RS, Forssberg H (2003) Evidence for the involvement of the posterior parietal cortex in coordination of fingertip forces for grasp stability in manipulation. J Neurophysiol 90(5):2978–2986. doi:10.1152/jn.00958.2002

    Article  Google Scholar 

  • Ellison A, Cowey A (2009) Differential and co-involvement of areas of the temporal and parietal streams in visual tasks. Neuropsychologia 47(6):1609–1614. doi:10.1016/j.neuropsychologia.2008.12.013

    Article  Google Scholar 

  • Fagg AH, Arbib MA (1998) Modeling parietal-premotor interactions in primate control of grasping. Neural Networks 11(7–8):1277–1303

    Article  Google Scholar 

  • Frey SH, Vinton D, Norlund R, Grafton ST (2005) Cortical topography of human anterior intraparietal cortex active during visually guided grasping. Cogn Brain Res 23(2–3):397–405. doi:10.1016/j.cogbrainres.2004.11.010

    Article  Google Scholar 

  • Fukuda H, Fukumura N, Katayama M, Uno Y (2000) Relation between object recognition and formation of hand shape: a computational approach to human grasping movements. Syst Comput Jpn 31(12):11–22

    Google Scholar 

  • Fukui T, Takemura N, Inui T (2006) Visuomotor transformation process in goal-directed prehension: utilization of online vision during preshaping phase of grasping. Jpn Psychol Res 48(3):188–203. doi:10.1111/j.1468-5884.2006.00318.x, http://www.blackwell-synergy.com/doi/pdf/10.1111/j.1468-5884.2006.00318.x

    Google Scholar 

  • Galea MP, Castiello U, Dalwood N (2001) Thumb invariance during prehension movement: effects of object orientation. Neuroreport 12(10):2185–2187

    Article  Google Scholar 

  • Gallese V, Craighero L, Fadiga L, Fogassi L (1999) Perception through action. Psyche 5(21):1

    Google Scholar 

  • Gentilucci M, Caselli L, Secchi C (2003) Finger control in the tripod grasp. Exp Brain Res 149(3):351–360. doi:10.1007/s00221-002-1359-3

    Google Scholar 

  • Goodale MA, Milner AD (2004) Sight unseen. Oxford University Press, Oxford

    Google Scholar 

  • Goodale MA, Westwood DA, Milner AD (2004) Two distinct modes of control for object-directed action. Prog Brain Res 144:131–144

    Google Scholar 

  • Gordon AM, Westling G, Cole KJ, Johansson RS (1993) Memory representations underlying motor commands used during manipulation of common and novel objects. J Neurophysiol 69(6):1789–1796

    Google Scholar 

  • Himmelbach M, Karnath HO (2005) Dorsal and ventral stream interaction: contributions from optic ataxia. J Cogn Neurosci 17(4):632–640. doi:10.1162/0898929053467514

    Article  Google Scholar 

  • Hoeren M, Kaller CP, Glauche V, Vry MS, Rijntjes M, Hamzei F, Weiller C (2013) Action semantics and movement characteristics engage distinct processing streams during the observation of tool use. Exp Brain Res 229(2):243–260. doi:10.1007/s00221-013-3610-5

    Article  Google Scholar 

  • Hubel D, Wiesel T (1962) Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J Physiol London 160(1):106–154

    Article  Google Scholar 

  • Iberall T (1987) The nature of human prehension: three dextrous hands in one. In: IEEE international conference on robotics and automation, pp 396–401

    Google Scholar 

  • Iberall T, Bingham G, Arbib MA (1986) Opposition space as a structuring concept for the analysis of skilled hand movements. In: Heuer H, Fromm C (eds) Generation and modulation of action patterns. Springer, Berlin, pp 158–173

    Chapter  Google Scholar 

  • Jeannerod M (1997) The cognitive neuroscience of action. Blackwell, Oxford

    Google Scholar 

  • Jenmalm P, Dahlstedt S, Johansson RS (2000) Visual and tactile information about object-curvature control fingertip forces and grasp kinematics in human dexterous manipulation. J Neurophysiol 84(6):2984–2997

    Google Scholar 

  • Kawasaki M, Watanabe M, Okuda J, Sakagami M, Aihara K (2008) Human posterior parietal cortex maintains color, shape and motion in visual short-term memory. Brain Res 1213:91–97. doi:10.1016/j.brainres.2008.03.037

    Article  Google Scholar 

  • Kawato M (1999) Internal models for motor control and trajectory planning. Curr Opin Neurobiol 9:718–727

    Article  Google Scholar 

  • Khosla D, Chen Y, Kim K (2014) A neuromorphic system for video object recognition. Front Comput Neurosci 8:147. doi:10.3389/fncom.2014.00147

    Article  Google Scholar 

  • Króliczak G, Cavina-Pratesi C, Goodman DA, Culham JC (2007) What does the brain do when you fake it? an fMRI study of pantomimed and real grasping. J Neurophysiol 97(3):2410–2422. doi:10.1152/jn.00778.2006

    Article  Google Scholar 

  • Lebedev MA, Wise SP (2002) Insights into seeing and grasping: Distinguishing the neural correlates of perception and action. Behav Cognitive Neurosci Rev 1(2):108–129. doi:10.1177/1534582302001002002

    Article  Google Scholar 

  • Lederman SJ, Wing AM (2003) Perceptual judgement, grasp point selection and object symmetry. Exp Brain Res 152(2):156–165. doi:10.1007/s00221-003-1522-5

    Article  Google Scholar 

  • Lourens T, Barakova EI (2007) Orientation contrast sensitive cells in primate V1 a computational model. Nat Comput 6(3):241–252

    Article  MATH  MathSciNet  Google Scholar 

  • Mackenzie C, Iberall T (1994) The grasping hand. North Holland, New York

    Google Scholar 

  • Makuuchi M, Someya Y, Ogawa S, Takayama Y (2012) Hand shape selection in pantomimed grasping: interaction between the dorsal and the ventral visual streams and convergence on the ventral premotor area. Hum Brain Mapp 33(8):1821–1833. doi:10.1002/hbm.21323

    Article  Google Scholar 

  • McIntosh RD, Dijkerman HC, Mon-Williams M, Milner AD (2004) Grasping what is graspable: evidence from visual form agnosia. Cortex 40(4–5):695–702

    Article  Google Scholar 

  • McLaughlin D, Shapley R, Shelley M (2003) Large-scale modeling of the primary visual cortex: influence of cortical architecture upon neuronal response. J Physiol Paris 97(2–3):237–252. doi:10.1016/j.jphysparis.2003.09.019

    Article  Google Scholar 

  • Miall R (2003) Connecting mirror neurons and forward models. Neuroreport 14(16):1–3

    Google Scholar 

  • Milner AD, Dijkerman HC, McIntosh RD, Rossetti Y, Pisella L (2003) Delayed reaching and grasping in patients with optic ataxia. Prog Brain Res 142:225–242

    Google Scholar 

  • Morales A, Sanz PJ, del Pobil AP, Fagg AH (2006) Vision-based three-finger grasp synthesis constrained by hand geometry. Rob Auton Syst 54(6):496–512

    Article  Google Scholar 

  • Morales A, Chinellato E, Fagg AH, del Pobil AP (2004) Using experience for assessing grasp reliability. Int J Humanoid Rob 1(4):671–691

    Google Scholar 

  • Napier JR (1956) The prehensile movements of the human hand. J Bone Joint Surg (Br) 38-B(4):902–913

    Google Scholar 

  • Ogawa K, Inui T, Sugio T, (2007) Neural correlates of state estimation in visually guided movements: an event-related fMRI study. Cortex 43(3):289–300

    Google Scholar 

  • O’Reilly RC, Munakata Y (2000) Computational explorations in cognitive neuroscience—understanding the mind by simulating the brain. MIT Press, Cambridge

    Google Scholar 

  • Oztop E, Bradley NS, Arbib MA (2004) Infant grasp learning: a computational model. Exp Brain Res 158(4):480–503. doi:10.1007/s00221-004-1914-1

    Article  Google Scholar 

  • Oztop E, Arbib MA (2002) Schema design and implementation of the grasp-related mirror neuron system. Biol Cybern 87(2):116–140. doi:10.1007/s00422-002-0318-1

    Article  MATH  Google Scholar 

  • Oztop E, Imamizu H, Cheng G, Kawato M (2006) A computational model of anterior intraparietal (AIP) neurons. Neurocomputing 69(10–12):1354–1361

    Google Scholar 

  • Passingham RE, Toni I (2001) Contrasting the dorsal and ventral visual systems: guidance of movement versus decision making. Neuroimage 14(1 Pt 2):S125–S131. doi:10.1006/nimg.2001.0836

    Article  Google Scholar 

  • Paulignan Y, Frak VG, Toni I, Jeannerod M (1997) Influence of object position and size on human prehension movements. Exp Brain Res 114(2):226–234

    Article  Google Scholar 

  • Pouget S, Sejnowski A (1997) Spatial transformations in the parietal cortex using basis functions. J Cogn Neurosci 9(2):222–237

    Article  Google Scholar 

  • Pouget A, Snyder LH (2000) Computational approaches to sensorimotor transformations. Nat Neurosci 3(Suppl):1192–1198. doi:10.1038/81469

    Article  Google Scholar 

  • Quinlan DJ, Goodale MA, Culham JC (2005) Don’t bite the hand that feeds you: a comparison of mouth and hand kinematics. J Vis 5(8):382

    Article  Google Scholar 

  • Riesenhuber M, Poggio T (2000) Models of object recognition. Nat Neurosci 3:1199–1204. doi:10.1038/81479

    Article  Google Scholar 

  • Rizzolatti G, Luppino G (2001) The cortical motor system. Neuron 31(6):889–901

    Article  Google Scholar 

  • Rolls E, Deco G (2002) Computational neuroscience of vision. Oxford University Press, Oxford, UK

    Google Scholar 

  • Rolls ET, Webb TJ (2014) Finding and recognizing objects in natural scenes: complementary computations in the dorsal and ventral visual systems. Front Comput Neurosci 8:85. doi:10.3389/fncom.2014.00085

    Article  Google Scholar 

  • Sabatini SP, Solari F, Andreani G, Bartolozzi C, Bisio GM (2001) A hierarchical model of complex cells in visual cortex for the binocular perception of motion-in-depth. Adv Neural In Process Syst 1271–1278

    Google Scholar 

  • Sakata H, Tsutsui KI, Taira M (2005) Toward an understanding of the neural processing for 3D shape perception. Neuropsychologia 43(2):151–161. doi:10.1016/j.neuropsychologia.2004.11.003

    Article  Google Scholar 

  • Sereno ME, Trinath T, Augath M, Logothetis NK (2002) Three-dimensional shape representation in monkey cortex. Neuron 33(4):635–652

    Article  Google Scholar 

  • Shikata E, Hamzei F, Glauche V, Koch M, Weiller C, Binkofski F, Büchel C (2003) Functional properties and interaction of the anterior and posterior intraparietal areas in humans. Eur J Neurosci 17(5):1105–1110

    Article  Google Scholar 

  • Snow JC, Pettypiece CE, McAdam TD, McLean AD, Stroman PW, Goodale MA, Culham JC (2011) Bringing the real world into the fmri scanner: repetition effects for pictures versus real objects. Sci Rep 1:130. doi:10.1038/srep00130

    Article  Google Scholar 

  • Sugio T, Ogawa K, Inui T (2003a) Neural correlates of semantic effects on grasping familiar objects. Neuroreport 14(18):2297–2301. doi:10.1097/01.wnr.0000092474.09492.3a

  • Sugio T, Ogawa K, Inui T (2003b) Multiple action representations of familiar objects with handles: an fMRI study. In: European conference on visual perception

    Google Scholar 

  • Takasawa M, Oku N, Osaki Y, Kinoshita H, Imaizumi M, Yoshikawa T, Kimura Y, Kajimoto K, Sasagaki M, Kitagawa K, Hori M, Hatazawa J (2003) Cerebral and cerebellar activation in power and precision grip movements: an \({H}_2\) \(^{15}{O}\) positron emission tomography study. J Cereb Blood Flow Metab 23(11):1378–1382, doi:DOIurl10.1097/01.WCB.0000091258.83091.C2

    Google Scholar 

  • Tankus A, Fried I (2012) Visuomotor coordination and motor representation by human temporal lobe neurons. J Cogn Neurosci 24(3):600–610

    Article  Google Scholar 

  • Tessari A, Rumiati RI (2002) Motor distal component and pragmatic representation of objects. Cogn Brain Res 14(2):218–227

    Article  Google Scholar 

  • Tresilian JR, Mon-Williams M, Kelly BM (1999) Increasing confidence in vergence as a cue to distance. Proc R Soc B: Biol Sci 266(1414):39–44

    Article  Google Scholar 

  • Tucker M, Ellis R (2004) Action priming by briefly presented objects. ACTA Psychol (Amsterdam) 116(2):185–203. doi:10.1016/j.actpsy.2004.01.004

    Article  Google Scholar 

  • Ullman S (1996) High-level vision. Object recognition and visual cognition. MIT Press, Cambridge

    MATH  Google Scholar 

  • Uno Y, Fukumura N, Suzuki R, Kawato M (1995) A computational model for recognizing objects and planning hand shapes in grasping movements. Neural Networks 8(6):839–851. doi:10.1016/0893-6080(95)00002-H

    Article  Google Scholar 

  • Valyear KF, Culham JC (2010) Observing learned object-specific functional grasps preferentially activates the ventral stream. J Cogn Neurosci 22(5):970–984. doi:10.1162/jocn.2009.21256

    Article  Google Scholar 

  • Wagman JB, Carello C (2003) Haptically creating affordances: the user-tool interface. J Exp Psychol: Appl 9(3):175–186. doi:10.1037/1076-898X.9.3.175

    Google Scholar 

  • Westwood DA, Chapman CD, Roy EA (2000a) Pantomimed actions may be controlled by the ventral visual stream. Exp Brain Res 130(4):545–548

    Google Scholar 

  • Wokke ME, Scholte HS, Lamme VAF (2014) Opposing dorsal/ventral stream dynamics during figure-ground segregation. J Cogn Neurosci 26(2):365–379

    Article  Google Scholar 

  • Yousefi B, Loo CK (2014a) Comparative study on interaction of form and motion processing streams by applying two different classifiers in mechanism for recognition of biological movement. Sci World J 723:213. doi:10.1155/2014/723213

    Google Scholar 

  • Yousefi B, Loo CK (2014b) Development of biological movement recognition by interaction between active basis model and fuzzy optical flow division. Sci World J 238:234. doi:10.1155/2014/238234

    Google Scholar 

  • Zachariou V, Klatzky R, Behrmann M (2014) Ventral and dorsal visual stream contributions to the perception of object shape and object location. J Cogn Neurosci 26(1):189–209

    Article  Google Scholar 

  • Zhong J, Cangelosi A, Wermter S (2014) Toward a self-organizing pre-symbolic neural model representing sensorimotor primitives. Front Behav Neurosci 8:22. doi:10.3389/fnbeh.2014.00022

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing, University of Leeds, Leeds, UK

    Eris Chinellato

  2. Department of ICC, Jaume I University, Castellón, Spain

    Angel P. del Pobil

Authors
  1. Eris Chinellato
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Angel P. del Pobil
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eris Chinellato .

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Chinellato, E., del Pobil, A.P. (2016). Vision-Based Grasping, Where Robotics Meets Neuroscience. In: The Visual Neuroscience of Robotic Grasping. Cognitive Systems Monographs, vol 28. Springer, Cham. https://doi.org/10.1007/978-3-319-20303-4_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-20303-4_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-20302-7

  • Online ISBN: 978-3-319-20303-4

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation