Skip to main content
SpringerLink
Log in

Reward-Based Learning, Model-Based and Model-Free

  • Living reference work entry
  • Latest version View entry history
  • First Online:
Encyclopedia of Computational Neuroscience

  • 64 Accesses

This is an updated version of the entry originally published as Huys Q.J.M., Cruickshank A., Seriès P. (2014) Reward-Based Learning, Model-Based and Model-Free. In: Jaeger D., Jung R. (eds) Encyclopedia of Computational Neuroscience. Springer, New York, NY.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  • Balleine B, Dickinson A (1994) Role of cholecystokinin in the motivational control of instrumental action in rats. Behav Neurosci 108(3):590–605

    Article  CAS  PubMed  Google Scholar 

  • Barto A, Sutton R, Anderson C (1983) Neuronlike elements that can solve difficult learning control problems. IEEE Trans Syst Man Cybern 13(5):834–846

    Article  Google Scholar 

  • Bayer HM, Glimcher PW (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47(1):129–141

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Bayer HM, Lau B, Glimcher PW (2007) Statistics of midbrain dopamine neuron spike trains in the awake primate. J Neurophysiol 98(3):1428–1439

    Article  PubMed  Google Scholar 

  • Bellman RE (1957) Dynamic programming. Princeton University Press, Princeton

    Google Scholar 

  • Bertsekas DP, Tsitsiklis JN (1996) Neuro-dynamic programming. Athena Scientific, Belmont

    Google Scholar 

  • Boutilier C, Dearden R, Goldszmidt M (1995) Exploiting structure in policy construction. In: IJCAI, vol 14, pp 1104–1113

    Google Scholar 

  • Bouton ME (2006) Learning and behavior: a contemporary synthesis. Sinauer, Sunderland

    Google Scholar 

  • Campbell M, Hoane A et al (2002) Deep Blue. Artif Intell 134(1–2):57–83

    Article  Google Scholar 

  • Cardinal RN, Parkinson JA, Lachenal G, Halkerston KM, Rudarakanchana N, Hall J, Morrison CH, Howes SR, Robbins TW, Everitt BJ (2002) Effects of selective excitotoxic lesions of the nucleus accumbens core, anterior cingulate cortex, and central nucleus of the amygdala on autoshaping performance in rats. Behav Neurosci 116(4):553–567

    Article  PubMed  Google Scholar 

  • Corbit LH, Balleine BW (2005) Double dissociation of basolateral and central amygdala lesions on the general and outcome-specific forms of Pavlovian-instrumental transfer. J Neurosci 25(4):962–970

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Corbit LH, Balleine BW (2011) The general and outcome-specific forms of pavlovian-instrumental transfer are differentially mediated by the nucleus accumbens core and shell. J Neurosci 31(33):11786–11794, https://doi.org/10.1523/JNEUROSCI.2711-11.2011

    Article  CAS  PubMed  Google Scholar 

  • D’Ardenne K, McClure SM, Nystrom LE, Cohen JD (2008) Bold responses reflecting dopaminergic signals in the human ventral tegmental area. Science 319(5867):1264–1267

    Article  PubMed  CAS  Google Scholar 

  • Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8(12):1704–1711

    Article  CAS  PubMed  Google Scholar 

  • Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011) Model-based influences on humans’ choices and striatal prediction errors. Neuron 69(6):1204–1215

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Day JJ, Roitman MF, Wightman RM, Carelli RM (2007) Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat Neurosci 10(8):1020–1028

    Article  CAS  PubMed  Google Scholar 

  • Dayan P (1993) Improving generalization for temporal difference learning: the successor representation. Neural Comput 5(4):613–624

    Article  Google Scholar 

  • Dayan P, Berridge KC (2014) Model-based and model-free pavlovian reward learning: revaluation, revision, and revelation. Cogn Affect Behav Neurosci 14(2):473–492

    Article  PubMed  PubMed Central  Google Scholar 

  • Dayan P, Niv Y, Seymour B, Daw ND (2006) The misbehavior of value and the discipline of the will. Neural Netw 19(8):1153–1160

    Article  PubMed  Google Scholar 

  • Dickinson A, Dearing MF (1979) Appetitive-aversive interactions and inhibitory processes. In: Dickinson A, Boakes RA (eds) Mechanisms of learning and motivation. Erlbaum, Hillsdale, pp 203–231

    Google Scholar 

  • Dickinson A, Smith J, Mirenowicz J (2000) Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists. Behav Neurosci 114(3):468–483

    Article  CAS  PubMed  Google Scholar 

  • Dietterich TG (1999) Hierarchical reinforcement learning with the maxq value function decomposition. CoRR, cs.LG/9905014

    Google Scholar 

  • Enomoto K, Matsumoto N, Nakai S, Satoh T, Sato TK, Ueda Y, Inokawa H, Haruno M, Kimura M (2011) Dopamine neurons learn to encode the long-term value of multiple future rewards. Proc Natl Acad Sci U S A 108(37):15462–15467

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Flagel SB, Clark JJ, Robinson TE, Mayo L, Czuj A, Willuhn I, Akers CA, Clinton SM, Phillips PEM, Akil H (2011) A selective role for dopamine in stimulus-reward learning. Nature 469(7328):53–57

    Article  CAS  PubMed  Google Scholar 

  • Frank MJ, Seeberger LC, O’Reilly RC (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306(5703):1940–1943

    Article  CAS  PubMed  Google Scholar 

  • Gillan CM, Papmeyer M, Morein-Zamir S, Sahakian BJ, Fineberg NA, Robbins TW, de Wit S (2011) Disruption in the balance between goal-directed behavior and habit learning in obsessive-compulsive disorder. Am J Psychiatry 168(7):718–726

    Article  PubMed  PubMed Central  Google Scholar 

  • Gillan CM, Kosinski M, Whelan R, Phelps EA, Daw ND (2016) Characterizing a psychiatric symptom dimension related to deficits in goal-directed control. eLife 2016; 5:e11305

    Google Scholar 

  • Gläscher J, Daw N, Dayan P, O’Doherty JP (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66(4):585–595

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Guitart-Masip M, Fuentemilla L, Bach DR, Huys QJM, Dayan P, Dolan RJ, Duzel E (2011) Action dominates valence in anticipatory representations in the human striatum and dopaminergic midbrain. J Neurosci 31(21):7867–7875

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Hampton AN, Bossaerts P, O’Doherty JP (2006) The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J Neurosci 26(32):8360–8367

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Hull C (1943) Principles of behavior. Appleton-Century-Crofts, New York

    Google Scholar 

  • Huys QJM (2007) Reinforcers and control. Towards a computational aetiology of depression. PhD thesis, Gatsby Computational Neuroscience Unit, UCL, University of London

    Google Scholar 

  • Huys QJM, Cools R, Gölzer M, Friedel E, Heinz A, Dolan RJ, Dayan P (2011) Disentangling the roles of approach, activation and valence in instrumental and Pavlovian responding. PLoS Comput Biol 7(4):e1002028

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Huys QJM, Eshel N, O’Nions E, Sheridan L, Dayan P, Roiser JP (2012) Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput Biol 8(3):e1002410

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Huys QJM, Tobler PN, Hasler G, Flagel SB (2014) The role of learning-related dopamine signals in addiction vulnerability. Prog Brain Res 211:31–77

    Article  CAS  PubMed  Google Scholar 

  • Johnson A, Redish AD (2007) Neural ensembles in ca3 transiently encode paths forward of the animal at a decision point. J Neurosci 27(45):12176–12189

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Kaelbling LP, Littman ML, Cassandra AR (1998) Planning and acting in partially observable stochastic domains. Artif Intell 101(1):99–134

    Article  Google Scholar 

  • Kamin LJ (1969) Predictability, surprise, attention and conditioning. In: Campbell BA, Church RM (eds) Punishment and aversive behavior. Appleton-Century-Crofts, New York

    Google Scholar 

  • Kearns M, Singh S (2002) Near-optimal reinforcement learning in polynomial time. Mach Learn 49(2–3):209–232

    Article  Google Scholar 

  • Keramati M, Dezfouli A, Piray P (2011) Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput Biol 7(5):e1002055

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Killcross S, Coutureau E (2003) Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb Cortex 13(4):400–408

    Article  PubMed  Google Scholar 

  • Knuth D, Moore R (1975) An analysis of alpha-Beta pruning. Artif Intell 6(4):293–326

    Article  Google Scholar 

  • Kocsis L, Szepesv’ari C (2006) Bandit based Monte-Carlo planning. In: Machine learning: ECML 2006. Springer, Berlin, pp 282–293

    Chapter  Google Scholar 

  • Maia TV, Frank MJ (2011) From reinforcement learning models to psychiatric and neurological disorders. Nat Neurosci 14(2):154–162

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • McClure SM, Daw ND, Montague PR (2003) A computational substrate for incentive salience. TINS 26:423–428

    CAS  PubMed  Google Scholar 

  • McDannald MA, Lucantonio F, Burke KA, Niv Y, Schoenbaum G (2011) Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J Neurosci 31(7):2700–2705

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Momennejad I, Russek EM, Cheong JH, Botvinick MM, Daw ND, Gershman SJ (2017) The successor representation in human reinforcement learning. Nat Hum Behav 1:680–692

    Article  CAS  PubMed  Google Scholar 

  • Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive hebbian learning. J Neurosci 16(5):1936–1947

    Article  CAS  PubMed  Google Scholar 

  • Morris G, Nevet A, Arkadir D, Vaadia E, Bergman H (2006) Midbrain dopamine neurons encode decisions for future action. Nat Neurosci 9(8):1057–1063

    Article  CAS  PubMed  Google Scholar 

  • Nebe S, Kroemer NB, Schad DJ, Bernhardt N, Sebold M, Mller DK, Scholl L, Kuitunen-Paul S, Heinz A, Rapp MA, Huys QJM, Smolka MN (2017) No association of goal-directed and habitual control with alcohol consumption in young adults. Addict Biol

    Google Scholar 

  • Nelson A, Killcross S (2006) Amphetamine exposure enhances habit formation. J Neurosci 26(14):3805–3812

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Pfeiffer BE, Foster DJ (2013) Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497(7447):74–79

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Puterman ML (2005) Markov decision processes: discrete stochastic dynamic programming (Wiley series in probability and statistics). Wiley-Interscience, New York

    Google Scholar 

  • Redish AD, Jensen S, Johnson A (2008) A unified framework for addiction: vulnerabilities in the decision process. Behav Brain Sci 31(4):415–437. discussion 437–87

    Article  PubMed  PubMed Central  Google Scholar 

  • Robbins TW, Gillan CM, Smith DG, de Wit S, Ersche KD (2012) Neurocognitive endophenotypes of impulsivity and compulsivity: towards dimensional psychiatry. Trends Cogn Sci 16(1):81–91

    Article  PubMed  Google Scholar 

  • Robinson MJF, Berridge KC (2013) Instant transformation of learned repulsion into motivational ‘wanting’. Curr Biol 23(4):282–289

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Roesch MR, Calu DJ, Schoenbaum G (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci 10(12):1615–1624

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Russek EM, Momennejad I, Botvinick MM, Gershman SJ, Daw ND (2017) Predictive representations can link model-based reinforcement learning to model-free mechanisms. PLoS Comput Biol 13:e1005768

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Saunders BT, Richard JM, Margolis EB, Janak PH (2018) Dopamine neurons create pavlovian conditioned stimuli with circuit-defined motivational properties. Nat Neurosci 21:1072–1083

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Schoenbaum G, Roesch MR, Stalnaker TA, Takahashi YK (2009) A new perspective on the role of the orbitofrontal cortex in adaptive behaviour. Nat Rev Neurosci 10(12):885–892

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Schultz W, Romo R (1990) Dopamine neurons of the monkey midbrain: contingencies of responses to stimuli eliciting immediate behavioral reactions. J Neurophysiol 63(3):607–624

    Article  CAS  PubMed  Google Scholar 

  • Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275(5306):1593–1599

    Article  CAS  PubMed  Google Scholar 

  • Sebold M, Nebe S, Garbusow M, Guggenmos M, Schad DJ, Beck A, Kuitunen-Paul S, Sommer C, Frank R, Neu P, Zimmermann US, Rapp MA, Smolka MN, Huys QJM, Schlagenhauf F, Heinz A (2017) When habits are dangerous: alcohol expectancies and habitual decision making predict relapse in alcohol dependence. Biol Psychiatry 82:847–856

    Article  PubMed  Google Scholar 

  • Smith KS, Graybiel AM (2013) A dual operator view of habitual behavior reflecting cortical and striatal dynamics. Neuron 79(2):361–374

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Steinberg EE, Keiflin R, Boivin JR, Witten IB, Deisseroth K, Janak PH (2013) A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci 16(7):966–973

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Sutton R (1990) Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In: Proceedings of the seventh international conference on machine learning, vol 216, p 224

    Chapter  Google Scholar 

  • Sutton RS, Barto AG (1998) Reinforcement learning: an introduction (adaptive computation and machine learning). The MIT Press, Cambridge

    Google Scholar 

  • Sutton RS, Precup D, Singh S et al (1999) Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning. Artif Intell 112(1):181–211

    Article  Google Scholar 

  • Tobler PN, Fiorillo CD, Schultz W (2005) Adaptive coding of reward value by dopamine neurons. Science 307(5715):1642–1645

    Article  CAS  PubMed  Google Scholar 

  • Tolman EC (1948) Cognitive maps in rats and men. Psychol Rev 55(4):189–208

    Article  CAS  PubMed  Google Scholar 

  • Valentin VV, Dickinson A, O’Doherty JP (2007) Determining the neural substrates of goaldirected learning in the human brain. J Neurosci 27(15):4019–4026

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Voon V, Derbyshire K, Rück C, Irvine MA, Worbe Y, Enander J, Schreiber LRN, Gillan C, Fineberg NA, Sahakian BJ, Robbins TW, Harrison NA, Wood J, Daw ND, Dayan P, Grant JE, Bullmore ET (2015) Disorders of compulsivity: a common bias towards learning habits. Mol Psychiatry 20(3):345–352

    Article  CAS  PubMed  Google Scholar 

  • Waelti P, Dickinson A, Schultz W (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature 412(6842):43–48

    Article  CAS  PubMed  Google Scholar 

  • Watkins C, Dayan P (1992) Q-learning. Mach Learn 8(3):279–292

    Google Scholar 

  • Wunderlich K, Smittenaar P, Dolan RJ (2012) Dopamine enhances model-based over modelfree choice behavior. Neuron 75(3):418–424

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Yin HH, Knowlton BJ, Balleine BW (2004) Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur J Neurosci 19(1):181–189

    Article  PubMed  Google Scholar 

  • Yin HH, Ostlund SB, Knowlton BJ, Balleine BW (2005) The role of the dorsomedial striatum in instrumental conditioning. Eur J Neurosci 22(2):513–523

    Article  PubMed  Google Scholar 

  • Zaghloul KA, Blanco JA, Weidemann CT, McGill K, Jaggi JL, Baltuch GH, Kahana MJ (2009) Human substantia nigra neurons encode unexpected financial rewards. Science 323(5920):1496–1499

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Division of Psychiatry and Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, UK

    Quentin J. M. Huys

  2. Department of Psychiatry, Psychotherapy and Psychosomatics, Hospital of Psychiatry, University of Zürich, Zürich, Switzerland

    Quentin J. M. Huys

  3. Translational Neuromodeling Unit, Department of Biomedical Engineering, ETH Zürich and University of Zürich, Zürich, Switzerland

    Quentin J. M. Huys

  4. Institute of Adaptive and Neural Computation, University of Edinburgh, Scotland, UK

    Peggy Seriès

Authors
  1. Quentin J. M. Huys
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peggy Seriès
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Quentin J. M. Huys .

Editor information

Editors and Affiliations

  1. Atlanta, GA, USA

    Dieter Jaeger

  2. Department of Biomedical Engineering, Florida International University, Miami, FL, USA

    Ranu Jung

Section Editor information

  1. Institute “Carlos I” for Theoretical and Computational Physics and Department of Electromagnetism and Matter Physics, Facultad de Ciencias, Universidad de Granada, Granada, Granada, Spain

    Joaquin J. Torres

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Science+Business Media, LLC, part of Springer Nature

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Huys, Q.J.M., Seriès, P. (2019). Reward-Based Learning, Model-Based and Model-Free. In: Jaeger, D., Jung, R. (eds) Encyclopedia of Computational Neuroscience. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-7320-6_674-2

Download citation

  • DOI: https://doi.org/10.1007/978-1-4614-7320-6_674-2

  • Received:

  • Accepted:

  • Published:

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4614-7320-6

  • Online ISBN: 978-1-4614-7320-6

  • eBook Packages: Springer Reference Biomedicine and Life SciencesReference Module Biomedical and Life Sciences

Publish with us

Policies and ethics

Chapter history

  1. Latest

    Reward-Based Learning, Model-Based and Model-Free
    Published:
    15 August 2019

    DOI: https://doi.org/10.1007/978-1-4614-7320-6_674-2

  2. Original

    Reward-Based Learning, Model-Based and Model-Free
    Published:
    21 February 2014

    DOI: https://doi.org/10.1007/978-1-4614-7320-6_674-1

Search

Navigation