Encyclopedia of Computational Neuroscience

2015 Edition
| Editors: Dieter Jaeger, Ranu Jung

Decision-Making, Models

Reference work entry
DOI: https://doi.org/10.1007/978-1-4614-6675-8_312

Definition

Models of decision making attempt to describe, using stochastic differential equations which represent either neural activity or more abstract psychological variables, the dynamical process that produces a commitment to a single action/outcome as a result of incoming evidence that can be ambiguous as to the action it supports.

Detailed Description

Background

Decision making can be separated into four processes (Doya 2008):
  1. 1.

    Acquisition of sensory information to determine the state of the environment and the organism within it

     
  2. 2.

    Evaluation of potential actions (options) in terms of the cost and benefit to the organism given its belief about the current state

     
  3. 3.

    Selection of an action based on, ideally, an optimal trade-off between the costs and benefits

     
  4. 4.

    Use of the outcome of the action to update the costs and benefits associated with it

     

Models of the dynamics of decision making have focused on perceptual decisions with only two possible responses available. The...

This is a preview of subscription content, log in to check access

References

  1. Balci F, Simen P, Niyogi R, Saxe A, Hughes JA, Holmes P, Cohen JD (2011) Acquisition of decision making criteria: reward rate ultimately beats accuracy. Atten Percept Psychophys 73:640–657PubMedCentralPubMedGoogle Scholar
  2. Barto AG (1994) Reinforcement learning control. Curr Opin Neurobiol 4:888–893PubMedGoogle Scholar
  3. Barto AG, Mahadevan S (2003) Recent advances in hierarchical reinforcement learning. Discrete Event Dyn Syst Theory Appl 13:343–379Google Scholar
  4. Beck JM, Ma WJ, Kiani R, Hanks T, Churchland AK, Roitman J, Shadlen MN, Latham PE, Pouget A (2008) Probabilistic population codes for Bayesian decision making. Neuron 60:1142–1152PubMedCentralPubMedGoogle Scholar
  5. Bellman R (1957) Dynamic programming. Princeton University Press, PrincetonGoogle Scholar
  6. Bogacz R, Brown E, Moehlis J, Holmes P, Cohen JD (2006) The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol Rev 113:700–765PubMedGoogle Scholar
  7. Bogacz R, McClure SM, Li J, Cohen JD, Montague PR (2007a) Short-term memory traces for action bias in human reinforcement learning. Brain Res 1153:111–121PubMedGoogle Scholar
  8. Bogacz R, Usher M, Zhang J, McClelland JL (2007b) Extending a biologically inspired model of choice: multi-alternatives, nonlinearity and value-based multidimensional choice. Philos Trans R Soc Lond B Biol Sci 362:1655–1670PubMedCentralPubMedGoogle Scholar
  9. Botvinick MM (2012) Hierarchical reinforcement learning and decision making. Curr Opin Neurobiol 22:956–962PubMedGoogle Scholar
  10. Brunton BW, Botvinick MM, Brody CD (2013) Rats and humans can optimally accumulate evidence for decision-making. Science 340:95–98PubMedGoogle Scholar
  11. Cain N, Shea-Brown E (2012) Computational models of decision making: integration, stability, and noise. Curr Opin Neurobiol 22:1047–1053PubMedGoogle Scholar
  12. Churchland AK, Ditterich J (2012) New advances in understanding decisions among multiple alternatives. Curr Opin Neurobiol 22:920–926PubMedCentralPubMedGoogle Scholar
  13. Churchland AK, Kiani R, Shadlen MN (2008) Decision-making with multiple alternatives. Nat Neurosci 11:693–702PubMedCentralPubMedGoogle Scholar
  14. Cisek P, Puskas GA, El-Murr S (2009) Decisions in changing conditions: the urgency-gating model. J Neurosci 29:11560–11571PubMedGoogle Scholar
  15. Daw ND, Doya K (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16:199–204PubMedGoogle Scholar
  16. Dayan P, Daw ND (2008) Decision theory, reinforcement learning, and the brain. Cogn Affect Behav Neurosci 8:429–453PubMedGoogle Scholar
  17. Dayan P, Niv Y (2008) Reinforcement learning: the good, the bad and the ugly. Curr Opin Neurobiol 18:185–196PubMedGoogle Scholar
  18. Deco G, Rolls ET (2005) Attention, short-term memory, and action selection: a unifying theory. Prog Neurobiol 76:236–256PubMedGoogle Scholar
  19. Deneve S (2012) Making decisions with unknown sensory reliability. Front Neurosci 6:75PubMedCentralPubMedGoogle Scholar
  20. Ditterich J (2010) A comparison between mechanisms of multi-alternative perceptual decision making: ability to explain human behavior, predictions for neurophysiology, and relationship with decision theory. Front Neurosci 4:184PubMedCentralPubMedGoogle Scholar
  21. Doya K (2008) Modulators of decision making. Nat Neurosci 11:410–416PubMedGoogle Scholar
  22. Drugowitsch J, Moreno-Bote R, Churchland AK, Shadlen MN, Pouget A (2012) The cost of accumulating evidence in perceptual decision making. J Neurosci Off J Soc Neurosci 32:3612–3628Google Scholar
  23. Furman M, Wang XJ (2008) Similarity effect and optimal control of multiple-choice decision making. Neuron 60:1153–1168PubMedCentralPubMedGoogle Scholar
  24. Gillespie DT (1992) Markov processes: an introduction for physical scientists. Academic, San DiegoGoogle Scholar
  25. Glimcher PW (2001) Making choices: the neurophysiology of visual-saccadic decision making. Trends Neurosci 24:654–659PubMedGoogle Scholar
  26. Glimcher PW (2003) The neurobiology of visual-saccadic decision making. Annu Rev Neurosci 26:133–179PubMedGoogle Scholar
  27. Gold JI, Shadlen MN (2001) Neural computations that underlie decisions about sensory stimuli. Trends Cogn Sci 5:10–16PubMedGoogle Scholar
  28. Gold JI, Shadlen MN (2007) The neural basis of decision making. Annu Rev Neurosci 30:535–574PubMedGoogle Scholar
  29. Hanks TD, Mazurek ME, Kiani R, Hopp E, Shadlen MN (2011) Elapsed decision time affects the weighting of prior probability in a perceptual decision task. J Neurosci Off J Soc Neurosci 31:6339–6352Google Scholar
  30. Huk AC, Shadlen MN (2005) Neural activity in macaque parietal cortex reflects temporal integration of visual motion signals during perceptual decision making. J Neurosci Off J Soc Neurosci 25:10420–10436Google Scholar
  31. Ito M, Doya K (2011) Multiple representations and algorithms for reinforcement learning in the cortico-basal ganglia circuit. Curr Opin Neurobiol 21:368–373PubMedGoogle Scholar
  32. Izhikevich EM (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17:2443–2452PubMedGoogle Scholar
  33. Joel D, Niv Y, Ruppin E (2002) Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw Off J Int Neural Netw Soc 15:535–547Google Scholar
  34. Johnson A, van der Meer MA, Redish AD (2007) Integrating hippocampus and striatum in decision-making. Curr Opin Neurobiol 17:692–697PubMedCentralPubMedGoogle Scholar
  35. Lawler GF (2006) Introduction to stochastic processes. Chapman & Hall/CRC, Boca RatonGoogle Scholar
  36. Lee D, Seo H (2007) Mechanisms of reinforcement learning and decision making in the primate dorsolateral prefrontal cortex. Ann N Y Acad Sci 1104:108–122PubMedGoogle Scholar
  37. Ludwig CJ, Davies JR (2011) Estimating the growth of internal evidence guiding perceptual decisions. Cogn Psychol 63:61–92PubMedGoogle Scholar
  38. Machens CK, Romo R, Brody CD (2005) Flexible control of mutual inhibition: a neural model of two-interval discrimination. Science 307:1121–1124PubMedGoogle Scholar
  39. Miller P, Katz DB (2013) Accuracy and response-time distributions for decision-making: linear perfect integrators versus nonlinear attractor-based neural circuits. J Comput Neurosci 35:261–294Google Scholar
  40. Miller P, Wang XJ (2006) Discrimination of temporally separated stimuli by integral feedback control. Proc Natl Acad Sci U S A 103:201–206PubMedCentralPubMedGoogle Scholar
  41. Newsome WT, Britten KH, Movshon JA (1989) Neuronal correlates of a perceptual decision. Nature 341:52–54PubMedGoogle Scholar
  42. Niwa M, Ditterich J (2008) Perceptual decisions between multiple directions of visual motion. J Neurosci Off J Soc Neurosci 28:4435–4445Google Scholar
  43. Ratcliff R (1978) A theory of memory retrieval. Psychol Rev 85:59–108Google Scholar
  44. Ratcliff R (2002) A diffusion model account of response time and accuracy in a brightness discrimination task: fitting real data and failing to fit fake but plausible data. Psychon Bull Rev 9:278–291PubMedGoogle Scholar
  45. Ratcliff R, Hasegawa YT, Hasegawa RP, Smith PL, Segraves MA (2007) Dual diffusion model for single-cell recording data from the superior colliculus in a brightness-discrimination task. J Neurophysiol 97:1756–1774PubMedCentralPubMedGoogle Scholar
  46. Ratcliff R, McKoon G (2008) The diffusion decision model: theory and data for two-choice decision tasks. Neural Comput 20:873–922PubMedCentralPubMedGoogle Scholar
  47. Ratcliff R, Smith PL (2004) A comparison of sequential sampling models for two-choice reaction time. Psychol Rev 111:333–367PubMedCentralPubMedGoogle Scholar
  48. Redish AD, Jensen S, Johnson A, Kurth-Nelson Z (2007) Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol Rev 114:784–805PubMedGoogle Scholar
  49. Romo R, Salinas E (2003) Flutter discrimination: neural codes, perception, memory and decision making. Nat Rev Neurosci 4:203–218PubMedGoogle Scholar
  50. Rorie AE, Gao J, McClelland JL, Newsome WT (2010) Integration of sensory and reward information during perceptual decision-making in lateral intraparietal cortex (LIP) of the macaque monkey. PLoS ONE 5:e9308PubMedCentralPubMedGoogle Scholar
  51. Rüter J, Marcille N, Sprekeler H, Gerstner W, Herzog MH (2012) Paradoxical evidence integration in rapid decision processes. PLoS Comput Biol 8:e1002382PubMedCentralPubMedGoogle Scholar
  52. Salinas E (2004) Fast remapping of sensory stimuli onto motor actions on the basis of contextual modulation. J Neurosci 24:1113–1118PubMedGoogle Scholar
  53. Seymour B, O'Doherty JP, Dayan P, Koltzenburg M, Jones AK, Dolan RJ, Friston KJ, Frackowiak RS (2004) Temporal difference models describe higher-order learning in humans. Nature 429:664–667PubMedGoogle Scholar
  54. Shadlen MN, Newsome WT (2001) Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesus monkey. J Neurophysiol 86:1916–1936PubMedGoogle Scholar
  55. Shankar S, Massoglia DP, Zhu D, Costello MG, Stanford TR, Salinas E (2011) Tracking the temporal evolution of a perceptual judgment using a compelled-response task. J Neurosci Off J Soc Neurosci 31:8406–8421Google Scholar
  56. Shea-Brown E, Gilzenrat MS, Cohen JD (2008) Optimization of decision making in multilayer networks: the role of locus coeruleus. Neural Comput 20:2863–2894PubMedGoogle Scholar
  57. Simen P, Contreras D, Buck C, Hu P, Holmes P, Cohen JD (2009) Reward rate optimization in two-alternative decision making: empirical tests of theoretical predictions. J Exp Psychol Hum Percept Perform 35:1865–1897PubMedCentralPubMedGoogle Scholar
  58. Smith PL, Ratcliff R (2004) Psychology and neurobiology of simple decisions. Trends Neurosci 27:161–168PubMedGoogle Scholar
  59. Soltani A, Wang XJ (2006) A biophysically-based neural model of matching law behavior: melioration by stochastic synapses. J Neurosci 26:3731–3744PubMedGoogle Scholar
  60. Soltani A, Wang XJ (2008) From biophysics to cognition: reward-dependent adaptive choice behavior. Curr Opin Neurobiol 18:209–216PubMedGoogle Scholar
  61. Soltani A, Wang XJ (2010) Synaptic computation underlying probabilistic inference. Nat Neurosci 13:112–119PubMedCentralPubMedGoogle Scholar
  62. Stanford TR, Shankar S, Massoglia DP, Costello MG, Salinas E (2010) Perceptual decision making in less than 30 milliseconds. Nat Neurosci 13:379–385PubMedCentralPubMedGoogle Scholar
  63. Sugrue LP, Corrado GS, Newsome WT (2005) Choosing the greater of two goods: neural currencies for valuation and decision making. Nat Rev Neurosci 6:363–375PubMedGoogle Scholar
  64. Thura D, Beauregard-Racine J, Fradet CW, Cisek P (2012) Decision making by urgency gating: theory and experimental support. J Neurophysiol 108:2912–2930PubMedGoogle Scholar
  65. Usher M, McClelland JL (2001) The time course of perceptual choice: the leaky, competing accumulator model. Psychol Rev 108:550–592PubMedGoogle Scholar
  66. Wald A (1947) Sequential analysis. Wiley, New YorkGoogle Scholar
  67. Wald A, Wolfowitz J (1948) Optimum character of the sequential probability ratio test. Ann Math Stat 19:326–339Google Scholar
  68. Wang XJ (2002) Probabilistic decision making by slow reverberation in cortical circuits. Neuron 36:955–968PubMedGoogle Scholar
  69. Wang XJ (2008) Decision making in recurrent neuronal circuits. Neuron 60:215–234PubMedCentralPubMedGoogle Scholar
  70. Wong KF, Wang XJ (2006) A recurrent network mechanism of time integration in perceptual decisions. J Neurosci Off J Soc Neurosci 26:1314–1328Google Scholar
  71. Wyart V, de Gardelle V, Scholl J, Summerfield C (2012) Rhythmic fluctuations in evidence accumulation during decision making in the human brain. Neuron 76:847–858PubMedCentralPubMedGoogle Scholar
  72. Zhou X, Wong-Lin K, Philip H (2009) Time-varying perturbations can distinguish among integrate-to-threshold models for perceptual decision making in reaction time tasks. Neural Comput 21:2336–2362PubMedCentralPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Department of Biology and Brandeis UniversityWalthamUSA
  2. 2.Volen National Center for Complex SystemsWalthamUSA