Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Modulators of decision making

Abstract

Human and animal decisions are modulated by a variety of environmental and intrinsic contexts. Here I consider computational factors that can affect decision making and review anatomical structures and neurochemical systems that are related to contextual modulation of decision making. Expectation of a high reward can motivate a subject to go for an action despite a large cost, a decision that is influenced by dopamine in the anterior cingulate cortex. Uncertainty of action outcomes can promote risk taking and exploratory choices, in which norepinephrine and the orbitofrontal cortex appear to be involved. Predictable environments should facilitate consideration of longer-delayed rewards, which depends on serotonin in the dorsal striatum and dorsal prefrontal cortex. This article aims to sort out factors that affect the process of decision making from the viewpoint of reinforcement learning theory and to bridge between such computational needs and their neurophysiological substrates.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Standard models of evaluation of amount, delay and probability of reward4.
Figure 2: A hypothetical model of realization of reinforcement learning in the cortex–basal ganglia network2.
Figure 3: Possible links between computational factors and parameters of decision making and learning, and their neurobiological substrates.

Similar content being viewed by others

References

  1. Doya, K., Ishii, S., Pouget, A. & Rao, R. Bayesian Brain: Probabilistic Approach to Neural Coding and Learning (MIT Press, Cambridge, Massachusetts, USA, (2007).

    Google Scholar 

  2. Doya, K. Reinforcement learning: computational theory and biological mechanisms. HFSP J. 1, 30–40 (2007).

    Article  PubMed  PubMed Central  Google Scholar 

  3. Sutton, R.S. & Barto, A.G. Reinforcement Learning (MIT Press, Cambridge, Massachusetts, USA, (1998).

    Google Scholar 

  4. Ho, M.Y., Mobini, S., Chiang, T.J., Bradshaw, C.M. & Szabadi, E. Theory and method in the quantitative analysis of “impulsive choice” behaviour: implications for psychopharmacology. Psychopharmacology (Berl.) 146, 362–372 (1999).

    Article  CAS  Google Scholar 

  5. Berns, G.S., Laibson, D. & Loewenstein, G. Intertemporal choice–toward an integrative framework. Trends Cogn. Sci. 11, 482–488 (2007).

    Article  PubMed  Google Scholar 

  6. Laibson, D.I. Golden eggs and hyperbolic discounting. Q. J. Econ. 62, 443–477 (1997).

    Article  Google Scholar 

  7. von Neumann, J. & Morgenstern, O. Theory of Games and Economic Behavior (Princeton Univ. Press, Princeton, New Jersey, USA, (1944).

    Google Scholar 

  8. Kahneman, D. & Tversky, A. Prospect theory: an analysis of decision under risk. Econometrica 47, 263–291 (1979).

    Article  Google Scholar 

  9. Ishii, S., Yoshida, W. & Yoshimoto, J. Control of exploitation-exploration meta-parameter in reinforcement learning. Neural Netw. 15, 665–687 (2002).

    Article  PubMed  Google Scholar 

  10. Sugrue, L.P., Corrado, G.S. & Newsome, W.T. Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787 (2004).

    Article  CAS  PubMed  Google Scholar 

  11. Herrnstein, R.J. Relative and absolute strength of response as a function of frequency of reinforcement. J. Exp. Anal. Behav. 4, 267–272 (1961).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Baum, W.M. Optimization and the matching law as accounts of instrumental behavior. J. Exp. Anal. Behav. 36, 387–403 (1981).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Puterman, M.L. Markov Decision Processes: Discrete Dynamic Stochastic Programming (Wiley, New York, (1994).

    Book  Google Scholar 

  14. Balleine, B.W., Delgado, M.R. & Hikosaka, O. The role of the dorsal striatum in reward and decision-making. J. Neurosci. 27, 8161–8165 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Balleine, B.W. & Killcross, S. Parallel incentive processing: an integrated view of amygdala function. Trends Neurosci. 29, 272–279 (2006).

    Article  CAS  PubMed  Google Scholar 

  16. Kakade, S. & Dayan, P. Acquisition and extinction in autoshaping. Psychol. Rev. 109, 533–544 (2002).

    Article  PubMed  Google Scholar 

  17. Daw, N.D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711 (2005).

    Article  CAS  PubMed  Google Scholar 

  18. Niv, Y., Daw, N.D., Joel, D. & Dayan, P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl.) 191, 507–520 (2007).

    Article  CAS  Google Scholar 

  19. Schweighofer, N. et al. Humans can adopt optimal discounting strategy under real-time constraints. PLOS Comput. Biol. 2, e152 (2006).

    Article  Google Scholar 

  20. Doya, K. Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr. Opin. Neurobiol. 10, 732–739 (2000).

    Article  CAS  PubMed  Google Scholar 

  21. Matsumoto, K., Suzuki, W. & Tanaka, K. Neuronal correlates of goal-based motor selection in the prefrontal cortex. Science 301, 229–232 (2003).

    Article  CAS  PubMed  Google Scholar 

  22. Platt, M.L. & Glimcher, P.W. Neural correlates of decision variables in parietal cortex. Nature 400, 233–238 (1999).

    Article  CAS  PubMed  Google Scholar 

  23. Schultz, W., Tremblay, L. & Hollerman, J.R. Reward processing in primate orbitofrontal cortex and basal ganglia. Cereb. Cortex 10, 272–284 (2000).

    Article  CAS  PubMed  Google Scholar 

  24. Watanabe, M. Reward expectancy in primate prefrontal neurons. Nature 382, 629–632 (1996).

    Article  CAS  PubMed  Google Scholar 

  25. Kawagoe, R., Takikawa, Y. & Hikosaka, O. Expectation of reward modulates cognitive signals in the basal ganglia. Nat. Neurosci. 1, 411–416 (1998).

    Article  CAS  PubMed  Google Scholar 

  26. Samejima, K., Ueda, Y., Doya, K. & Kimura, M. Representation of action-specific reward values in the striatum. Science 310, 1337–1340 (2005).

    Article  CAS  PubMed  Google Scholar 

  27. Pasquereau, B. et al. Shaping of motor responses by incentive values through the basal ganglia. J. Neurosci. 27, 1176–1183 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Komura, Y., Tamura, R., Uwano, T., Nishijo, H. & Ono, T. Auditory thalamus integrates visual inputs into behavioral gains. Nat. Neurosci. 8, 1203–1209 (2005).

    Article  CAS  PubMed  Google Scholar 

  29. Minamimoto, T., Hori, Y. & Kimura, M. Complementary process to response bias in the centromedian nucleus of the thalamus. Science 308, 1798–1801 (2005).

    Article  CAS  PubMed  Google Scholar 

  30. Montague, P.R., Dayan, P. & Sejnowski, T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).

    Article  CAS  PubMed  Google Scholar 

  32. McClure, S.M., Berns, G.S. & Montague, P.R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 38, 339–346 (2003).

  33. O'Doherty, J. et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454 (2004).

    Article  CAS  PubMed  Google Scholar 

  34. O'Doherty, J.P., Dayan, P., Friston, K., Critchley, H. & Dolan, R.J. Temporal difference models and reward-related learning in the human brain. Neuron 38, 329–337 (2003).

    Article  CAS  PubMed  Google Scholar 

  35. Seymour, B. et al. Temporal difference models describe higher-order learning in humans. Nature 429, 664–667 (2004).

    Article  CAS  PubMed  Google Scholar 

  36. Reynolds, J.N. & Wickens, J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15, 507–521 (2002).

    Article  PubMed  Google Scholar 

  37. Wickens, J.R., Begg, A.J. & Arbuthnott, G.W. Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience 70, 1–5 (1996).

    Article  CAS  PubMed  Google Scholar 

  38. Doya, K. Metalearning and neuromodulation. Neural Netw. 15, 495–506 (2002).

    Article  PubMed  Google Scholar 

  39. Yacubian, J. et al. Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain. J. Neurosci. 26, 9530–9537 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  40. Belova, M.A., Paton, J.J., Morrison, S.E. & Salzman, C.D. Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala. Neuron 55, 970–984 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Seymour, B., Daw, N., Dayan, P., Singer, T. & Dolan, R. Differential encoding of losses and gains in the human striatum. J. Neurosci. 27, 4826–4831 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Satoh, T., Nakai, S., Sato, T. & Kimura, M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913–9923 (2003).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Daw, N.D., Kakade, S. & Dayan, P. Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603–616 (2002).

    Article  PubMed  Google Scholar 

  44. Matsumoto, M. & Hikosaka, O. Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447, 1111–1115 (2007).

    Article  CAS  PubMed  Google Scholar 

  45. Denk, F. et al. Differential involvement of serotonin and dopamine systems in cost-benefit decisions about delay or effort. Psychopharmacology (Berl.) 179, 587–596 (2005).

    Article  CAS  Google Scholar 

  46. Kuhnen, C.M. & Knutson, B. The neural basis of financial risk taking. Neuron 47, 763–770 (2005).

    Article  CAS  PubMed  Google Scholar 

  47. Tobler, P.N., O'Doherty, J.P., Dolan, R.J. & Schultz, W. Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J. Neurophysiol. 97, 1621–1632 (2007).

    Article  PubMed  Google Scholar 

  48. Cardinal, R.N. Neural systems implicated in delayed and probabilistic reinforcement. Neural Netw. 19, 1277–1301 (2006).

    Article  PubMed  Google Scholar 

  49. Tanaka, S.C. et al. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat. Neurosci. 7, 887–893 (2004).

    Article  CAS  PubMed  Google Scholar 

  50. Tanaka, S.C. et al. Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics. Neural Netw. 19, 1233–1241 (2006).

    Article  PubMed  Google Scholar 

  51. McClure, S.M., Laibson, D.I., Loewenstein, G. & Cohen, J.D. Separate neural systems value immediate and delayed monetary rewards. Science 306, 503–507 (2004).

    Article  CAS  PubMed  Google Scholar 

  52. McClure, S.M., Ericson, K.M., Laibson, D.I., Loewenstein, G. & Cohen, J.D. Time discounting for primary rewards. J. Neurosci. 27, 5796–5804 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  53. Kable, J.W. & Glimcher, P.W. The neural correlates of subjective value during intertemporal choice. Nat. Neurosci. 10, 1625–1633 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  54. Rudebeck, P.H., Walton, M.E., Smyth, A.N., Bannerman, D.M. & Rushworth, M.F. Separate neural pathways process different decision costs. Nat. Neurosci. 9, 1161–1168 (2006).

    Article  CAS  PubMed  Google Scholar 

  55. Winstanley, C.A., Theobald, D.E., Dalley, J.W., Cardinal, R.N. & Robbins, T.W. Double dissociation between serotonergic and dopaminergic modulation of medial prefrontal and orbitofrontal cortex during a test of impulsive choice. Cereb. Cortex 16, 106–114 (2006).

    Article  PubMed  Google Scholar 

  56. Tanaka, S.C. et al. Serotonin differentially regulates short- and long-term prediction of rewards in the ventral and dorsal striatum. PLoS ONE 2, e1333 (2007).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  57. Behrens, T.E., Woolrich, M.W., Walton, M.E. & Rushworth, M.F. Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214–1221 (2007).

    Article  CAS  PubMed  Google Scholar 

  58. Bouret, S. & Sara, S.J. Network reset: a simplified overarching theory of locus coeruleus noradrenaline function. Trends Neurosci. 28, 574–582 (2005).

    Article  CAS  PubMed  Google Scholar 

  59. Dayan, P. & Yu, A.J. Phasic norepinephrine: A neural interrupt signal for unexpected events. Network 17, 335–350 (2006).

    Article  PubMed  Google Scholar 

  60. Aston-Jones, G. & Cohen, J.D. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403–450 (2005).

    Article  CAS  PubMed  Google Scholar 

  61. Clarke, H.F., Dalley, J.W., Crofts, H.S., Robbins, T.W. & Roberts, A.C. Cognitive inflexibility after prefrontal serotonin depletion. Science 304, 878–880 (2004).

    Article  CAS  PubMed  Google Scholar 

  62. Floresco, S.B., Tse, M.T. & Ghods-Sharifi, S. Dopaminergic and glutamatergic regulation of effort- and delay-based decision making. Neuropsychopharmacology, published online 5 September 2007 (doi:10.1038/sj.npp.1301565).

    Article  PubMed  CAS  Google Scholar 

  63. Walton, M.E., Bannerman, D.M. & Rushworth, M.F. The role of rat medial frontal cortex in effort-based decision making. J. Neurosci. 22, 10996–11003 (2002).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  64. Rogers, R.D., Lancaster, M., Wakeley, J. & Bhagwagar, Z. Effects of beta-adrenoceptor blockade on components of human decision-making. Psychopharmacology (Berl.) 172, 157–164 (2004).

    Article  CAS  Google Scholar 

  65. Cardinal, R.N. & Howes, N.J. Effects of lesions of the nucleus accumbens core on choice between small certain rewards and large uncertain rewards in rats. BMC Neurosci. 6, 37 (2005).

  66. Winstanley, C.A., Dalley, J.W., Theobald, D.E. & Robbins, T.W. Fractionating impulsivity: contrasting effects of central 5-HT depletion on different measures of impulsive behavior. Neuropsychopharmacology 29, 1331–1343 (2004).

    Article  CAS  PubMed  Google Scholar 

  67. van Gaalen, M.M., van Koten, R., Schoffelmeer, A.N. & Vanderschuren, L.J. Critical involvement of dopaminergic neurotransmission in impulsive decision making. Biol. Psychiatry 60, 66–73 (2006).

    Article  CAS  PubMed  Google Scholar 

  68. Kheramin, S. et al. Effects of orbital prefrontal cortex dopamine depletion on inter-temporal choice: a quantitative analysis. Psychopharmacology (Berl.) 175, 206–214 (2004).

    Article  CAS  Google Scholar 

  69. Robinson, E.S. et al. Similar effects of the selective noradrenaline reuptake inhibitor atomoxetine on three distinct forms of impulsivity in the rat. Neuropsychopharmacology, published online 18 July 2007 (doi:10.1038/sj.npp.1301487).

    Article  PubMed  CAS  Google Scholar 

  70. Rudebeck, P.H., Buckley, M.J., Walton, M.E. & Rushworth, M.F. A role for the macaque anterior cingulate gyrus in social valuation. Science 313, 1310–1312 (2006).

    Article  CAS  PubMed  Google Scholar 

  71. Mobini, S. et al. Effects of lesions of the orbitofrontal cortex on sensitivity to delayed and probabilistic reinforcement. Psychopharmacology (Berl.) 160, 290–298 (2002).

    Article  CAS  Google Scholar 

  72. Kheramin, S. et al. Effects of quinolinic acid-induced lesions of the orbital prefrontal cortex on inter-temporal choice: a quantitative analysis. Psychopharmacology (Berl.) 165, 9–17 (2002).

    Article  CAS  Google Scholar 

  73. Kheramin, S. et al. The effect of orbital prefrontal cortex lesions on performance on a progressive ratio schedule: implications for models of inter-temporal choice. Behav. Brain Res. 156, 145–152 (2005).

    Article  CAS  PubMed  Google Scholar 

  74. Winstanley, C.A., Theobald, D.E., Cardinal, R.N. & Robbins, T.W. Contrasting roles of basolateral amygdala and orbitofrontal cortex in impulsive choice. J. Neurosci. 24, 4718–4722 (2004).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  75. Cardinal, R.N., Pennicott, D.R., Sugathapala, C.L., Robbins, T.W. & Everitt, B.J. Impulsive choice induced in rats by lesions of the nucleus accumbens core. Science 292, 2499–2501 (2001).

    Article  CAS  PubMed  Google Scholar 

  76. Pothuizen, H.H., Jongen-Relo, A.L., Feldon, J. & Yee, B.K. Double dissociation of the effects of selective nucleus accumbens core and shell lesions on impulsive-choice behaviour and salience learning in rats. Eur. J. Neurosci. 22, 2605–2616 (2005).

    Article  PubMed  Google Scholar 

  77. Hariri, A.R. et al. Preference for immediate over delayed rewards is associated with magnitude of ventral striatal activity. J. Neurosci. 26, 13213–13217 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  78. Kennerley, S.W., Walton, M.E., Behrens, T.E., Buckley, M.J. & Rushworth, M.F. Optimal decision making and the anterior cingulate cortex. Nat. Neurosci. 9, 940–947 (2006).

    Article  CAS  PubMed  Google Scholar 

  79. Yu, A.J. & Dayan, P. Uncertainty, neuromodulation, and attention. Neuron 46, 681–692 (2005).

    Article  CAS  PubMed  Google Scholar 

  80. Rogers, R.D. et al. Tryptophan depletion impairs stimulus-reward learning while methylphenidate disrupts attentional control in healthy young adults: implications for the monoaminergic basis of impulsive behaviour. Psychopharmacology (Berl.) 146, 482–491 (1999).

    Article  CAS  Google Scholar 

  81. Rogers, R.D. et al. Dissociable deficits in the decision-making cognition of chronic amphetamine abusers, opiate abusers, patients with focal damage to prefrontal cortex, and tryptophan-depleted normal volunteers: evidence for monoaminergic mechanisms. Neuropsychopharmacology 20, 322–339 (1999).

    Article  CAS  PubMed  Google Scholar 

  82. Chamberlain, S.R. et al. Neurochemical modulation of response inhibition and probabilistic learning in humans. Science 311, 861–863 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  83. Clarke, H.F., Walker, S.C., Dalley, J.W., Robbins, T.W. & Roberts, A.C. Cognitive inflexibility after prefrontal serotonin depletion is behaviorally and neurochemically specific. Cereb. Cortex 17, 18–27 (2007).

    Article  CAS  Google Scholar 

  84. van der Plasse, G. et al. Medial prefrontal serotonin in the rat is involved in goal-directed behaviour when affect guides decision making. Psychopharmacology (Berl.) 195, 435–449 (2007).

    Article  CAS  Google Scholar 

  85. Daw, N.D., O'Doherty, J.P., Dayan, P., Seymour, B. & Dolan, R.J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  86. Corbit, L.H. & Balleine, B.W. The role of prelimbic cortex in instrumental conditioning. Behav. Brain Res. 146, 145–157 (2003).

    Article  PubMed  Google Scholar 

  87. Balleine, B.W. Neural bases of food-seeking: affect, arousal and reward in corticostriatolimbic circuits. Physiol. Behav. 86, 717–730 (2005).

    Article  CAS  PubMed  Google Scholar 

  88. Rachlin, H., Raineri, A. & Cross, D. Subjective probability and delay. J. Exp. Anal. Behav. 55, 233–244 (1991).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The author thanks P. Dayan, N. Daw and N. Schweighofer for discussions on temporal discounting. This research was supported by a Grant-in-Aid for Scientific Research on Priority Areas, Ministry of Education, Culture, Sports, Science and Technology of Japan.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Kenji Doya.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Doya, K. Modulators of decision making. Nat Neurosci 11, 410–416 (2008). https://doi.org/10.1038/nn2077

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn2077

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing