Chapter 3 - The role of learning-related dopamine signals in addiction vulnerability
Section snippets
Background
Humans have used alcohol and various kinds of drugs of abuse for thousands of years. The early Egyptians consumed wine and narcotics, and the first documented use of marijuana in China dates back to 2737 B.C. However, the recognition of addiction as a problem occurred relatively recently and developed gradually in the eighteenth and nineteenth centuries (e.g., see Thomas de Quincey's “Confessions of an Opium Eater,” 1821). The emergence of more potent formulations, better methods of delivery (
Model-free and Model-based Learning from Rewards
Choosing behaviors that maximize rewards and minimize losses in the longer term is the central problem that RL theory addresses. A difficulty in doing so is the appropriate balancing of short-term gains against long-term losses. Choices made now can have many different consequences tomorrow. The choice to enjoy another drink now may lead to social disinhibition and facilitate friendships or encounters, but it may also impair the ability to fulfill duties at work the next day, with more
Phasic Dopamine Signals Represent Model-free Prediction Errors
The neural bases of model-based learning are not very clear, with only few direct measurements of tree search available (Johnson and Redish, 2007, Pfeiffer and Foster, 2013, van der Meer and Redish, 2009). However, the neural representation of prediction-error signals as required for model-free learning has been examined in exacting detail (Montague et al., 1996, Schultz et al., 1997), and we turn to this evidence next. It focuses on the dopamine neurons of the ventral tegmental area (VTA) and,
Behavioral Characteristics of Model-free and Model-based Choices
Above we have seen that phasic dopamine signals covary with a TD prediction error. Henceforth, we will consider these signals as model-free. Model-free learning evaluates the total future reward by summing up the prediction errors over time into either or values. We briefly review several domains in which this has qualitative behavioral consequences that distinguish model-free from model-based choices.
Individual Variability
We have now reviewed model-based and model-free learning, the role of dopamine in model-free learning, and behavioral and neurobiological characteristics of both systems. Recent findings have highlighted substantial individual variability in how and what subjects learn in standard Pavlovian conditioning paradigms. This has consequences for learning accounts of addiction as some learning tendencies appear to confer vulnerability toward developing addiction. In this part, we first present the
Addiction
Addiction is a disorder with profound deficits in decision-making. Most addictive drugs have rapid effects and impact the dopaminergic system either directly or indirectly (Koob, 1992, Olds, 1956, Tsai et al., 2009). Several features of addiction are at least partially amenable to explanations within the overall framework outlined earlier. We will briefly consider partial accounts of addiction based on (a) drug-induced alterations to phasic dopaminergic signals and (b) individual (and
Acknowledgments
We would like to acknowledge financial support by the National Institute of Health (1P01DA03165601) to S. B. F., the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG) to Q. J. M. H. (FOR 1617: grant RA1047/2-1), and the Swiss National Science Foundation to G. H. (32003B 138264) and P. N. T. (PP00P1 128574 and CRSII3 141965).We thank Peter Dayan, Maria Garbusow, Rike Petzschner, and Terry Robinson for helpful comments and Katie Long for the drawings in Fig. 3A and D.
References (257)
- et al.
Prediction error as a linear function of reward probability is coded in human nucleus accumbens
Neuroimage
(2006) - et al.
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Neuron
(2005) - et al.
Ventral striatal activation during reward anticipation correlates with impulsivity in alcoholics
Biol. Psychiatry
(2009) - et al.
Novelty seeking, incentive salience and acquisition of cocaine self-administration in the rat
Behav. Brain Res.
(2011) - et al.
Cocaine seeking habits depend upon dopamine-dependent serial connectivity linking the ventral with the dorsal striatum
Neuron
(2008) Food reward: brain substrates of wanting and liking
Neurosci. Biobehav. Rev.
(1996)Motivation concepts in behavioral neuroscience
Physiol. Behav.
(2004)- et al.
Parsing reward
Trends Neurosci.
(2003) - et al.
Deep blue
Artif. Intell.
(2002) - et al.
Striatal dopamine responses to intranasal cocaine self-administration in humans
Biol. Psychiatry
(2009)
Personality, addiction, dopamine: insights from parkinson's disease
Neuron
Dopamine receptors in the learning, memory and drug reward circuitry
Semin. Cell Dev. Biol.
Dopamine, serotonin and impulsivity
Neuroscience
Model-based influences on humans’ choices and striatal prediction errors
Neuron
Reinforcement learning: the good, the bad and the ugly
Curr. Opin. Neurobiol.
Individual differences in the reinforcing and subjective effects of amphetamine and diazepam
Drug Alcohol Depend.
Functional imaging of the human dopaminergic midbrain
Trends Neurosci.
The neuropsychological basis of addictive behaviour
Brain Res. Brain Res. Rev.
Individual differences in the attribution of incentive salience to reward-related cues: implications for addiction
Neuropharmacology
A food predictive cue must be attributed with incentive salience for it to induce c-fos mrna expression in cortico-striatal-thalamic brain regions
Neuroscience
Antecedents and consequences of drug abuse in rats selectively bred for high and low response to novelty
Neuropharmacology
A role for dopamine in the processing of drug cues in heroin dependent patients
Eur. Neuropsychopharmacol.
Enhanced avoidance habits in obsessive-compulsive disorder
Biol. Psychiatry
States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning
Neuron
Severity of neuropsychological impairment in cocaine and alcohol addiction: association with metabolism in the prefrontal cortex
Neuropsychologia
Go and no-go learning in reward and punishment: interactions between affect and effect
Neuroimage
Multiple forms of value learning and the function of dopamine
Neuronlike elements that can solve difficult learning control problems
IEEE Trans. Syst. Man Cybern.
Statistics of midbrain dopamine neuron spike trains in the awake primate
J. Neurophysiol.
Dopamine modulates reward-related vigor
Neuropsychopharmacology
Dynamic Programming
Cocaine supersensitivity and enhanced motivation for reward in mice lacking dopamine d(2) autoreceptors
Nat. Neurosci.
The debate over dopamine's role in reward: the case for incentive salience
Psychopharmacology (Berl)
From prediction error to incentive salience: mesolimbic computation of reward motivation
Eur. J. Neurosci.
What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience?
Brain Res. Rev.
Neuro-Dynamic Programming
Performance on learning to associate a stimulus with positive reinforcement
Alcohol promotes dopamine release in the human nucleus accumbens
Synapse
Conditioned dopamine release in humans: a positron emission tomography [11c]raclopride study with amphetamine
J. Neurosci.
Context and behavioral processes in extinction
Learn. Mem.
Learning and Behavior: A Contemporary Synthesis
Persistent alterations in cognitive function and prefrontal dopamine d2 receptors following extended, but not limited, access to self-administered cocaine
Neuropsychopharmacology
Phasic excitation of dopamine neurons in ventral vta by noxious stimuli
Proc. Natl. Acad. Sci. U. S. A.
A pallidus-habenula-dopamine pathway signals inferred stimulus values
J. Neurophysiol.
Dopaminergic network differences in human impulsivity
Science
Neural mechanisms of observational learning
Proc. Natl. Acad. Sci. U.S.A
Effects of selective excitotoxic lesions of the nucleus accumbens core, anterior cingulate cortex, and central nucleus of the amygdala on autoshaping performance in rats
Behav. Neurosci.
Meta-analysis of cue-reactivity in addiction research
Addiction
Frontal theta overrides pavlovian learning biases
J. Neurosci.
Rescuing cocaine-induced prefrontal cortex hypoactivity prevents compulsive cocaine seeking
Nature
Cited by (70)
Recent Opioid Use Impedes Range Adaptation in Reinforcement Learning in Human Addiction
2024, Biological PsychiatryIndividual differences in learning positive affective value
2021, Current Opinion in Behavioral SciencesCitation Excerpt :While both phenotypes learn the predictive value of the CS, only sign-trackers assign it incentive value; for them, reward cues become ‘motivational magnets’ [39,13]. Furthermore, these behavioral differences seem to reflect specific computational strategies: while goal-trackers appear to engage cortical regions and model-based computations, sign-trackers appear to rely on a model-free, subcortical, dopamine-dependent form of learning [13,40,17]. Crucially, relative to goal-trackers, sign-trackers tend to present concomitant characteristics typically associated with compulsive reward-seeking behaviors such as addiction — namely, attentional deficits and personality traits such as novelty-seeking, risk-seeking and impulsivity [39,41,13].
Computational theory-driven studies of reinforcement learning and decision-making in addiction: what have we learned?
2021, Current Opinion in Behavioral SciencesDecision-making deficits in substance use disorders
2020, Cognition and Addiction: A Researcher's Guide from Mechanisms Towards InterventionsModel-Free and Model-Based Influences in Addiction-Related Behaviors
2019, Biological PsychiatryThe Necessity of a Trauma-Informed Paradigm in Substance Use Disorder Services
2023, Journal of the American Psychiatric Nurses Association