Chapter 3 - The role of learning-related dopamine signals in addiction vulnerability

https://doi.org/10.1016/B978-0-444-63425-2.00003-9Get rights and content

Abstract

Dopaminergic signals play a mathematically precise role in reward-related learning, and variations in dopaminergic signaling have been implicated in vulnerability to addiction. Here, we provide a detailed overview of the relationship between theoretical, mathematical, and experimental accounts of phasic dopamine signaling, with implications for the role of learning-related dopamine signaling in addiction and related disorders. We describe the theoretical and behavioral characteristics of model-free learning based on errors in the prediction of reward, including step-by-step explanations of the underlying equations. We then use recent insights from an animal model that highlights individual variation in learning during a Pavlovian conditioning paradigm to describe overlapping aspects of incentive salience attribution and model-free learning. We argue that this provides a computationally coherent account of some features of addiction.

Section snippets

Background

Humans have used alcohol and various kinds of drugs of abuse for thousands of years. The early Egyptians consumed wine and narcotics, and the first documented use of marijuana in China dates back to 2737 B.C. However, the recognition of addiction as a problem occurred relatively recently and developed gradually in the eighteenth and nineteenth centuries (e.g., see Thomas de Quincey's “Confessions of an Opium Eater,” 1821). The emergence of more potent formulations, better methods of delivery (

Model-free and Model-based Learning from Rewards

Choosing behaviors that maximize rewards and minimize losses in the longer term is the central problem that RL theory addresses. A difficulty in doing so is the appropriate balancing of short-term gains against long-term losses. Choices made now can have many different consequences tomorrow. The choice to enjoy another drink now may lead to social disinhibition and facilitate friendships or encounters, but it may also impair the ability to fulfill duties at work the next day, with more

Phasic Dopamine Signals Represent Model-free Prediction Errors

The neural bases of model-based learning are not very clear, with only few direct measurements of tree search available (Johnson and Redish, 2007, Pfeiffer and Foster, 2013, van der Meer and Redish, 2009). However, the neural representation of prediction-error signals as required for model-free learning has been examined in exacting detail (Montague et al., 1996, Schultz et al., 1997), and we turn to this evidence next. It focuses on the dopamine neurons of the ventral tegmental area (VTA) and,

Behavioral Characteristics of Model-free and Model-based Choices

Above we have seen that phasic dopamine signals covary with a TD prediction error. Henceforth, we will consider these signals as model-free. Model-free learning evaluates the total future reward by summing up the prediction errors over time into either VMFs or QMFsa values. We briefly review several domains in which this has qualitative behavioral consequences that distinguish model-free from model-based choices.

Individual Variability

We have now reviewed model-based and model-free learning, the role of dopamine in model-free learning, and behavioral and neurobiological characteristics of both systems. Recent findings have highlighted substantial individual variability in how and what subjects learn in standard Pavlovian conditioning paradigms. This has consequences for learning accounts of addiction as some learning tendencies appear to confer vulnerability toward developing addiction. In this part, we first present the

Addiction

Addiction is a disorder with profound deficits in decision-making. Most addictive drugs have rapid effects and impact the dopaminergic system either directly or indirectly (Koob, 1992, Olds, 1956, Tsai et al., 2009). Several features of addiction are at least partially amenable to explanations within the overall framework outlined earlier. We will briefly consider partial accounts of addiction based on (a) drug-induced alterations to phasic dopaminergic signals and (b) individual (and

Acknowledgments

We would like to acknowledge financial support by the National Institute of Health (1P01DA03165601) to S. B. F., the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG) to Q. J. M. H. (FOR 1617: grant RA1047/2-1), and the Swiss National Science Foundation to G. H. (32003B 138264) and P. N. T. (PP00P1 128574 and CRSII3 141965).We thank Peter Dayan, Maria Garbusow, Rike Petzschner, and Terry Robinson for helpful comments and Katie Long for the drawings in Fig. 3A and D.

References (257)

  • A. Dagher et al.

    Personality, addiction, dopamine: insights from parkinson's disease

    Neuron

    (2009)
  • J.W. Dalley et al.

    Dopamine receptors in the learning, memory and drug reward circuitry

    Semin. Cell Dev. Biol.

    (2009)
  • J.W. Dalley et al.

    Dopamine, serotonin and impulsivity

    Neuroscience

    (2012)
  • N.D. Daw et al.

    Model-based influences on humans’ choices and striatal prediction errors

    Neuron

    (2011)
  • P. Dayan et al.

    Reinforcement learning: the good, the bad and the ugly

    Curr. Opin. Neurobiol.

    (2008)
  • H. de Wit et al.

    Individual differences in the reinforcing and subjective effects of amphetamine and diazepam

    Drug Alcohol Depend.

    (1986)
  • E. Düzel et al.

    Functional imaging of the human dopaminergic midbrain

    Trends Neurosci.

    (2009)
  • B.J. Everitt et al.

    The neuropsychological basis of addictive behaviour

    Brain Res. Brain Res. Rev.

    (2001)
  • S.B. Flagel et al.

    Individual differences in the attribution of incentive salience to reward-related cues: implications for addiction

    Neuropharmacology

    (2009)
  • S.B. Flagel et al.

    A food predictive cue must be attributed with incentive salience for it to induce c-fos mrna expression in cortico-striatal-thalamic brain regions

    Neuroscience

    (2011)
  • S.B. Flagel et al.

    Antecedents and consequences of drug abuse in rats selectively bred for high and low response to novelty

    Neuropharmacology

    (2014)
  • I.H.A. Franken et al.

    A role for dopamine in the processing of drug cues in heroin dependent patients

    Eur. Neuropsychopharmacol.

    (2004)
  • C.M. Gillan et al.

    Enhanced avoidance habits in obsessive-compulsive disorder

    Biol. Psychiatry

    (2014)
  • J. Gläscher et al.

    States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning

    Neuron

    (2010)
  • R.Z. Goldstein et al.

    Severity of neuropsychological impairment in cocaine and alcohol addiction: association with metabolism in the prefrontal cortex

    Neuropsychologia

    (2004)
  • M. Guitart-Masip et al.

    Go and no-go learning in reward and punishment: interactions between affect and effect

    Neuroimage

    (2012)
  • B. Balleine et al.

    Multiple forms of value learning and the function of dopamine

  • A. Barto et al.

    Neuronlike elements that can solve difficult learning control problems

    IEEE Trans. Syst. Man Cybern.

    (1983)
  • H.M. Bayer et al.

    Statistics of midbrain dopamine neuron spike trains in the awake primate

    J. Neurophysiol.

    (2007)
  • U. Beierholm et al.

    Dopamine modulates reward-related vigor

    Neuropsychopharmacology

    (2013)
  • R.E. Bellman

    Dynamic Programming

    (1957)
  • E.P. Bello et al.

    Cocaine supersensitivity and enhanced motivation for reward in mice lacking dopamine d(2) autoreceptors

    Nat. Neurosci.

    (2011)
  • K.C. Berridge

    The debate over dopamine's role in reward: the case for incentive salience

    Psychopharmacology (Berl)

    (2007)
  • K.C. Berridge

    From prediction error to incentive salience: mesolimbic computation of reward motivation

    Eur. J. Neurosci.

    (2012)
  • K.C. Berridge et al.

    What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience?

    Brain Res. Rev.

    (1998)
  • D.P. Bertsekas et al.

    Neuro-Dynamic Programming

    (1996)
  • R. Boakes

    Performance on learning to associate a stimulus with positive reinforcement

  • I. Boileau et al.

    Alcohol promotes dopamine release in the human nucleus accumbens

    Synapse

    (2003)
  • I. Boileau et al.

    Conditioned dopamine release in humans: a positron emission tomography [11c]raclopride study with amphetamine

    J. Neurosci.

    (2007)
  • M.E. Bouton

    Context and behavioral processes in extinction

    Learn. Mem.

    (2004)
  • M.E. Bouton

    Learning and Behavior: A Contemporary Synthesis

    (2006)
  • L.A. Briand et al.

    Persistent alterations in cognitive function and prefrontal dopamine d2 receptors following extended, but not limited, access to self-administered cocaine

    Neuropsychopharmacology

    (2008)
  • F. Brischoux et al.

    Phasic excitation of dopamine neurons in ventral vta by noxious stimuli

    Proc. Natl. Acad. Sci. U. S. A.

    (2009)
  • E.S. Bromberg-Martin et al.

    A pallidus-habenula-dopamine pathway signals inferred stimulus values

    J. Neurophysiol.

    (2010)
  • J.W. Buckholtz et al.

    Dopaminergic network differences in human impulsivity

    Science

    (2010)
  • C.J. Burke et al.

    Neural mechanisms of observational learning

    Proc. Natl. Acad. Sci. U.S.A

    (2010)
  • R.N. Cardinal et al.

    Effects of selective excitotoxic lesions of the nucleus accumbens core, anterior cingulate cortex, and central nucleus of the amygdala on autoshaping performance in rats

    Behav. Neurosci.

    (2002)
  • B.L. Carter et al.

    Meta-analysis of cue-reactivity in addiction research

    Addiction

    (1999)
  • J. Cavanagh et al.

    Frontal theta overrides pavlovian learning biases

    J. Neurosci.

    (2013)
  • B.T. Chen et al.

    Rescuing cocaine-induced prefrontal cortex hypoactivity prevents compulsive cocaine seeking

    Nature

    (2013)
  • Cited by (70)

    • Individual differences in learning positive affective value

      2021, Current Opinion in Behavioral Sciences
      Citation Excerpt :

      While both phenotypes learn the predictive value of the CS, only sign-trackers assign it incentive value; for them, reward cues become ‘motivational magnets’ [39,13]. Furthermore, these behavioral differences seem to reflect specific computational strategies: while goal-trackers appear to engage cortical regions and model-based computations, sign-trackers appear to rely on a model-free, subcortical, dopamine-dependent form of learning [13,40,17]. Crucially, relative to goal-trackers, sign-trackers tend to present concomitant characteristics typically associated with compulsive reward-seeking behaviors such as addiction — namely, attentional deficits and personality traits such as novelty-seeking, risk-seeking and impulsivity [39,41,13].

    • Decision-making deficits in substance use disorders

      2020, Cognition and Addiction: A Researcher's Guide from Mechanisms Towards Interventions
    • The Necessity of a Trauma-Informed Paradigm in Substance Use Disorder Services

      2023, Journal of the American Psychiatric Nurses Association
    View all citing articles on Scopus
    View full text