Multiplicity of control in the basal ganglia: computational roles of striatal subregions
Highlights
► RL algorithms offer a more finely divided functional framework for striatum. ► Specifically, dorsomedial striatum and accumbens shell together support ‘model-based’ RL. ► RL models of BG are being combined with Bayesian models of perceptual decision-making.
Introduction
Perhaps more than any other brain areas, recent advances in the understanding of the basal ganglia (BG) have been driven by computational models. This is largely because of the fact that core functions commonly ascribed to the BG — action selection and value learning — have been the subject of intensive study in both economics and computer science, particularly the subfield of artificial intelligence known as reinforcement learning (RL) [1]. Theories from these areas propose mathematical definitions for quantities relevant to these functions and step-by-step procedures for computing them. Accordingly, these models have rapidly progressed from general frameworks for interpreting data toward playing a more integral quantitative role in experimental design and analysis, and now often serve as explicit hypotheses about trial-by-trial fluctuations in biological signals, such as action potentials or blood oxygenation level dependent (BOLD) signals. The poster children for this approach are the influential, albeit controversial, temporal difference (TD) learning models, which describe a reward prediction error (RPE) signal that has proved a strong match to phasic firing of midbrain dopamine neurons as well as BOLD in the ventral striatum (VS) [2, 3, 4, 5, 6, 7]. The present review considers recent work that has expanded upon this initial achievement, shedding further light on the computational and functional suborganization of striatum, and then considers questions raised by this work in light of other empirical data and computational modeling that, together, point the way for future work in this area.
Section snippets
The suborganization of striatum
Although anatomical considerations such as the topographical gradient of afferents from cortex to striatum have long suggested considerable functional heterogeneity (for instance, five distinct corticostriatal loops in one seminal review [8]), there have until recently been surprisingly few clear correlates of this presumptive suborganization in unit recordings or functional neuroimaging within BG. In parallel, there have been few functional subdivisions suggested among the dominant
Questions and anomalies
At the same time, many of these studies point to three serious questions for the RL models: on their overall architecture, their mechanisms for learning, and how they are deployed during choice.
References and recommended reading
Papers of particular interest, published within the period of review, have been highlighted as:
•• of outstanding interest
Acknowledgements
The authors are supported by a Scholar Award from the McKnight Foundation, a NARSAD Young Investigator Award, Human Frontiers Science Program Grant RGP0036/2009-C, and NIMH grant 1R01MH087882-01, part of the CRCNS program. We thank Fenna Krienen, Amitai Shenhav, Dylan Simon and Elliott Wimmer for helpful conversations.
References (90)
- et al.
Temporal difference models and reward-related learning in the human brain
Neuron
(2003) - et al.
Temporal prediction errors in a passive learning task activate human striatum
Neuron
(2003) - et al.
Modulation of caudate activity by action contingency
Neuron
(2004) What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?
Neural Netw
(1999)- et al.
A normative perspective on motivation
Trends Cogn Sci
(2006) - et al.
A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task
Neuroscience
(1999) - et al.
Integrating hippocampus and striatum in decision-making
Curr Opin Neurobiol
(2007) - et al.
Constructive episodic simulation of the future and the past: distinct subsystems of a core brain network mediate imagining and remembering
Neuropsychologia
(2009) - et al.
The integrative function of the basal ganglia in instrumental conditioning
Behav Brain Res
(2009) - Simon ND, Daw ND: Neural correlates of forward planning in a spatial decision task in humans. J Neurosci; in...
Forward and reverse hippocampal place-cell sequences during ripples
Nat Neurosci
Dynamic encoding of action selection by the medial striatum
J Neurosci
Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation
Nat Neurosci
Distinct tonic and phasic anticipatory activity in lateral habenula and dopamine neurons
Neuron
A neural computational model of incentive salience
PLoS Comput Biol
The evolving theory of basal forebrain functional-anatomical ‘macrosystems’
Neurosci Biobehav Rev
At the limbic-motor interface: disconnection of basolateral amygdala from nucleus accumbens core and shell reveals dissociable components of incentive motivation
Eur J Neurosci
Semi-rational models of conditioning: the case of trial order
Banburismus and the brain: decoding the relationship between sensory stimuli, decisions, and reward
Neuron
Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment
Science
Loss aversion, diminishing sensitivity, and the effect of experience on repeated decisions
J Behav Decis Mak
Caudate encodes multiple computations for perceptual decisions
J Neurosci
Reinforcement Learning: An Introduction
Adaptive critics and the basal ganglia
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
J Neurosc
Validation of decision-making models and analysis of decision variables in the rat basal ganglia
J Neurosci
Role of striatum in updating values of chosen actions
J Neurosci
Parallel organization of functionally segregated circuits linking basal ganglia and cortex
Ann N Y Acad Sci
Learning and memory functions of the basal ganglia
Annu Rev Neurosci
Effects of selective excitotoxic lesions of the nucleus accumbens core, anterior cingulate cortex, and central nucleus of the amygdala on autoshaping performance in rats
Behav Neurosci
Dissociable roles of ventral and dorsal striatum in instrumental conditioning
Science
A specific role for posterior dorsolateral striatum in human habit learning
Eur J Neurosci
Instrumental responding following reinforcer devaluation
Q J Exp Psychol Sect B
Variations in the sensitivity of instrumental responding to reinforcer devaluation
Q J Exp Psychol Sect B
Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control
Nat Neurosci
Addiction as vulnerabilities in the decision process
Behav Brain Sci
A framework for studying the neurobiology of value-based decision making
Nat Rev Neurosci
Multiple forms of value learning and the function of dopamine
Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation
Nat Neurosci
Two-factor theory, the actor/critic model, and conditioned avoidance
Learn Behav
Cognitive maps in rats and men
Psychol Rev
Goal-directed decision making in prefrontal cortex: a computational framework
Evidence for model-based action planning in a sequential finger movement task
J Motor Behav
Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning
Eur J Neurosci
The role of the dorsomedial striatum in instrumental conditioning
Eur J Neurosci
Cited by (79)
Circuit mechanisms of navigation strategy learning in mice
2024, Current BiologyEndocannabinoid modulation of dopamine release during reward seeking, interval timing, and avoidance
2021, Progress in Neuro-Psychopharmacology and Biological PsychiatryContributions of the basal ganglia to action sequence learning and performance
2019, Neuroscience and Biobehavioral ReviewsAn Analysis of Decision under Risk in Rats
2019, Current BiologyManipulating memory efficacy affects the behavioral and neural profiles of deterministic learning and decision-making
2018, NeuropsychologiaCitation Excerpt :Representations arising from striatal activity may influence discrimination decisions directly and may provide additional decision evidence with or without MTL-based contributions (Bornstein and Daw, 2012, 2013; Doll et al., 2015; Tremel et al., 2016). Bornstein and Daw (2011) demonstrated that different subsections of the striatum contribute to decision-making in different ways. In particular the dorsomedial striatum acts as a model-based reinforcement learning agent, wherein planning of actions and their dynamic representations drive learning.
The many worlds hypothesis of dopamine prediction error: implications of a parallel circuit architecture in the basal ganglia
2017, Current Opinion in Neurobiology