Multiplicity of control in the basal ganglia: computational roles of striatal subregions

https://doi.org/10.1016/j.conb.2011.02.009Get rights and content

The basal ganglia, in particular the striatum, are central to theories of behavioral control, and often identified as a seat of action selection. Reinforcement learning (RL) models  which have driven much recent experimental work on this region  cast striatum as a dynamic controller, integrating sensory and motivational information to construct efficient and enriching behavioral policies. Befitting this informationally central role, the BG sit at the nexus of multiple anatomical ‘loops’ of synaptic projections, connecting a wide range of cortical and subcortical structures. Numerous pioneering anatomical studies conducted over the past several decades have meticulously catalogued these loops, and labeled them according to the inferred functions of the connected regions. The specific cotermina of the projections are highly localized to several different subregions of the striatum, leading to the suggestion that these subregions perform complementary but distinct functions. However, until recently, the dominant computational framework outlined only a bipartite, dorsal/ventral, division of striatum. We review recent computational and experimental advances that argue for a more finely fractionated delineation. In particular, experimental data provide extensive insight into unique functions subserved by the dorsomedial striatum (DMS). These functions appear to correspond well with theories of a ‘model-based’ RL subunit, and may also shed light on the suborganization of ventral striatum. Finally, we discuss the limitations of these ideas and how they point the way toward future refinements of neurocomputational theories of striatal function, bringing them into contact with other areas of computational theory and other regions of the brain.

Highlights

► RL algorithms offer a more finely divided functional framework for striatum. ► Specifically, dorsomedial striatum and accumbens shell together support ‘model-based’ RL. ► RL models of BG are being combined with Bayesian models of perceptual decision-making.

Introduction

Perhaps more than any other brain areas, recent advances in the understanding of the basal ganglia (BG) have been driven by computational models. This is largely because of the fact that core functions commonly ascribed to the BG  action selection and value learning  have been the subject of intensive study in both economics and computer science, particularly the subfield of artificial intelligence known as reinforcement learning (RL) [1]. Theories from these areas propose mathematical definitions for quantities relevant to these functions and step-by-step procedures for computing them. Accordingly, these models have rapidly progressed from general frameworks for interpreting data toward playing a more integral quantitative role in experimental design and analysis, and now often serve as explicit hypotheses about trial-by-trial fluctuations in biological signals, such as action potentials or blood oxygenation level dependent (BOLD) signals. The poster children for this approach are the influential, albeit controversial, temporal difference (TD) learning models, which describe a reward prediction error (RPE) signal that has proved a strong match to phasic firing of midbrain dopamine neurons as well as BOLD in the ventral striatum (VS) [2, 3, 4, 5, 6, 7]. The present review considers recent work that has expanded upon this initial achievement, shedding further light on the computational and functional suborganization of striatum, and then considers questions raised by this work in light of other empirical data and computational modeling that, together, point the way for future work in this area.

Section snippets

The suborganization of striatum

Although anatomical considerations such as the topographical gradient of afferents from cortex to striatum have long suggested considerable functional heterogeneity (for instance, five distinct corticostriatal loops in one seminal review [8]), there have until recently been surprisingly few clear correlates of this presumptive suborganization in unit recordings or functional neuroimaging within BG. In parallel, there have been few functional subdivisions suggested among the dominant

Questions and anomalies

At the same time, many of these studies point to three serious questions for the RL models: on their overall architecture, their mechanisms for learning, and how they are deployed during choice.

References and recommended reading

Papers of particular interest, published within the period of review, have been highlighted as:

  • •• of outstanding interest

Acknowledgements

The authors are supported by a Scholar Award from the McKnight Foundation, a NARSAD Young Investigator Award, Human Frontiers Science Program Grant RGP0036/2009-C, and NIMH grant 1R01MH087882-01, part of the CRCNS program. We thank Fenna Krienen, Amitai Shenhav, Dylan Simon and Elliott Wimmer for helpful conversations.

References (90)

  • K. Diba et al.

    Forward and reverse hippocampal place-cell sequences during ripples

    Nat Neurosci

    (2007)
  • E.Y. Kimchi et al.

    Dynamic encoding of action selection by the medial striatum

    J Neurosci

    (2009)
  • M. Frank et al.

    Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation

    Nat Neurosci

    (2009)
  • E.S. Bromberg-Martin et al.

    Distinct tonic and phasic anticipatory activity in lateral habenula and dopamine neurons

    Neuron

    (2010)
  • J. Zhang et al.

    A neural computational model of incentive salience

    PLoS Comput Biol

    (2009)
  • D.S. Zahm

    The evolving theory of basal forebrain functional-anatomical ‘macrosystems’

    Neurosci Biobehav Rev

    (2006)
  • M. Shiflett et al.

    At the limbic-motor interface: disconnection of basolateral amygdala from nucleus accumbens core and shell reveals dissociable components of incentive motivation

    Eur J Neurosci

    (2010)
  • N. Daw et al.

    Semi-rational models of conditioning: the case of trial order

  • J. Gold et al.

    Banburismus and the brain: decoding the relationship between sensory stimuli, decisions, and reward

    Neuron

    (2002)
  • P. Berkes et al.

    Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment

    Science

    (2011)
  • I. Erev et al.

    Loss aversion, diminishing sensitivity, and the effect of experience on repeated decisions

    J Behav Decis Mak

    (2008)
  • L. Ding et al.

    Caudate encodes multiple computations for perceptual decisions

    J Neurosci

    (2010)
  • R. Sutton et al.

    Reinforcement Learning: An Introduction

    (1998)
  • A.C. Barto

    Adaptive critics and the basal ganglia

  • P.R. Montague et al.

    A framework for mesencephalic dopamine systems based on predictive Hebbian learning

    J Neurosc

    (1996)
  • M. Ito et al.

    Validation of decision-making models and analysis of decision variables in the rat basal ganglia

    J Neurosci

    (2009)
  • H. Kim et al.

    Role of striatum in updating values of chosen actions

    J Neurosci

    (2009)
  • G.E. Alexander et al.

    Parallel organization of functionally segregated circuits linking basal ganglia and cortex

    Ann N Y Acad Sci

    (1986)
  • M. Packard et al.

    Learning and memory functions of the basal ganglia

    Annu Rev Neurosci

    (2002)
  • R. Cardinal et al.

    Effects of selective excitotoxic lesions of the nucleus accumbens core, anterior cingulate cortex, and central nucleus of the amygdala on autoshaping performance in rats

    Behav Neurosci

    (2002)
  • J.P. O’Doherty et al.

    Dissociable roles of ventral and dorsal striatum in instrumental conditioning

    Science

    (2004)
  • E. Tricomi et al.

    A specific role for posterior dorsolateral striatum in human habit learning

    Eur J Neurosci

    (2009)
  • C. Adams et al.

    Instrumental responding following reinforcer devaluation

    Q J Exp Psychol Sect B

    (1981)
  • C. Adams

    Variations in the sensitivity of instrumental responding to reinforcer devaluation

    Q J Exp Psychol Sect B

    (1982)
  • N.D. Daw et al.

    Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control

    Nat Neurosci

    (2005)
  • A. Redish et al.

    Addiction as vulnerabilities in the decision process

    Behav Brain Sci

    (2008)
  • A. Rangel et al.

    A framework for studying the neurobiology of value-based decision making

    Nat Rev Neurosci

    (2008)
  • B. Balleine et al.

    Multiple forms of value learning and the function of dopamine

  • M.J. Frank et al.

    Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation

    Nat Neurosci

    (2009)
  • T. Maia

    Two-factor theory, the actor/critic model, and conditioned avoidance

    Learn Behav

    (2010)
  • E.C. Tolman

    Cognitive maps in rats and men

    Psychol Rev

    (1948)
  • M. Botvinick et al.

    Goal-directed decision making in prefrontal cortex: a computational framework

  • A. Fermin et al.

    Evidence for model-based action planning in a sequential finger movement task

    J Motor Behav

    (2010)
  • H.H. Yin et al.

    Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning

    Eur J Neurosci

    (2004)
  • H.H. Yin et al.

    The role of the dorsomedial striatum in instrumental conditioning

    Eur J Neurosci

    (2005)
  • Cited by (79)

    • Endocannabinoid modulation of dopamine release during reward seeking, interval timing, and avoidance

      2021, Progress in Neuro-Psychopharmacology and Biological Psychiatry
    • Manipulating memory efficacy affects the behavioral and neural profiles of deterministic learning and decision-making

      2018, Neuropsychologia
      Citation Excerpt :

      Representations arising from striatal activity may influence discrimination decisions directly and may provide additional decision evidence with or without MTL-based contributions (Bornstein and Daw, 2012, 2013; Doll et al., 2015; Tremel et al., 2016). Bornstein and Daw (2011) demonstrated that different subsections of the striatum contribute to decision-making in different ways. In particular the dorsomedial striatum acts as a model-based reinforcement learning agent, wherein planning of actions and their dynamic representations drive learning.

    View all citing articles on Scopus
    View full text