Value normalization in decision making: theory and evidence

https://doi.org/10.1016/j.conb.2012.07.011Get rights and content

A sizable body of evidence has shown that the brain computes several types of value-related signals to guide decision making, such as stimulus values, outcome values, and prediction errors. A critical question for understanding decision-making mechanisms is whether these value signals are computed using an absolute or a normalized code. Under an absolute code, the neural response used to represent the value of a given stimulus does not depend on what other values might have been encountered. By contrast, under a normalized code, the neural response associated with a given value depends on its relative position in the distribution of values. This review provides a simple framework for thinking about value normalization, and uses it to evaluate the existing experimental evidence.

Highlights

► A key question is whether the brain computes value signals using a normalized code. ► Absolute value signals are independent of the value of other stimuli. ► Normalized value signals depend on their relative position in the distribution of values. ► We provide a framework for thinking about value normalization and evaluating the existing evidence.

Introduction

A rapidly growing and convergent body of evidence has shown that the brain computes several types of value-related signals during decision making (for reviews, see [1, 2, 3, 4, 5]). Three particularly important signals are stimulus values, outcome values, and prediction errors. Stimulus values (SV) are computed at the time of choice for the purpose of guiding decisions, and reflect the anticipated value of the outcomes associated with each option, regardless of whether or not the option is chosen. Neurophysiology [6••, 7, 8, 9], functional magnetic resonance imaging (fMRI) [10, 11, 12, 13, 14•, 15, 16, 17, 18, 19] and electroencephalography (EEG) [20] studies have found signals in orbitofrontal cortex (OFC) and ventromedial prefrontal cortex (vmPFC) consistent with the encoding of SV. Outcome values (OV, sometimes called experienced utility) indicate the value of consumption experiences, and measure their desirability. Activity consistent with the encoding of OV has also been found in similar areas of OFC and vmPFC [21, 22, 23, 24, 25, 26]. Prediction error (PE) signals are computed whenever individuals receive new information about their rewards, measure the change in expected rewards, and can be used to learn SV. PE signals have been most closely associated with the responses of midbrain dopamine neurons, which project to large segments of cortex [27, 28, 29, 30, 31, 32]. Other important signals include the net value of taking an action (action values [33]) and the values of chosen and unchosen options (for more details see [3, 4, 5]).

A basic question is whether the SV, OV, and PE signals are computed using an absolute or a normalized code. Under an absolute code, the neural response used to represent a given value is always the same. By contrast, under a normalized code, the neural response associated with that same value depends on its relative position in the distribution of values that might be encountered. For example, consider the response of a neuron that encodes SV when a subject is deciding whether or not to accept a lottery that pays $100 with 75% probability and entails a loss of $150 with 25% probability. In particular, compare the response of this neuron in two different decision contexts: a low reward context in which most other stimuli (e.g. other lotteries) encountered by the subject have much lower values (e.g. gains of $10 and losses of $15), and a high reward context in which most other stimuli have much higher values (e.g. gains of $1000 and losses of $1500). Under an absolute code, the firing rates in the neuron encoding the SV at the time of evaluating the lottery are the same in both reward contexts. By contrast, under the type of normalized codes described here, the firing rates in the neuron are higher in the lower reward context.

Several different motivations underlie the growing interest in value normalization. First, the presence and shape of a normalized value code has important behavioral implications. Consider a binary decision. The probability an individual chooses the item with the highest value is likely to be a function of the value of two options (Vleft and Vright, Figure 1a). If choices are a stochastic function of values (e.g. as described by a logistic choice model or by the drift-diffusion model [34, 35, 36, 37]), then the probability of choosing the left item increases with its relative value. Furthermore, under the type of value normalization schemes described below, the sensitivity of the choice curve to the relative values decreases as the range of values encountered increases during the choice task (Figure 1a.). In some settings, the psychometric choice curves are invariant to specific linear rescalings of the value of the options (i.e. multiplying all payoffs by a constant factor x > 0, Figure 1b), as a result of value normalization. Second, there is a growing belief in decision neuroscience that our ability to understand and predict choice will be greatly increased by understanding the detailed processes and mechanisms at work in value computation. We believe value normalization to be a crucial piece of the decision-making process since, as illustrated by the previous two examples, the quantitative behavioral predictions of value normalization depend on the exact functional form that it takes in different contexts.

This review provides a simple framework for thinking about value normalization, and uses it to evaluate the existing experimental evidence.

Section snippets

Normalization in sensory systems

Although the issue of normalization is relatively new to decision neuroscience, it has been widely investigated in sensory systems, where it has been found to be a pervasive feature of sensory coding (see [38••] for a recent review). Several convergent results and concepts from that literature provide a useful starting point for thinking about value normalization in decision making.

First, consider the problem of choosing a sensory coding scheme that maximizes the information contained in the

Computational framework for value normalization

In this section, we describe a simple decision-making task that dissociates SV, OV, and PE signals, and that provides a clean test of whether they exhibit absolute or normalized coding.

As shown in Figure 2a, subjects make choices in two different contexts (blue, green) that are held constant across long blocks of trials. In each context, the subject is shown one of three stimuli (triangle, square, star), with equal probability and has to choose between it and a fixed constant option (purple

Basic experimental tests of value normalization

A sizeable number of studies have attempted to understand how the brain normalizes SV, OV, and PE signals. Table 1 provides a detailed summary of the existing experimental evidence on value normalization, which includes monkey neurophysiology and human fMRI studies. Here, we highlight several recent macaque neurophysiology studies that have provided the strongest support to date for value normalization, as well as some studies that are subject to some of the concerns described in the previous

Tests of temporal properties of value normalization

A richer way to test a theory of value normalization is to also investigate how its normalization parameters evolve over time. It may be that different brain regions, such as those discussed in the previous section, have different time scales for normalization [47••]. For example, in ACC and OFC, does the distribution of values that gives shape to the normalization curve evolve quickly or slowly? Although this problem is only beginning to be investigated [6••, 47••, 55••], a natural hypothesis

Other mechanisms through which context affects decisions

Our focus has been on a very specific type of value normalization. Here, we discuss other mechanisms through which context can affect values and choices [56]. Importantly, we view these computations as distinct from value normalization, but highlight them because of their close relationship to our discussion of normalization.

Conclusion

The studies highlighted here provide support in favor of the hypothesis that neural representations of value are normalized based on the local distribution of values, that the functional form of the normalization is consistent with the efficient coding hypothesis, and that the normalization parameters are dynamically tuned. These findings point to several questions of critical importance for understanding how the brain computes subjective values.

First, despite clear evidence for value

References and recommended reading

Papers of particular interest, published within the period of review, have been highlighted as:

  • • of special interest

  • •• of outstanding interest

Acknowledgements

We would like to thank Wolfram Schultz for very useful comments. This research was supported by the NSF (SES-0851408, SES-0926544, SES-0850840), NIH (R01 AA018736, R21 AG038866), the Betty and Gordon Moore Foundation, and the Lipper Foundation.

References (74)

  • N. Bunzeck et al.

    A common mechanism for adaptive scaling of reward and novelty

    Hum Brain Mapp

    (2010)
  • L. Tremblay et al.

    Relative reward preference in primate orbitofrontal cortex

    Nature

    (1999)
  • I.E. de Araujo et al.

    Cognitive modulation of olfactory processing

    Neuron

    (2005)
  • T. Hosokawa et al.

    Neurons in the macaque orbitofrontal cortex code relative preference of both rewarding and aversive outcomes

    Neurosci Res

    (2007)
  • C. Padoa-Schioppa

    Neurobiology of economic choice: a good-based model

    Annu Rev Neurosci

    (2011)
  • X. Cai et al.

    Neuronal encoding of subjective value in dorsal and ventral anterior cingulate cortex

    J Neurosci

    (2012)
  • S.W. Kennerley et al.

    Evaluating choices by single neurons in the frontal lobe: outcome value encoded across multiple decision variables

    Eur J Neurosci

    (2009)
  • C. Padoa-Schioppa et al.

    The representation of economic value in the orbitofrontal cortex is invariant for changes of menu

    Nat Neurosci

    (2008)
  • C. Padoa-Schioppa et al.

    Neurons in the orbitofrontal cortex encode economic value

    Nature

    (2006)
  • H. Plassmann et al.

    Orbitofrontal cortex encodes willingness to pay in everyday economic transactions

    J Neurosci

    (2007)
  • H. Plassmann et al.

    Appetitive and aversive goal values are encoded in the medial orbitofrontal cortex at the time of decision making

    J Neurosci

    (2010)
  • A. Litt et al.

    Dissociating valuation and saliency signals during decision-making

    Cereb Cortex

    (2011)
  • V.S. Chib et al.

    Evidence for a common representation of decision values for dissimilar goods in human ventromedial prefrontal cortex

    J Neurosci

    (2009)
  • S.L. Lim et al.

    The decision value computations in the vmPFC and striatum use a relative value code that is guided by visual attention

    J Neurosci

    (2011)
  • E.D. Boorman et al.

    How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action

    Neuron

    (2009)
  • D.J. Levy et al.

    Comparing apples and oranges: using reward-specific and reward-general subjective value representation in the brain

    J Neurosci

    (2011)
  • T.H. FitzGerald et al.

    The role of human orbitofrontal cortex in value comparison for incommensurable objects

    J Neurosci

    (2009)
  • K. Wunderlich et al.

    Economic choices can be made using only stimulus values

    Proc Natl Acad Sci USA

    (2010)
  • A. Harris et al.

    Dynamic construction of stimulus values in the ventromedial prefrontal cortex

    PLoS ONE

    (2011)
  • F. Grabenhorst et al.

    A common neural scale for the subjective pleasantness of different primary rewards

    Neuroimage

    (2010)
  • H. Plassmann et al.

    Marketing actions can modulate neural representations of experienced pleasantness

    Proc Natl Acad Sci USA

    (2008)
  • A.K. Anderson et al.

    Dissociated neural representations of intensity and valence in human olfaction

    Nat Neurosci

    (2003)
  • M.L. Kringelbach et al.

    Activation of the human orbitofrontal cortex to a liquid food stimulus is correlated with its subjective pleasantness

    Cereb Cortex

    (2003)
  • D.M. Small et al.

    Dissociation of neural representation of intensity and affective valuation in human gustation

    Neuron

    (2003)
  • J.R. Hollerman et al.

    Dopamine neurons report an error in the temporal prediction of reward during learning

    Nat Neurosci

    (1998)
  • H.M. Bayer et al.

    Midbrain dopamine neurons encode a quantitative reward prediction error signal

    Neuron

    (2005)
  • K. D’Ardenne et al.

    BOLD responses reflecting dopaminergic signals in the human ventral tegmental area

    Science

    (2008)
  • Cited by (98)

    • Context-dependent outcome encoding in human reinforcement learning

      2021, Current Opinion in Behavioral Sciences
      Citation Excerpt :

      A century later, the pervasiveness of perceptual illusions and decision-making biases, combined with decades of research in psychology, economics and neurosciences, consolidated the notion that perceptual and economic decisions are highly susceptible to contextual effects [3]. A significant fraction of contextual effects seems to result from two fundamental computations: reference-point centring and range adaptation [4–6]. In most ecological and real-life situations, decisions are arguably strongly influenced by the retrospective recollection of past outcomes experienced in similar situations [7].

    • Resource-rational decision making

      2021, Current Opinion in Behavioral Sciences
    View all citing articles on Scopus
    View full text