Value normalization in decision making: theory and evidence
Highlights
► A key question is whether the brain computes value signals using a normalized code. ► Absolute value signals are independent of the value of other stimuli. ► Normalized value signals depend on their relative position in the distribution of values. ► We provide a framework for thinking about value normalization and evaluating the existing evidence.
Introduction
A rapidly growing and convergent body of evidence has shown that the brain computes several types of value-related signals during decision making (for reviews, see [1, 2, 3, 4, 5]). Three particularly important signals are stimulus values, outcome values, and prediction errors. Stimulus values (SV) are computed at the time of choice for the purpose of guiding decisions, and reflect the anticipated value of the outcomes associated with each option, regardless of whether or not the option is chosen. Neurophysiology [6••, 7, 8, 9], functional magnetic resonance imaging (fMRI) [10, 11, 12, 13, 14•, 15, 16, 17, 18, 19] and electroencephalography (EEG) [20] studies have found signals in orbitofrontal cortex (OFC) and ventromedial prefrontal cortex (vmPFC) consistent with the encoding of SV. Outcome values (OV, sometimes called experienced utility) indicate the value of consumption experiences, and measure their desirability. Activity consistent with the encoding of OV has also been found in similar areas of OFC and vmPFC [21, 22, 23, 24, 25, 26]. Prediction error (PE) signals are computed whenever individuals receive new information about their rewards, measure the change in expected rewards, and can be used to learn SV. PE signals have been most closely associated with the responses of midbrain dopamine neurons, which project to large segments of cortex [27, 28, 29, 30, 31, 32]. Other important signals include the net value of taking an action (action values [33]) and the values of chosen and unchosen options (for more details see [3, 4, 5]).
A basic question is whether the SV, OV, and PE signals are computed using an absolute or a normalized code. Under an absolute code, the neural response used to represent a given value is always the same. By contrast, under a normalized code, the neural response associated with that same value depends on its relative position in the distribution of values that might be encountered. For example, consider the response of a neuron that encodes SV when a subject is deciding whether or not to accept a lottery that pays $100 with 75% probability and entails a loss of $150 with 25% probability. In particular, compare the response of this neuron in two different decision contexts: a low reward context in which most other stimuli (e.g. other lotteries) encountered by the subject have much lower values (e.g. gains of $10 and losses of $15), and a high reward context in which most other stimuli have much higher values (e.g. gains of $1000 and losses of $1500). Under an absolute code, the firing rates in the neuron encoding the SV at the time of evaluating the lottery are the same in both reward contexts. By contrast, under the type of normalized codes described here, the firing rates in the neuron are higher in the lower reward context.
Several different motivations underlie the growing interest in value normalization. First, the presence and shape of a normalized value code has important behavioral implications. Consider a binary decision. The probability an individual chooses the item with the highest value is likely to be a function of the value of two options (Vleft and Vright, Figure 1a). If choices are a stochastic function of values (e.g. as described by a logistic choice model or by the drift-diffusion model [34, 35, 36, 37]), then the probability of choosing the left item increases with its relative value. Furthermore, under the type of value normalization schemes described below, the sensitivity of the choice curve to the relative values decreases as the range of values encountered increases during the choice task (Figure 1a.). In some settings, the psychometric choice curves are invariant to specific linear rescalings of the value of the options (i.e. multiplying all payoffs by a constant factor x > 0, Figure 1b), as a result of value normalization. Second, there is a growing belief in decision neuroscience that our ability to understand and predict choice will be greatly increased by understanding the detailed processes and mechanisms at work in value computation. We believe value normalization to be a crucial piece of the decision-making process since, as illustrated by the previous two examples, the quantitative behavioral predictions of value normalization depend on the exact functional form that it takes in different contexts.
This review provides a simple framework for thinking about value normalization, and uses it to evaluate the existing experimental evidence.
Section snippets
Normalization in sensory systems
Although the issue of normalization is relatively new to decision neuroscience, it has been widely investigated in sensory systems, where it has been found to be a pervasive feature of sensory coding (see [38••] for a recent review). Several convergent results and concepts from that literature provide a useful starting point for thinking about value normalization in decision making.
First, consider the problem of choosing a sensory coding scheme that maximizes the information contained in the
Computational framework for value normalization
In this section, we describe a simple decision-making task that dissociates SV, OV, and PE signals, and that provides a clean test of whether they exhibit absolute or normalized coding.
As shown in Figure 2a, subjects make choices in two different contexts (blue, green) that are held constant across long blocks of trials. In each context, the subject is shown one of three stimuli (triangle, square, star), with equal probability and has to choose between it and a fixed constant option (purple
Basic experimental tests of value normalization
A sizeable number of studies have attempted to understand how the brain normalizes SV, OV, and PE signals. Table 1 provides a detailed summary of the existing experimental evidence on value normalization, which includes monkey neurophysiology and human fMRI studies. Here, we highlight several recent macaque neurophysiology studies that have provided the strongest support to date for value normalization, as well as some studies that are subject to some of the concerns described in the previous
Tests of temporal properties of value normalization
A richer way to test a theory of value normalization is to also investigate how its normalization parameters evolve over time. It may be that different brain regions, such as those discussed in the previous section, have different time scales for normalization [47••]. For example, in ACC and OFC, does the distribution of values that gives shape to the normalization curve evolve quickly or slowly? Although this problem is only beginning to be investigated [6••, 47••, 55••], a natural hypothesis
Other mechanisms through which context affects decisions
Our focus has been on a very specific type of value normalization. Here, we discuss other mechanisms through which context can affect values and choices [56]. Importantly, we view these computations as distinct from value normalization, but highlight them because of their close relationship to our discussion of normalization.
Conclusion
The studies highlighted here provide support in favor of the hypothesis that neural representations of value are normalized based on the local distribution of values, that the functional form of the normalization is consistent with the efficient coding hypothesis, and that the normalization parameters are dynamically tuned. These findings point to several questions of critical importance for understanding how the brain computes subjective values.
First, despite clear evidence for value
References and recommended reading
Papers of particular interest, published within the period of review, have been highlighted as:
• of special interest
•• of outstanding interest
Acknowledgements
We would like to thank Wolfram Schultz for very useful comments. This research was supported by the NSF (SES-0851408, SES-0926544, SES-0850840), NIH (R01 AA018736, R21 AG038866), the Betty and Gordon Moore Foundation, and the Lipper Foundation.
References (74)
- et al.
Value, pleasure and choice in the ventral prefrontal cortex
Trends Cogn Sci
(2011) - et al.
Neural computations associated with goal-directed choice
Curr Opin Neurobiol
(2010) - et al.
Frontal cortex and reward-guided learning and decision-making
Neuron
(2011) - et al.
Heterogeneous reward signals in prefrontal cortex
Curr Opin Neurobiol
(2010) - et al.
Neural representation of subjective value under risk and ambiguity
J Neurophysiol
(2010) - et al.
Human cortical responses to water in the mouth, and the effects of thirst
J Neurophysiol
(2003) Multiple reward signals in the brain
Nat Rev Neurosci
(2000)- et al.
Temporal difference models and reward-related learning in the human brain
Neuron
(2003) - et al.
Neural computations underlying action-based decision making in the human brain
Proc Natl Acad Sci USA
(2009) - et al.
Effects of background illumination on the photoresponses of red and green cones
J Physiol
(1979)
A common mechanism for adaptive scaling of reward and novelty
Hum Brain Mapp
Relative reward preference in primate orbitofrontal cortex
Nature
Cognitive modulation of olfactory processing
Neuron
Neurons in the macaque orbitofrontal cortex code relative preference of both rewarding and aversive outcomes
Neurosci Res
Neurobiology of economic choice: a good-based model
Annu Rev Neurosci
Neuronal encoding of subjective value in dorsal and ventral anterior cingulate cortex
J Neurosci
Evaluating choices by single neurons in the frontal lobe: outcome value encoded across multiple decision variables
Eur J Neurosci
The representation of economic value in the orbitofrontal cortex is invariant for changes of menu
Nat Neurosci
Neurons in the orbitofrontal cortex encode economic value
Nature
Orbitofrontal cortex encodes willingness to pay in everyday economic transactions
J Neurosci
Appetitive and aversive goal values are encoded in the medial orbitofrontal cortex at the time of decision making
J Neurosci
Dissociating valuation and saliency signals during decision-making
Cereb Cortex
Evidence for a common representation of decision values for dissimilar goods in human ventromedial prefrontal cortex
J Neurosci
The decision value computations in the vmPFC and striatum use a relative value code that is guided by visual attention
J Neurosci
How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action
Neuron
Comparing apples and oranges: using reward-specific and reward-general subjective value representation in the brain
J Neurosci
The role of human orbitofrontal cortex in value comparison for incommensurable objects
J Neurosci
Economic choices can be made using only stimulus values
Proc Natl Acad Sci USA
Dynamic construction of stimulus values in the ventromedial prefrontal cortex
PLoS ONE
A common neural scale for the subjective pleasantness of different primary rewards
Neuroimage
Marketing actions can modulate neural representations of experienced pleasantness
Proc Natl Acad Sci USA
Dissociated neural representations of intensity and valence in human olfaction
Nat Neurosci
Activation of the human orbitofrontal cortex to a liquid food stimulus is correlated with its subjective pleasantness
Cereb Cortex
Dissociation of neural representation of intensity and affective valuation in human gustation
Neuron
Dopamine neurons report an error in the temporal prediction of reward during learning
Nat Neurosci
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Neuron
BOLD responses reflecting dopaminergic signals in the human ventral tegmental area
Science
Cited by (98)
Range adaptation in schizophrenia: A one-year longitudinal study
2024, Schizophrenia Research: CognitionRange adaptive value representations in schizophrenia and major depression
2024, Asian Journal of PsychiatryAltered delay discounting in neurodegeneration: insight into the underlying mechanisms and perspectives for clinical applications
2023, Neuroscience and Biobehavioral ReviewsContext-dependent outcome encoding in human reinforcement learning
2021, Current Opinion in Behavioral SciencesCitation Excerpt :A century later, the pervasiveness of perceptual illusions and decision-making biases, combined with decades of research in psychology, economics and neurosciences, consolidated the notion that perceptual and economic decisions are highly susceptible to contextual effects [3]. A significant fraction of contextual effects seems to result from two fundamental computations: reference-point centring and range adaptation [4–6]. In most ecological and real-life situations, decisions are arguably strongly influenced by the retrospective recollection of past outcomes experienced in similar situations [7].
Resource-rational decision making
2021, Current Opinion in Behavioral Sciences