A Neural Circuit Mechanism for the Involvements of Dopamine in Effort-Related Choices: Decay of Learned Values, Secondary Effects of Depletion, and Calculation of Temporal Difference Error

Simulation of the DA depletion experiments and motivation for considering the secondary effects

A representative experimental paradigm to test roles of DA in effort-related choices is the T-maze task (Salamone et al., 1994; Cousins et al., 1996), which consisted of three conditions (Fig. 1A). In Condition 1, large reward was placed in one arm (HD, high-reinforcement-density arm), whereas small reward was placed in the other arm (LD, low-reinforcement-density arm), and a physical barrier was placed only in the HD arm. Intact rats preferred the HD (i.e., high-cost, large-reward) arm. However, after DA was depleted by intra-accumbens injection of 6-hydroxydopamine (6-OHDA), the rats changed their preferences to the LD (i.e., low-cost, small-reward) arm (Fig. 1B, orange-backed bars). In Condition 2, in which there was no barrier in the HD arm, DA depletion hardly changed the preference for the HD arm (Fig. 1B, blue-backed bars). In Condition 3, where the LD arm did not contain any reward, DA depletion mildly weakened the preference for the HD arm, but only transiently (Fig. 1D, pink-backed bars). In all three conditions, whether in the presence or absence of a preference change, DA depletion increased the latency of start-door opening (Fig. 1C,E), although this effect was also transient. These results, together with the results in a different paradigm (Salamone et al., 1991; Cousins and Salamone, 1994), have been interpreted that DA specifically serves for reward-oriented effort exertion rather than reward evaluation or effort exertion per se (Salamone et al., 2007).

First, we simulated the effort-related T-maze choice task (Salamone et al., 1994; Cousins et al., 1996) by the model considered in our previous study (Kato and Morita, 2016). The model describes this task as a set of states, each of which represents a particular location in the maze (Fig. 2A). The simulated subject selects Go action to go to a next state or Stay action to stay at the same state depending on the learned values of actions. The value of taken action is updated according to TD-RPE, while all the learned values are subject to a mild decay (Fig. 2B). DA depletion was assumed to cause a reduction of the size of nonnegative TD-RPE–dependent value increment to a quarter of the original size (TD-RPE was always nonnegative in the simulations shown in Fig. 3). Fig. 3 shows the simulation results on the ratio of choosing the HD arm (panels b) and the latency (number of time steps) of reaching the T-junction (State 4; panels c) in the three conditions [the results for Condisions 1 and 2 were already reported in Kato and Morita (2016)]. In Condition 1, DA depletion drastically changed the preference for the HD arm to the LD arm (Fig. 3Ab). By contrast, in Condition 2, DA depletion did not largely change the preference (Fig. 3Bb). These results are consistent with the experimental results (Fig. 1B), as shown in our previous study (Kato and Morita, 2016). In Condition 3, however, although DA depletion caused only a weak and transient reduction in the preference for the HD arm in the experiment (Fig. 1D), a more prominent and persistent decrease was caused in the simulation (Fig. 3Cb). Moreover, regarding the latency, although DA depletion caused only a transient increase in all the three conditions in the experiments (Fig. 1C,E), a persistent increase was caused in the simulations [Fig. 3, panels c; as previously shown for Conditions 1 and 2 (Kato and Morita, 2016)]. In this way, the previously considered model could reproduce some results, but not others, of the experiments.

We explored how the discrepancy between the experimental and simulation results could be resolved by extending the model. It has been shown that DA depletion causes an increase in the neural activity in the pedunculopontine nucleus (PPN; Breit et al., 2001; Zhang et al., 2008; Geng et al., 2016), where separate populations of neurons were shown to represent obtained reward and expected reward values (Okada et al., 2009). Either or both populations have been proposed to contribute to the calculation of TD-RPE, in DA neurons by providing the information of the obtained reward [R(t); Kawato and Samejima, 2007; Okada et al., 2009; Morita et al., 2012], the value of previous action/state (–Q_Previous; Cohen et al., 2012), and/or the value of upcoming action/state (Q_Upcoming; Kawato and Samejima, 2007; Okada et al., 2009) via direct excitatory projections and/or indirect projections. In reference to these findings and suggestions, we considered extended models in which the gain of one or more terms of TD-RPE was assumed to gradually increase after DA depletion. DA depletion was assumed to cause a reduction of the size of TD-RPE–dependent value-update to a quarter of the original size (i) only when TD-RPE was nonnegative, and in separate sets of simulations, (ii) regardless of whether TD-RPE was nonnegative or negative. In the simulations shown in Figs. 3, 5, 6, 7, 8A–C, Dd–g, and S1 and the gray lines in Fig. 15, TD-RPE was always nonnegative and thus results for i and those for ii should be the same; practically, results for i were used to plot Fig. 3, 5, 6, 7, and 8A–C, Dd–g , whereas results for ii were used to plot Fig. S1 and the gray lines in Fig. 15. In the simulations shown in Figs. 8Dbc and 9, results for ii are shown (results for i can be obtained by using the codes uploaded in the ModelDB).

• Download figure
• Open in new tab
• Download powerpoint

Figure 5.

Results of the simulations of the effort-related T-maze choice task with the assumption that the gain of the obtained-reward-representing input to DA neurons gradually increases up to twice of the original after DA depletion. A, Assumed gradual increase in the gain of the obtained-reward-representing input to DA neurons (the middle and the right graphs are identical to one another). B–D, Simulation results for Conditions 1, 2, and 3. The configurations are the same as those in Fig. 3. a, Schematics of the simulated task conditions. b, The ratio of choosing the HD arm. c, The latency (number of time steps) of reaching the T-junction (State 4).

• Download figure
• Open in new tab
• Download powerpoint

Figure 6.

Evolutions of the learned values of actions in the simulations shown in Fig. 5. A, Assumed gradual increase in the gain of the obtained-reward-representing input to DA neurons (the middle graph is a magnification of the right graph). B–D, Results for Conditions 1, 2, and 3. a, Schematics of the simulated task conditions. b–e, The color arrows and circles along the T-shape indicate the learned values of Go actions and Stay actions, respectively, for the last 10 trials of 500 (b), 550 (c), 625 (d), and 700 (e) trials, averaged across 20 simulations, in reference to the bottom color scale bar. f, Trial-by-trial evolutions of the learned values of Go_3→4 (pink line), Stay_3→3 (purple line), Go_4→5 (thick black line), and Go_4→6 (thick gray line) averaged across 20 simulations.

• Download figure
• Open in new tab
• Download powerpoint

Figure 7.

Detailed evolutions of the action values, the effective TD-RPEs, the ratio of choosing the HD arm, and the latency of reaching the T-junction after DA depletion in Condition 1. A, Left: Schematics of the simulated task condition. Right: Trial-by-trial evolutions of the learned values of Go_4→5 (black solid line), Go_4→6 (gray line), and Go_5→7 (black dotted line) for 480∼600th trials (i.e., from 20 trials before DA depletion to 100 trials after depletion) averaged across 20 simulations. The black and gray solid lines plot the same results that were shown in Fig. 6Bf with magnification. B, Effective TD-RPE (i.e., post-DA-depletion TD-RPE was multiplied by 0.25, which was the assumed factor for size reduction of value increment due to DA depletion) after taking Go_4→6 (left), Go_4→5 (middle), and Go_5→7 (right). The top panels show the results of individual simulation runs, where the white-red color indicates the magnitude of the effective TD-RPEs in reference to the rightmost color bar, and the black indicates the trials in which the corresponding action was not taken. The bottom panels show the trial-by-trial average of the effective TD-RPEs across simulations where the corresponding action was taken. The black thick line and the gray thin lines indicate the mean ± SEM of those simulations. C, The ratio of choosing the HD arm (a), the latency (number of time steps) of reaching the T-junction (State 4; b), and the mean effective TD-RPE per trial averaged over each successive 10 trials (c) until the 2000th trials in the cases where the obtained-reward-representing input to DA neurons gradually increases up to 2, 2.5, 3, 3.5, or 4 times of the original after DA depletion. The black thick line and the gray thin lines indicate the mean ± SEM of 20 simulations, and the vertical dotted line indicates the onset of DA depletion.

• Download figure
• Open in new tab
• Download powerpoint

Figure 8.

Results of the simulations with different assumptions. A, Results with the assumption that the gain of the obtained-reward-representing input to DA neurons (black arrows in a) gradually increases up to three times of the original after DA depletion. Panels b–g show the ratio of choosing the HD arm (b, d, f) and the latency (number of time steps) of reaching the T-junction (State 4; c, e, g) in Conditions 1 (b, c), 2 (d, e), and 3 (f, g). The black thick line and the gray thin lines indicate the mean ± SEM of 20 simulations. The same configurations are used in B–D, except that in C, 5 (b, c), 2 (d, e), and 2 (f, g) simulation runs where action values became extremely large (action value larger than 100 times of the size of the large reward was detected) were omitted from the calculation of the mean and SEM. B, Results with the assumption that the gain of all the three inputs that constitute the TD-RPE gradually increases up to three times of the original after DA depletion. C, Results with the assumption that the gain of the obtained-reward-representing input and the upcoming-value-representing input gradually increases up to three times of the original after DA depletion. D, Results with the assumption that the gain of the obtained-reward-representing input and the previous-value-representing input gradually increases up to three times of the original after DA depletion.

• Download figure
• Open in new tab
• Download powerpoint

Figure 9.

Systematic exploration of the possible secondary effects of DA depletion that best explain the experimental results. A, a, The gains of the inputs representing the obtained reward, upcoming value, and previous value were assumed to increase up to x, y, and z times, respectively, after DA depletion, and simulations of Conditions 1, 2, and 3 were conducted with x, y, and z systematically varied. b, Criteria set to evaluate the simulation results by considering four features: for the ratio of choosing the HD arm (left panel), (1) >0.1 decrease during 501–550th trials and (2) >0.5 decrease during 901–1000th trials, and for the latency (number of time steps) of reaching the T-junction (State 4; right), (3) >0.5 increase during 501–550th trials and (4) <0.5 increase during 901–1000th trials (on average across trials and completed simulation runs). Simulation results were regarded as similar to the experimental results if the abovementioned features—1, 2, 3, and 4 were (1,1,1,1), (0,0,1,1), and (1,0,1,1), where 1 and 0 represent satisfied and unsatisfied—for Conditions 1, 2, and 3, respectively. B, Number of features (1–4 above across the three conditions, in total 12) that were not satisfied in each set of simulation results with particular x (five panels), y (horizontal axes), and z (vertical axes).

Simulation assuming the post-depletion increase of PPN input representing the obtained reward

We first considered a model in which the gain of the obtained-reward term [R(t)] selectively increases after DA depletion (Fig. 4), in accordance with some proposals (Morita et al., 2012). Specifically, we assumed that the gain gradually increases for 200 trials after DA depletion up to twice of the original, and then reaches a plateau (Fig. 4C). This time course and the plateau level were determined in reference to experimental literature (see Materials and Methods for details). Fig. 5 shows the results of simulations of the T-maze experiments (Salamone et al., 1994; Cousins et al., 1996) by using this model. As shown in the figure, the experimental results that DA depletion drastically changed the preference in Condition 1 but not in Condition 2 were reproduced by this model, as well (Fig. 5Bb,Cb). The extended model also reproduced the weak, transient reduction in the preference for the HD arm in Condition 3 (Fig. 5Db), as well as the transient increase and the subsequent decrease in the latency (Fig. 5B–D, panels c) although the decrease was less prominent compared with the experimental results.

To understand the mechanisms of how the extended model could reproduce the experimental results, we looked at the evolutions of the action values (Fig. 6). In Condition 1, the Go_4→5 value was higher than the Go_4→6 value before DA depletion (Fig. 6Bb). However, shortly after DA depletion (Fig. 6Bc), the Go_4→5 value severely degraded and became smaller (although slightly) than the Go_4→6 value (see also Fig. 6Bf), explaining the drastic change in the choice preference (Fig. 5Bb). This occurred because Go_4→5 was separated from the rewarded state (State 7) and thus its value suffered the effect of DA depletion doubly. Specifically, the Go_4→5 value was updated according to TD-RPE that contained the Go_5→7 value, which was also updated according to TD-RPE, and both of these TD-RPE–dependent updates were affected by DA depletion. Indeed, although TD-RPE after taking Go_5→7 should benefit from the increase in the gain of the obtained-reward term, such a benefit could not immediately, nor fully, compensate for the depletion effect, and thus the Go_5→7 value decreased after depletion (Fig. 7A, right, black dotted line). Therefore, update of the Go_4→5 value suffered from this decrease of the Go_5→7 value, as well as the direct effect of depletion on TD-RPE after taking Go_4→5, resulting in the even more severe decrease than the Go_5→7 value (Fig. 7A, right, black solid line). In contrast, Go_4→6 was next to the rewarded state (State 6) and so its value suffered the effect of DA depletion only singly. In fact, the Go_4→6 value increased sometime after DA depletion (Fig. 7A right, gray line). This is because the severe decrease of the Go_4→5 value caused less frequent choices of Go_4→5 (i.e., the HD arm) and in turn more frequent choices of Go_4→6 (the LD arm), and thereby the Go_4→6 value became more frequently updated according to TD-RPE so that the balance between value-update and value-decay was shifted.

Fig. 7B shows effective TD-RPE, i.e., TD-RPE whose post-DA-depletion part was multiplied by one-quarter (because DA depletion was assumed to reduce the size of TD-RPE–dependent value update to one-quarter, as mentioned above) after taking Go_4→6, Go_4→5, or Go_5→7. As shown in the right panel, the effective TD-RPE after taking Go_5→7 once decreased after DA depletion, but subsequently increased again. This subsequent increase occurs because of a combination of the increase in the gain of the obtained-reward term and the decrease of the Go_5→7 value, i.e., increase in the gap between the Go_5→7 value and the obtained-reward term. Notably, although the effective TD-RPE after taking Go_5→7 turned to increase, the Go_5→7 value continued to decrease as seen above (Fig. 7A), because the frequency that Go_5→7 was taken (i.e., the frequency of HD choice) went down (Fig. 5Bb; also in Fig. 7B, top panels) and the balance between value-update and value-decay shifted. In this way, the value-decay critically underlies the preference reversal: indeed, preference reversal did not occur when the decay rate was set to 0.001 instead of 0.01 [data not shown, but can be seen in the ModelDB (Fig. S1)].

In Condition 2 (Fig. 6C), the decrease of the Go_4→5 value after DA depletion was much less prominent because Go_4→5 was next to the rewarded state, explaining that the preference did not largely change (Fig. 5Cb). In Condition 3 (Fig. 6D), Go_4→5 was again separated from the rewarded state, and so the Go_4→5 value severely degraded after DA depletion. However, this time the Go_4→5 value remained larger than the Go_4→6 value, which was 0 because Go_4→6 led to no reward (Fig. 6Df), explaining that the preference for the HD arm was weakened but not reversed to the LD arm (Fig. 5Db). Then, as the gain of the obtained-reward-representing input to DA neurons gradually increased as assumed, the values of Go_5→7 and Go_4→5 also gradually increased (Fig. 6Dc–e). The difference between the values of Go_4→5 and Go_4→6 thereby increased again (Fig. 6Df), explaining that the preference for the HD arm eventually recovered (Fig. 5Db).

Next, to consider the latency to reach the T-junction (State 4), we looked at the values of actions from the start to State 4. Before DA depletion, there were large value-contrasts between Go (arrows in Fig. 6Bb,Cb,Db) and Stay (circles) in all the three conditions (see, e.g., Go_3→4 and Stay_3→3 in Fig. 6Bb,Cb,Db; see also the pink and purple lines in Fig. 6Bf,Cf,Df). However, these value-contrasts degraded shortly after DA depletion, as shown in the figures. This degradation should cause an increase of the probability to choose Stay, and thereby explains the increase of the latency (Fig. 5Bc,Cc,Dc). Subsequently, as the gain of the obtained-reward term increased, value-contrasts between Go and Stay became reshaped, explaining the subsequent decrease of the latency. This mechanism suggested that the insufficient prominence in the subsequent latency decrease in the simulation results compared with the experimental results could be resolved if the gain of the obtained-reward term was further increased.

This was indeed confirmed, as shown in Fig. 7Cb. Specifically, when the gain increased up to 3 or 3.5 times of the original, the average latency of reaching the T-junction for 991∼1000th trials, as well as the average for 1991∼2000th trials, did not significantly differ from the average for 491∼500th trials (paired t test; p > 0.1). On the other hand, reversal of the preference in the arm choice could still occur in these cases (Fig. 7Ca), although it did not occur in some simulation runs, resulting in the relatively large standard errors. Fig. 7Cc shows the mean effective TD-RPE per trial averaged over each successive 10 trials (see Discussion).

Simulation with different assumptions on the secondary effects of DA depletion

Next, we considered models assuming post-depletion increase of the gain of both upcoming-value and previous-value terms (Q_Upcoming – Q_Previous), or either the upcoming-value term (Q_Upcoming) or the previous-value term (–Q_Previous), in addition to the obtained-reward term. As a reference, Fig. 8A shows the case in which the gain of only the obtained-reward term increased up to three times of the original (different simulation runs with the same assumptions as those shown in Fig. 7C). Fig. 8B shows the results of simulations assuming the gain increase of all the three terms of TD-RPE. As shown in Fig. 8Bb, in Condition 1, DA depletion once drastically decreased the preference for the HD arm, but subsequently the preference increased again. This is inconsistent with the experimental results (Fig. 1B,D).

Fig. 8C shows the results of simulations assuming the gain increase of the obtained-reward and upcoming-value terms. With this assumption, in some of the simulation runs, action values became extremely large (action value larger than 100 times of the size of the large reward was detected) and simulation was quitted (5, 2, and 2 runs of 20 runs in Conditions 1, 2, and 3, respectively). Even in the other simulation runs, action values became quite large. Occurrence of such an inflation of action values was actually expected because, with this assumption, the upcoming-value term, whose gain increased after DA depletion, could not be well canceled out by the previous-value term, whose gain remained unchanged. The choice and latency patterns in the simulations (Fig. 8Cb–g) significantly deviated from the experimental results: the extremely short latency after DA depletion is considered to reflect the inflation of action (Go) values.

Fig. 8D shows the results of simulations assuming the gain increase of the obtained-reward and previous-value terms. The choice and latency patterns (Fig. 8Db–g) look somewhat similar to those observed in the simulations without assuming secondary effects (Fig. 3). In particular, the DA depletion-induced increase in the latency (Fig. 8Dc,e,g), as well as the decrease in the preference for the HD arm in Condition 3 (Fig. 8Df), were persistent rather than transient as observed in the experiments (Fig. 1C–E).

As so far shown, the experimentally observed behavioral results of DA depletion in the T-maze experiments could be reproduced when the gain of only the obtained-reward term increased after DA depletion but not when the other assumptions were made. To more systematically explore the possible secondary effects of DA depletion that best explain the experimental results, next we assumed that the gains of the inputs representing the obtained reward, upcoming value, and previous value increased up to x, y, and z times, respectively, after DA depletion, and simulations of Conditions 1, 2, and 3 were conducted with the parameters x, y, and z systematically varied [simulation was quitted when action value larger than 100 times of the size of the large reward (i.e., extremely large) was detected]. We then set criteria to evaluate the simulation results by considering the following four features: for the ratio of choosing the HD arm (average across trials and completed simulation runs), (1) >0.1 decrease during 501–550th trials (i.e., soon after depletion) and (2) >0.5 decrease during 901–1000th trials, and for the latency (number of time steps) of reaching the T-junction (State 4; average across trials and completed simulation runs), (3) >0.5 increase during 501–550th trials and (4) <0.5 increase during 901–1000th trials. Simulation results were regarded as similar to the experimental results if the abovementioned features 1,2,3,4 were (1,1,1,1), (0,0,1,1), and (1,0,1,1), where 1 and 0 represent satisfied and unsatisfied for Conditions 1, 2, and 3, respectively, and the number of unsatisfied features, of 4 features/condition × 3 conditions = 12 features in total, were counted for each set of simulation results with particular x, y, and z; when all the simulation runs for a given condition were quitted due to extremely large action value, all the features were regarded to be unsatisfied. As a result of this systematic exploration (Fig. 9B), parameter sets (among tested ones) with which all the features were satisfied turned out to be x = 2.5 or 3 and y = z = 1 or 1.5. This result supports the possibility that the gain of the obtained-reward-representing input prominently increased after DA depletion, whereas the previous and upcoming values-representing inputs entailed no or mild gain increase.

Simulation of the D2 receptor antagonism experiments

In addition to the effects of DA depletion, effects of D2R antagonism have been examined in the T-maze experiments (Salamone et al., 1994). It was shown that injection of D2R antagonist haloperidol decreased HD choices in Condition 1, but not in Condition 2 (Fig. 10B), and also increased the latency in both Conditions 1 and 2 (Fig. 10C). Recent work (Pardo et al., 2012) examined the effects of haloperidol in mice in Condition 1 and a new condition, Condition 4, in which a barrier was placed in both arms (Fig. 10A), and showed that haloperidol decreased HD choices in Condition 1 but not in Condition 4 (Fig. 10D). We explored whether these results could also be explained by our model if possible effects of D2R antagonist on the circuit operation were incorporated.

• Download figure
• Open in new tab
• Download powerpoint

Figure 10.

Experimental results on the effects of D2 receptor (D2R) antagonist in the effort-related T-maze choice task. B, C, and D were reprinted from Salamone et al. (1994), Copyright 1994, and Pardo et al. (2012), Copyright 2012, respectively, with permission from Elsevier; colors were added on the bars, also with permission. A, Task conditions tested with D2R antagonist. In addition to Conditions 1 and 2, Condition 4, in which a physical barrier was placed in both the HD and LD arms, was tested. B, C, The ratio of selecting the HD arm (B) and the latency of start-door opening (C) in Condition 1 (orange-marked bars) and Condition 2 (blue-marked bars) in Salamone et al. (1994). The left two bars and the right two bars in each panel indicate the data for the rats that were injected with vehicle or D2R antagonist haloperidol, respectively. D, The ratio of selecting the HD arm in Condition 1 (orange-marked bars) and Condition 4 (green-marked bars) in Pardo et al. (2012). The horizontal axis indicates the dose of D2R antagonist haloperidol injected into the mice.

Activation of D2Rs on the DA axons inhibits DA release, causing a negative feedback, and D2R antagonist relieves such an inhibition (Gonon and Buda, 1985) and also inhibits DA uptake (Benoit-Marand et al., 2011), causing an enhancement of DA signaling. We incorporated this into the model as an amplification of TD-RPE–dependent value update. D2Rs are also expressed in about half of the striatal medium spiny neurons (MSNs), while the other half of MSNs express D1Rs (Gerfen and Surmeier, 2011). Activation of D2Rs causes a reduction of the responsiveness of D2-MSNs (Gerfen and Surmeier, 2011), and D2R antagonist is considered to block such a reduction and thereby amplify the output of D2-MSNs. The roles of D2Rs or D2-MSNs in reward learning have been examined by using pharmacological (Pessiglione et al., 2006; Lee et al., 2015) and optogenetic (Kravitz et al., 2012) manipulations. Their results have suggested that administration of L-DOPA or D2R antagonist (haloperidol) resulted in differential magnitude of RPE (larger in the former) in humans (Pessiglione et al., 2006), while injection of D2R antagonist (eticlopride) into the dorsal striatum of monkeys resulted in a decrease in the inverse temperature (Lee et al., 2015), and also that stimulation of D2-MSNs induced transient punishment in mice (Kravitz et al., 2012). One hypothesis, the CS-TD hypothesis (Morita et al., 2012, 2013; Morita, 2014; Morita and Kawaguchi, 2015), posits that D2-MSNs represent the value of previous action/state and negatively impact the DA neurons via the indirect pathway of the basal ganglia (Fig. 11A, left). This hypothesis could potentially explain [as shown in Morita et al. (2013)] the abovementioned optogenetic results (Kravitz et al., 2012) although in a different way from the authors’ explanations. If this hypothesis holds, the presumable amplification of the D2-MSNs output by D2R antagonist should cause an amplification of the previous-value term (–Q_Previous) of TD-RPE, which we incorporated into the model (Fig. 11A, right). In the results (Fig. 11B–D), D2R antagonist reduced the preference for the HD arm prominently in Condition 1 but much less prominently in Conditions 2 and 4, while increasing the latency in all the conditions. These results are at least partially in line with the experimental results (Fig. 10), although the absence of the effect in the choices in Conditions 2 and 4 was not reproduced.

• Download figure
• Open in new tab
• Download powerpoint

Figure 11.

Results of the simulations of the effects of D2R antagonist with the assumption that D2R antagonist enhances DA/TD-RPE signaling and also amplifies the output of D2R-expressing striatal medium spiny neurons (D2 MSNs) that presumably encode the value of previous action. A, Assumed effects of D2R antagonist. The antagonist was assumed to enhance DA/TD-RPE signaling, and also relieve D2R-mediated inhibition of D2 MSNs, which presumably encode the value of previous action (as illustrated) according to one hypothesis on the mechanism of TD-RPE calculation, named the CS-TD hypothesis (Morita et al., 2012, 2013; Morita, 2014; Morita and Kawaguchi, 2015). These presumed effects were incorporated into the model as an amplification (1.25 times) of TD-RPE–dependent value update and also an amplification (1.25 times) of the previous-value term in TD-RPE. B, C, D, Simulation results for Conditions 1, 2, and 4. The configurations are the same as those in Fig. 5B,C and Fig. 6Bf,Cf. a, Schematics of the simulated task conditions. b, The ratio of choosing the HD arm. c, The latency (number of time steps) of reaching the T-junction (State 4). d, Trial-by-trial evolutions of the learned values of Go_3→4, Stay_3→3, Go_4→5, and Go_4→6.

To understand the underlying mechanisms, we looked at the action values. In Condition 1 (Fig. 11Bd), the Go_4→5 value, referred to as Q(Go_4→5), markedly decreased after the administration of D2R antagonist. This should be because TD-RPE generated after taking Go_4→5 negatively shifted due to the presumed antagonist-induced amplification of the previous-value term [i.e., −Q(Go_4→5)] so that Q(Go_4→5) was negatively updated according to the TD-RPE. In contrast, as for TD-RPE generated after taking Go_4→6, amplification of the previous-value term [i.e., − Q(Go_4→6)] could cause only a weaker effect because Q(Go_4→6) was smaller than Q(Go_4→5), and thus marked decrease of Q(Go_4→6) did not occur. The marked decrease of Q(Go_4→5) but not of Q(Go_4→6) led to the prominent decrease of the ratio of choosing Go_4→5 (HD arm), which eventually resulted in a drastic shift in the balance between value-update and value-decay, causing an increase of Q(Go_4→6). Also, D2R antagonist similarly caused a prominent decrease of Q(Go_3→4), but not of Q(Stay_3→3), explaining the increase in the latency.

In Conditions 2 and 4 (Fig. 11Cd,Dd), D2R antagonist caused a decrease of Q(Go_4→5) similarly to Condition 1. However, because the difference between Q(Go_4→5) and Q(Go_4→6) before antagonist administration was larger than the case of Condition 1, a shift in the balance between value-update and value-decay was caused less prominently and therefore the change in the ratios of HD and LD choices was less prominent. This mechanism suggested that moderately increasing the magnitudes of HD and LD rewards in simulations might increase the initial difference between Q(Go_4→5) and Q(Go_4→6) in Conditions 2 and 4 and reduce the changes in the choice ratios, making the results closer to the experimental results. Indeed, this expectation was successfully confirmed (Fig. 12).

• Download figure
• Open in new tab
• Download powerpoint

Figure 12.

Results of the simulations of the effects of D2R antagonist, assuming the enhancement of DA/TD-RPE signaling and the amplification of the output of D2 MSNs (in the same manner as in Fig. 11), with the sizes of the rewards increased from the original ones (large = 1, small = 0.5) to large = 1.2, small = 0.6. A–C, Simulation results for Conditions 1, 2, and 4. The configurations are the same as those in Fig. 11B,C,Da–c. a, Schematics of the simulated task conditions. b, The ratio of choosing the HD arm. c, The latency (number of time steps) of reaching the T-junction (State 4).

Simulation of the D1 receptor antagonism experiments

It has been shown that D1R antagonism also specifically impaired the choice of high-cost, large-reward option (Fig. 13A) and increased the latency of start-door opening (Yohn et al., 2015). D1Rs are expressed in the other half of MSNs, and activation of D1Rs causes an enhancement of the responsiveness of D1-MSNs (Gerfen and Surmeier, 2011). D1R antagonist is considered to block such an enhancement and thereby reduce the output of D1-MSNs. The CS-TD hypothesis, introduced above, posits that D1-MSNs represent the value of upcoming action/state and positively impact the DA neurons via the direct pathway of the basal ganglia (Fig. 13B, left). If this is the case, the presumable reduction of the D1-MSNs output by D1R antagonist should cause a reduction of the upcoming-value term (+Q_Upcoming) of TD-RPE. We incorporated such a reduction into the model (Fig. 13B, right; see Materials and Methods for details) and conducted simulations. In the results (Fig. 13C,D), D1R antagonist reduced the preference for the HD arm prominently in Condition 1 (Fig. 13Cb) but almost not in Condition 2 (Fig. 13Db), while increasing the latency in both conditions (Fig. 13Cc,Dc). These results are in line with the experimental results. Looking at the action values in Condition 1, Q(Go_4→5) and Q(Go_3→4) prominently decreased after antagonist administration (Fig. 13Cd,Dd). These decreases, which explain the impairment in HD choices and the increase in the latency, are considered to occur because the assumed reduction of the upcoming-value term (Q_Upcoming) of TD-RPE negatively shifted TD-RPE.

• Download figure
• Open in new tab
• Download powerpoint

Figure 13.

Experimental results on the effects of D1R antagonist, and simulation results with the assumption that D1R antagonist reduces the output of D1 MSNs that presumably encode the value of upcoming action. A was reprinted from Yohn et al. (2015), Copyright 2015, with permission from Elsevier; colors were added on the bars, also with permission. A, The number of selecting the HD arm, out of a total of 30 trials, in Condition 1 (orange-marked bars) and Condition 2 (blue-marked bars) in Yohn et al. (2015). The horizontal axis indicates injection of vehicle (VEH) or the dose of D1R antagonist ecopipam. B, Assumed effect of D1R antagonist. The antagonist blocks D1R-mediated upregulation of D1 MSNs, which presumably encode the value of upcoming action (as illustrated) according to the CS-TD hypothesis. This effect was incorporated into the model as a reduction (to 0.8 times of the original) of the upcoming-value term of TD-RPE. C, D, Simulation results for Conditions 1 and 2. The configurations are the same as those in Fig. 11B,C. a, Schematics of the simulated task conditions. b, The ratio of choosing the HD arm. c, The latency (number of time steps) of reaching the T-junction (State 4). d, Trial-by-trial evolutions of the learned values of Go_3→4, Stay_3→3, Go_4→5, and Go_4→6.

Simulation of the DA depletion experiments, with alternative assumptions for depletion effects

In the above, we simulated DA depletion experiments with the assumption that DA depletion causes quarterization of TD-RPE–dependent value increment or value update (Fig. 14A, the first equation). However, DA depletion could instead, or in addition, causes modulations of the responsiveness of MSNs and DA axons expressing D1Rs/D2Rs, similarly to D1R/D2R antagonisms. We tested this possibility by performing separate sets of simulations assuming the same effects as assumed in the simulations of D1R/D2R antagonisms in the above (Fig. 14A, second equation), or those effects in addition to quarterization or halving of TD-RPE–dependent value update (regardless of whether TD-RPE was nonnegative or negative; Fig. 14A, third equation); the gain increase of the obtained-reward term of TD-RPE was also assumed in all the cases. In the results (Fig. 14B–D), the choice and latency patterns were largely in line with the experimental results (Fig. 1), except that eventual decrease in the latency was less prominent in the case assuming both modulations of MSNs/DA axons and quarterization of value update (Fig. 14C). These results indicate that both of the assumed effects of DA depletion, i.e., modulations of MSNs/DA axons and (mild) size reduction of value update, might contribute to the experimentally observed behavioral effects.

• Download figure
• Open in new tab
• Download powerpoint

Figure 14.

Simulation of the DA depletion experiments, with alternative assumptions for depletion effects. A, DA depletion was assumed to cause a reduction of the size of TD-RPE–dependent value update (indicated by the blue cross in the schematic and blue terms in the equations) in the simulations shown in Fig. 8A, but DA depletion can instead or in addition cause modulations of the responsiveness of MSNs and DA axons expressing D1Rs/D2Rs, i.e., effects similar to those assumed to be caused by D1R/D2R antagonisms (red crosses in the schematic and red terms in the equations). B, Results of simulations for Conditions 1–4 (a–d), assuming that DA depletion causes modulations of the responsiveness of MSNs and DA axons expressing D1Rs/D2Rs, as well as a gradual increase of the gain of the obtained-reward input, but not a reduction of the size of TD-RPE–dependent value update. C, Results of simulations for Conditions 1–4 (a–d), assuming that DA depletion causes both modulations of the responsiveness of MSNs and DA axons expressing D1Rs/D2Rs and a reduction of the size of TD-RPE–dependent value update, as well as a gain increase of the obtained-reward input. D, Same as C except for assuming that the DA-depletion-induced reduction of the size of TD-RPE–dependent value update was milder: 50%, rather than 25%, of the original. The configurations in B–D are the same as those in the previous figures.

Predictions of the model

Because our model describes the temporal change in the activity of DA neurons and striatal MSNs, our model provides predictions about the pattern of neural activity and how it is affected by DA manipulations. The black lines in Fig. 15B show the predicted activity pattern of DA neurons, at the time steps aligned at the times of start and reward, after learning has settled down (averaged over 251∼500 trials) in Condition 1 (Fig. 15A) without DA manipulations: the two panels separately show the cases where the HD or LD arm was chosen. Although learning has settled down, DA neurons are predicted to show activity not only at the time of start but also at the time of reward. Such sustained DA signals have been experimentally observed (Howe et al., 2013; Hamid et al., 2016), and our model successfully explains such signals by virtue of the value-decay, as we have previously explained (Morita and Kato, 2014; Kato and Morita, 2016). Our model further predicts that DA neuronal activity at the time of reward is higher when the LD arm is chosen than when the HD arm is chosen (t test; p < 10⁻⁹). This is because the LD arm is not frequently chosen, so the value-decay is relatively predominant compared to the value-update, resulting in the Go value preceding the reward remaining small and TD-RPE remaining large.

• Download figure
• Open in new tab
• Download powerpoint

Figure 15.

Model’s predictions about activity patterns of DA neurons, D1 MSNs, and D2 MSNs in the cases without or with DA manipulations. A, Schematic illustration of Condition 1 of the T-maze task. B, Predicted activity pattern of DA neurons without or with DA depletion. The black lines indicate the cases without DA depletion. The gray lines indicate the cases with DA depletion modeled as a reduction of the size of TD-RPE–dependent value update to 25% of the original, and also with the secondary effect (increase in the gain of the obtained-reward-representing input to DA neurons up to three times). The purple-gray lines indicate the cases with DA depletion modeled in an alternative way, i.e., as a reduction of the size of TD-RPE–dependent value update to 50% of the original and modulations of the responsiveness of MSNs and DA axons expressing D1Rs/D2Rs, and also with the secondary effect (the same assumption as made in Fig. 14D). The lines indicate the mean activity in the trials in which the HD arm was chosen (panels indicated as HD chosen) or the LD arm was chosen (LD chosen) during 251–500 trials (in the cases without DA manipulations) or 751–1000 trials (i.e., 251–500 trials from the onset of DA manipulations, in the cases with DA manipulations) in Condition 1 at the time steps aligned at the start (open triangle) or the reward (filled triangle), averaged across 20 simulations. C, Predicted activity pattern of DA neurons without or with DA antagonism. The black lines indicate the cases without DA antagonism and are identical to the black lines in B, i.e., repeatedly show the same single set of results for the purpose of comparison. The red and blue lines indicate the cases with D1R antagonism or D2R antagonism, respectively. D, Predicted activity patterns of MSNs corresponding to Go actions without or with DA depletion. The black lines indicate the cases without DA depletion, and the gray lines and purple-gray lines indicate the cases with DA depletion modeled in the two ways as in B. E, Predicted activity patterns of MSNs corresponding to Go actions without or with DA antagonism. The black lines indicate the cases without DA antagonism and are identical to the black lines in D, i.e., repeatedly show the same single set of results for the purpose of comparison. The red lines and blue lines indicate the cases with D1R antagonism or D2R antagonism, respectively.

The gray lines and purple-gray lines in Fig. 15B indicate the predicted activity pattern of DA neurons in the case with DA depletion, averaged over 751–1000 trials, i.e., 251–500 trials from the onset of DA depletion, with DA depletion modeled either as quarterization of TD-RPE–dependent value update (gray lines; same as in Fig. 8A) or as halving of value update and modulations of the responsiveness of MSNs and DA axons (purple-gray lines; same as in Fig. 14D); the gain increase of the obtained-reward-representing input was assumed in both cases. As shown in the figures, the model predicts that DA depletion decreases the DA neuronal activity at the start of trial, regardless of the way DA depletion is modeled. Meanwhile, predicted DA neuronal activity is lower at most timings when depletion is assumed to also cause modulations of the responsiveness of MSNs and DA axons (purple-gray lines). This is reasonable because the modulations of MSNs’ responsiveness are assumed to cause a negative shift in the net input to DA neurons. The red and blue lines in Fig. 15C indicate the predicted DA neuronal activity in the cases with D1R or D2R antagonism, respectively. As shown in the figures, in both cases, the activity at the times of start and LD-reward is predicted to be smaller than the intact case.

The black lines in Fig. 15D indicate the predicted activity pattern of D1 MSNs (left panels) and D2 MSNs (right panels) representing the value of a single Go action in the cases without DA manipulations. As shown in the figure, there are predicted to be neurons with activity peaking at various time points, in both D1 MSNs and D2 MSNs populations. The gray and purple-gray lines in Fig. 15D indicate the cases with DA depletion, modeled in the two different ways, as well as the secondary effect, and the red and blue lines in Fig. 15E indicate the cases with D1R or D2R antagonism, respectively. As shown in these figures, DA depletion and antagonisms are predicted to down-regulate the activity of MSNs in most cases, although the activity of MSNs representing the value of Go_4→6 is up-regulated by the manipulations, corresponding to the shift in the choice preference toward the LD arm.

Potential explanation of apparently contradictory results

Finally, we examined whether our model could also explain apparently contradictory results in the literature. Whereas there are a number of studies suggesting the involvements of DA in effort-related choices, DA measurement during a task with benefit-based or cost-based choices (Gan et al., 2010) revealed that DA evoked at the presentation of a cue predicting the level of benefit or cost encoded the benefit level well, but the cost level only in a limited manner. Specifically, rats were trained to make either a benefit (reward)-based choice, choosing a cue associated with large or small benefit with the cost (number of lever presses) equalized, or a cost (effort)-based choice, choosing a cue associated with small or large cost with the benefit equalized. Rats successfully learned to preferentially choose a large-benefit cue (Fig. 16Aa), as well as a small-cost cue (Fig. 16Ab). However, cue-evoked nucleus accumbens (NAc) DA in forced trials, where only a single cue was presented, well encoded the benefit level (Fig. 16Ac) but not the cost level (Fig. 16Ad): cost encoding was limited to the case with low cost and it diminished over training.

• Download figure
• Open in new tab
• Download powerpoint

Figure 16.

Potential explanation of the experimentally observed DA’s limited encoding of cost level in a different task paradigm by the model. Panels in A and B were reprinted by permission from Springer Customer Service Center GmbH: Springer Nature, Nature Neuroscience (Gan et al., 2010; Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine, Gan JO, Walton ME, Phillips PE, Nat Neurosci 13:25–27), Copyright 2010. A, a, b, The choice rate of options associated with different sizes of benefit (horizontal axis) with the cost equalized (a), or options associated with different amounts of cost (horizontal axis) with the benefit equalized (b). c, d, Cue-evoked nucleus accumbens DA in forced trials with different sizes of benefit (c) or different amounts of cost (d). B, Comparison of cue-evoked DA between forced trials (left bars of a, b) and choice trials where high-utility option was chosen (right bars of a, b) with different amounts of benefit (a) or cost (b). C, a, Schematics of the simulated benefit-based choice trials, in which larger or smaller benefit (size 0.5 or 0.25) could be obtained by first reaching State 6 or 5, respectively. b, Schematics of the simulated cost-based choice trials, in which equal benefit (size 0.5) could be obtained by first reaching State 6 or 7. c, d, The choice rate of arms with different sizes of benefit (c) or different amounts of cost (d) in the simulated task. The bar height indicates the mean choice rate for 251∼500 trials averaged across 20 simulations, and the error bar indicates the SEM of 20 simulations (these are applied also to panels D, e, and f, and E, b and d). D, a, b, Schematics of the simulated forced trials with larger benefit (a) or smaller (b) benefit, where the red crosses indicate that Go_4→5 or Go_4→6 was disabled, respectively (the same is applied also to c, d). c, d, Schematics of the simulated forced trials with smaller cost (c) or larger cost (d). e, f, TD-RPE generated at the start of a trial in the simulated forced trials with different sizes of benefit (e) or different amounts of cost (f). Notably, whereas choice trials and forced trials with one of the two options in the choice trials were intermingled, or more specifically, blocks of 4 forced trials and subsequent 4 choice trials were repeated in sessions where DA recording was made in the experiments (Gan et al., 2010), we simulated different types of forced trials separately from each other and also separately from choice trials. Also notably, in the model, the larger-benefit forced trials (D, a) were identical to the smaller-cost forced trials (D, c), and thus we conducted only a single set of simulations that corresponded to both types of forced trials and therefore the left shaded bars in e and f are identical, i.e., repeatedly show the same single simulation result for the purpose of comparison. E, a, c, Schematics of the simulated forced trials and choice trials. b, d, The right bars indicate TD-RPE at the start of the simulated benefit-based (b) or cost-based (d) choice trials where high-utility option was chosen. The left shaded bars indicate TD-RPE at the start of the simulated larger-benefit/smaller-cost forced trials, and are both identical to the left bars in D, e and f, i.e., repeatedly show the same single simulation result for the purpose of comparison. F, Learned values of Go actions in the simulated smaller-cost (a) or larger-cost (b) forced trials and cost-based choice trials (c). The lines indicate the mean for 251∼500 trials averaged across 20 simulations, and the error bars indicate the SEM of 20 simulations. In c, the gray line indicates the values of Go_4→5 and Go_5→7.

We qualitatively simulated this task by our model, having different sizes of benefits (size 0.5 and 0.25) on the states near the T-junction to simulate benefit-based choices (Fig. 16Ca) or the equal benefits (size 0.5) on the states near to and distant from the T-junction to simulate cost-based choices (Fig. 16Cb). With the same set of parameters (learning rate, inverse temperature, value-decay, and no temporal discounting) used so far, simulated subjects learned to preferentially choose the larger-benefit arm (Fig. 16Cc) and the smaller-cost arm (Fig. 16Cd) to a comparable level (t test, p = 0.487). Next, we simulated forced trials by disabling Go action from the T-junction to an arm corresponding to unavailable option (Fig. 16Da–d). Notably, whereas choice trials and forced trials were intermingled in the experiments, or more specifically, blocks of four forced trials and subsequent four choice trials were repeated in sessions where DA recording was made in the experiments (Gan et al., 2010), we simulated different types of forced trials separately from each other and also separately from choice trials. Also notably, in the model, the larger-benefit forced trials (Fig. 16Da) were identical to the smaller-cost forced trials (Fig. 16Dc), and thus we conducted only a single set of simulations that corresponded to both types of forced trials. We looked at TD-RPE generated at the start of a trial, after learning has settled down separately for each type of forced trials, as a counterpart of DA evoked at the presentation of a cue, and found that the TD-RPE differed prominently between cases with different sizes of benefit (Fig. 16De) but more mildly between cases with different amounts of cost (Fig. 16Df). This limited encoding of cost levels as compared to benefit-level encoding, despite the comparable behavioral preference, resembles the experimental results (Fig. 16A) to a certain extent, although the across-session diminishment of cost encoding is not explained in the model.

In our simulations, forced smaller-cost trials and forced larger-cost trials differed in the number of states from start to reward (Fig. 16Dc,d), and difference in the TD-RPE at the start between them corresponds to difference between the values of neighboring Go actions (Fig. 16Fa,b), which is shaped by the value-decay (Kato and Morita, 2016) and is relatively small. The reason that the difference in the cost level could nevertheless cause the prominent difference in the simulated choice trials is because the value-decay causes value-contrasts between well-chosen actions and less-chosen actions (Kato and Morita, 2016); i.e., in the simulated choice trials, the values of actions on a less frequently chosen arm are less frequently updated by TD-RPEs and thereby effectively decay more (Fig. 16Fc, gray line), amplifying the difference in the frequencies of arm choices. A key feature of our simulation of the task is that we simulated choice trials and each type of forced trials separately, as mentioned above. Notably, TD-RPE at the start of choice trials where high-utility option was chosen was close to TD-RPE in forced trials with the same option (Fig. 16E), appearing to resemble the experimental results (Fig. 16B), although in the simulations there were actually statistical differences because of small variances. Our results suggest that choice trials and forced trials might indeed be learned individually, at least to a certain extent, in the experiments, although they were intermingled, and the value of the action that imposes an extra cost significantly decayed only in the case where it needed not to be taken, i.e., in choice trials but not in forced trials, resulting in the observed apparently contradictory prominent cost avoidance in choice trials and DA’s limited cost encoding in forced trials.