Sex Differences in Behavioral Responding and Dopamine Release during Pavlovian Learning

Visual Abstract


Introduction
Efficient reward seeking involves identifying cues that predict rewards and discriminating between cues that signal different reward options. The mesolimbic dopamine system plays an integral role in regulating behavioral responses toward reward-associated cues (Phillips et al., 2007;Salamone and Correa, 2012). Cue-evoked dopamine responses convey reward-related information such as the relative reward size (Tobler et al., 2005;Roesch et al., 2007;Gan et al., 2010), reward probability (Fiorillo et al., 2003;Hart et al., 2015), and reward rate (Fonzi et al., 2017). While this effect is evident in extensively trained animals, the emergence of these signals during initial training sessions has not been well characterized. We recently used a Pavlovian conditioning task to demonstrate that cue-evoked dopamine release encodes reward rate (i.e., the time elapsed since the previous reward delivery) after extensive training (.24 sessions; Fonzi et al., 2017), but not during the first 6 training sessions (Stelly et al., 2021). These findings suggest that cue-evoked dopamine encodes reward rate through a multistep process, by first signaling an upcoming reward independent of value during initial training sessions and, after additional training, conveying the relative difference in value between cues. However, it remains unclear whether extensive training is similarly required for cue-evoked dopamine signals to convey prospective value-related parameters, such as reward size.
In this study, we trained male and female rats on a Pavlovian task where one cue was associated with a small reward (one pellet) and another cue was associated with a large reward (three pellets). Female rats displayed higher levels of conditioned responding, a faster latency to the food port, and a higher number of post-reward head entries compared with male rats. We used fast-scan cyclic voltammetry to record changes in dopamine release in the nucleus accumbens (NAc) throughout learning. The cue-evoked dopamine response did not encode differences in reward size during the first six training sessions but did signal differences in value during later sessions. There were no differences in cue-evoked dopamine release between males and females. In contrast, the dynamics of reward-evoked dopamine release was influenced by both reward size and sex. Both male and female rats displayed higher reward-evoked dopamine release to the larger reward option, though females exhibited lower reward-evoked dopamine levels compared with males. These data illustrate that sex differences in dopamine transmission are stimulus specific.

Subjects and surgery
All procedures were approved by the Institutional Animal Care and Use Committee at The University of Texas at San Antonio. Male (weight range, 300-350 g) and female (weight range, 200-250 g) Sprague Dawley rats [Charles River; age range, postnatal day 60 (P60) to P65] were pair housed on arrival, and were given ad libitum access to water and chow and maintained on a 12 h light/dark cycle (n = 8 male rats/9 electrodes; 5 female rats/5 electrodes). Carbon fiber voltammetry electrodes consisted of a carbon fiber housed in silica tubing and cut to a length of ;150 mm (Clark et al., 2010). Voltammetry electrodes were surgically implanted to target the NAc (relative to bregma: 1.3 mm anterior; 61.3 mm lateral; 7.0 mm ventral) along with an Ag/AgCl reference electrode. Rats were single housed following surgery and allowed to recover for .3 weeks before beginning training.

Behavioral procedures
After recovering from surgery, rats were placed and maintained on mild food restriction (standard laboratory chow: ;8 g/d for females; ;15 g/d for males) to target 90% free-feeding weight, allowing for an increase of 1.5% per week. Behavioral sessions were performed in chambers (Med Associates) that had grid floors, a house light, a food tray, and auditory stimulus generators (2.5 and 4.5 kHz tones). The chamber floors were thoroughly cleansed with a disinfectant, and the walls and food port were cleaned with 70% ethanol solution between every subject. To familiarize rats with the chamber and food retrieval, rats underwent a single magazine training session in which 20 food pellets (45 mg; BioServ) were noncontingently delivered at a 90 6 15 s variable interval. Rats then underwent up to nine Pavlovian conditioning sessions (1/ d) that each consisted of 50 trials where the termination of a 5 s audio cue [conditioned stimulus (CS); 2.5 or 4.5 kHz tone, counterbalanced across animals] resulted in the delivery of a single food pellet (Small Reward trials) or three food pellets (Large Reward trials) and illumination of the food port light for 4.5 s. The three food pellets on Large Reward trials were delivered within 0.4 s after the end of the CS presentation. Each session contained 25 Small Reward trials and 25 Large Reward trials delivered in a pseudorandom order, with a 45 6 5 s intertrial interval between all trials. Conditioned responding was quantified as the change in the rate of head entries during the 5 s CS relative to the 5 s preceding the CS delivery (Fonzi et al., 2017;Stelly et al., 2021). We also quantified the latency to initiate a head entry during the CS. For the post-unconditioned stimulus (US) analysis, we calculated the average number of head entries made during a 9 s post-US delivery time window. For more detailed analysis, we also broke the 9 s time window into two 4.5 s epochs that corresponded to when the food tray light was illuminated (Early US; 0-4.5 s) and an equivalent period of time when the food tray light was turned off (Late US; 4.5-9 s).

Voltammetry recordings and analysis
Chronically implanted electrodes were connected to a head-mounted amplifier to monitor changes in dopamine release in behaving rats using fast-scan cyclic voltammetry, as described previously (Clark et al., 2010;Fonzi et al., 2017;Oliva and Wanat, 2019;Stelly et al., 2019Stelly et al., , 2020Stelly et al., , 2021Oliva et al., 2021). The carbon fiber electrodes were held at À0.4 V (vs Ag/AgCl) with voltammetric scans applied at 10 Hz in which the potential was ramped in a triangular waveform to 11.3 V and back to À0.4 V at a rate of 400 V/s. A principal component regression analysis (Heien et al., 2005) was performed on the voltammetry signal using a standard training set that accounts for dopamine, pH, and drift. The average postimplantation sensitivity of electrodes (34 nA/mM) was used to estimate dopamine concentration (Clark et al., 2010). Chemical verification of dopamine was achieved by obtaining a high correlation of the cyclic voltammogram during a reward-related event to that of a dopamine standard (correlation coefficient r 2 ! 0.75, by linear regression). Voltammetry data for a session were excluded from analysis if the detected voltammetry signal did not satisfy these chemical verification criteria (Fonzi et al., 2017;Stelly et al., 2021). Voltammetry data for a given trial were excluded if the principal component regression analysis failed to extract dopamine current on .25% of the data points for a given trial (i.e., the residual Q value for the regression analysis exceeded the 95.5% confidence limit for the training set; Heien et al., 2005;Rodeberg et al., 2017;Stelly et al., 2020Stelly et al., , 2021. The CS-evoked dopamine response was quantified as the average dopamine response during the 5 s CS relative to the 5 s before the CS delivery (Fonzi et al., 2017;Stelly et al., 2021). The peak US-evoked dopamine response was quantified as the maximum dopamine response in the 3 s following US delivery relative to the 0.5 s before US delivery. The area under the curve (AUC) of the post-US dopamine response was quantified as the average dopamine response in the 9 s following US delivery relative to 0.5 s before US delivery. To determine potential differences in the decay of reward-evoked dopamine release between males and females, we normalized and aligned to the peak US dopamine response following the Small Reward delivery of the first session. These data were then fit to a single-phase decay curve to calculate the tau for each electrode (Oliva and Wanat, 2019).

Data analysis
Statistical analyses were performed in GraphPad Prism 9 and RStudio. Behavioral responding and dopamine quantification were analyzed using a mixed-effects model fit (restricted maximum likelihood method), repeated measures (rm) where appropriate, followed by a post hoc Sidak's test. The Geisser-Greenhouse correction was applied to address unequal variances between groups. A repeated-measures correlation was used to correlate dopamine signals and behavioral outcomes across all subjects, trial types, and training sessions (Bakdash and Marusich, 2017;Stelly et al., 2021). The full list of statistical analyses is presented in Extended Data Tables 1-1, 2-1, 3-1, 4-1, and 5-1.

Histology
Rats were deeply anesthetized, and electrical lesions were applied to the voltammetry electrodes followed by intracardial perfusion with 4% paraformaldehyde. Brains were removed and postfixed for at least 24 h, then subsequently placed in 15% and 30% sucrose solutions in PBS. Brains were then flash frozen on dry ice, coronally sectioned, and stained with cresyl violet. Electrode locations were mapped onto a standardized rat brain atlas.

Results
Rats were trained on a Pavlovian conditioning task in which one audio cue (CS) signaled the delivery of a single sucrose pellet (US; Small Reward trial) and another audio cue signaled the delivery of three sucrose pellets (Large Reward trial, Fig. 1A). Conditioned responding was quantified as the change in the rate of head entries during the 5 s CS relative to the rate of head entries during the 5 s preceding the CS (Fonzi et al., 2017;Stelly et al., 2020Stelly et al., , 2021. Rats increased conditioned responding across sessions, with no difference between Small and Large Reward cues (three-way mixed-effects analysis; session effect: F (2.26,24.86) = 14.01, p , 0.0001; reward size effect: F (1,11) = 0.03, p = 0.86; n = 13 rats; Fig. 1B Table 1-1), consistent with prior findings (Eubig et al., 2014;Stringfield et al., 2019). We further analyzed these behavioral responses when averaged into threesession bins. During the first three sessions there were no sex differences in conditioned responding (two-way mixedeffects analysis; sex effect: F (1,11) = 0.90, p = 0.36; Fig. 1C, Extended Data Table 1-1), though females exhibited a faster latency to enter the food port (two-way mixed-effects analysis; sex effect: F (1,11) = 14.56, p = 0.003; Fig. 1F, Extended Data Table 1-1). During the latter three sessions, female rats displayed higher levels of conditioned responding (two-way mixed-effects analysis; sex effect: F (1,11) = 5.11, p , 0.05; Fig.  1D, Extended Data Table 1-1), though there were no sex differences in the latency to respond (two-way mixed-effects analysis; sex effect: F (1,11) = 2.71, p = 0.13; Fig. 1G, Extended Data Table 1-1). Collectively, these findings illustrate that female rats display augmented behavioral responding within the CS presentation compared with male rats during the first six training sessions of Pavlovian learning. However, these behavioral responses during the cue presentation did not reflect differences in the upcoming reward size.
Given the sex differences in CS-evoked behavior, we next examined whether male and female rats differed in their behavioral responses following the reward delivery. Female rats performed a higher number of non-CS head entries relative to males (Extended Data Fig. 1-1), which suggests that females performed more head entries following the US. To address this possibility, we examined the head entries performed in the 9 s after the reward was delivered ( Fig. 2A). Female rats exhibited greater post-US head entries compared with male rats (sex effect: F (1,55) = 17.44, p = 0.0001; Fig. 2B, Extended Data Table 2-1) Table 2-1). Together, these results illustrate that sex and reward size influence the number of post-US head entries.
The emergence of Pavlovian conditioned responses depends on dopamine signaling within the ventral striatum (Darvas et al., 2014). Here, we performed voltammetry recordings in the NAc to examine how the CS-and USevoked dopamine responses progressed across training (Fig. 3A,B). Both male and female subjects exhibited dopamine release to the CS presentation (Fig. 3C). We quantified CS-evoked dopamine release as the average response during the 5 s CS relative to the 5 s before the CS, identical to the manner in which conditioned responding was calculated (Fig. 1). CS-evoked dopamine release did not differ between sexes or trial type in the first six training sessions (three-way mixed-effects analysis; session effect: F (1.97,23.64) = 3.22, p = 0.06; sex effect: F (1,30) = 0.07, p = 0.80; reward size effect: F (1,12) = 3.54, p = 0.09; Fig. 3D, Extended Data Table 3-1).
To determine whether reward-evoked dopamine release scaled with reward size, we quantified the maximum dopamine response during the 3 s after the reward was delivered relative to the 0.5 s before the US delivery (peak US). Both male and female rats display a higher peak US dopamine response during Large Reward trials compared with Small Reward trials (three-way mixed-effects analysis; reward size effect: F (1,12) = 17.40, p = 0.001; Fig. 3C,E, Extended Data Table 3-1). We additionally analyzed the area under the curve for the average dopamine response during the 9 s after the US delivery relative to 0.5 s at the end of the CS (US AUC). Dopamine levels during this post-US period were higher following delivery of the Large Reward compared with the Small Reward (three-way mixed-effects analysis; reward size effect: F (1,12) = 17.98, p = 0.001; Fig. 3F, Extended Data Table  3-1). Furthermore, female rats displayed a lower US AUC dopamine response compared with male rats (three-way mixed-effects analysis; sex effect: F (1,30) = 7.91, p = 0.009; Fig. 3F, Extended Data Table 3-1). This difference in dopamine levels between trial types was also evident when examining the dopamine response during the Early and Late post-US epochs (Extended Data Fig. 3-1).
We next examined whether the lower post-US dopamine levels in female rats could be explained by a difference in dopamine clearance. To address this, the data were normalized to the peak US dopamine response and fit to a single-phase decay curve (Oliva and Wanat, 2019). This analysis was performed on the first training session when there was a robust US dopamine response and only for the Small Reward trials to minimize the potential influence of multiple reward deliveries on the dynamics of the dopamine response. Females exhibited a decreased plateau (i.e., lower dopamine levels) compared with males (unpaired t test, t (9) = 2.65, p = 0.03; Extended Data Fig. 3-2). However, there was no difference in the tau between male and female rats (unpaired t test, t (9) = 0.92, p = 0.38; Extended Data Fig. 3-2), which indicates that the rate of the decay of the US-evoked dopamine response is not influenced by sex. Collectively, these results suggest that in contrast to CS-evoked dopamine release, USevoked dopamine release encodes differences in reward size throughout the post-US period. Additionally, female rats exhibited a smaller US-evoked dopamine response relative to male rats.
CS-evoked dopamine release did not convey differences in reward size during early training sessions (Fig. 4). However, many studies demonstrate that the dopamine response to cues can convey differences in reward value in well trained animals (Fiorillo et al., 2003;Tobler et al., 2005;Roesch et al., 2007;Gan et al., 2010;Hart et al., 2015;Fonzi et al., 2017). To determine whether differences in CS-evoked dopamine emerge with further training, a subset of rats underwent three additional training sessions. In contrast to the first six training sessions, CSevoked dopamine release signals differences in reward size in the following three training sessions (two-way mixed-effects analysis; reward size effect: F (1,10) = 5.78, p = 0.04; Fig. 4A,B, Extended Data Table 4-1). The peak US dopamine response did not differ by trial type in later sessions (two-way mixed-effects analysis; reward size effect: F (1,10) = 2.83, p = 0.12; Fig. 4C, Extended Data Table  4-1). However, the US AUC dopamine response remained higher following delivery of the Large Reward compared with the Small Reward (three-way mixed-effects analysis; reward size effect: F (1,10) = 24.54, p = 0.0006; Fig. 4D, Extended Data Table 4-1), and lower in female rats compared with male rats (three-way mixed-effects analysis; sex effect: F (1,10) = 8.76, p = 0.01; Fig. 4D, Extended Data Fig. 4-2), which is consistent with the findings from the first six training sessions. Furthermore, significant sex differences in conditioned responding and the post-US head entries were also evident during these later training sessions (Extended Data Fig. 4-1).
Prior studies have linked CS-and US-evoked dopamine release to conditioned responding (Darvas et al., 2014;Fonzi et al., 2017;Coddington and Dudman, 2018;Heymann et al., 2020;Lee et al., 2020;Morrens et al., 2020;Stelly et al., 2020Stelly et al., , 2021. Here, we used a repeated-measures correlation analysis to determine how conditioned responding relates to dopamine transmission across all subjects and training sessions. While CS-evoked dopamine release was not correlated with conditioned responding (repeated-measures correlation; conditioned responding: r rm = À0.04, p = 0.60; latency: r rm = 0.06, p = 0.40; Fig. 5A, Extended Data Table 5-1), there was an inverse relationship between conditioned responding and the peak US dopamine response (repeated-measures correlation: r rm = À0.15, p = 0.04; Fig.  5B, Extended Data Table 5-1). Furthermore, the number of head entries occurring in the 9 s following reward delivery was related to the US AUC dopamine response (repeated-measures correlation: r rm = 0.17, p = 0.02; Extended Data Table 5-1). These results highlight that behavioral responding during early Pavlovian learning is linked to US-evoked dopamine levels and unrelated to CS-evoked dopamine levels.

Discussion
The dopamine response to cues can signal differences in value-related information in well trained animals (Fiorillo  , 2003;Tobler et al., 2005;Roesch et al., 2007;Gan et al., 2010;Hart et al., 2015;Fonzi et al., 2017). For example, cue-evoked dopamine release conveys differences in the reward rate (i.e., the time elapsed since the previous reward delivery) after extensive training (Fonzi et al., 2017). However, cue-evoked dopamine release does not signal differences in reward rate during the first six Pavlovian training sessions (Stelly et al., 2021). Our current results extend on these findings and demonstrate that during the first six training sessions, cue-evoked dopamine release did not signal differences in reward size. However, with further training, we find that reward size is encoded by the dopamine response to cues. Together, this suggests that cue-evoked dopamine signals differences in reward value through a multistep process. First, cue-evoked dopamine signals an upcoming reward independent of value. Additional training is then required for cue-evoked dopamine release to encode differences in reward value.
Prior studies have identified sex differences in behavioral responding, as females display elevated motor activity compared with males in locomotor and anxiety-like assays (Scholl et al., 2019;Bishnoi et al., 2021;Knight et al., 2021). Furthermore, sex differences have been observed across a variety of dopamine-dependent behaviors in mice and rats (Zachry et al., 2019;Kutlu et al., 2020;Chen et al., 2021). For example, female rats display a faster acquisition rate and elevated responding during drug self-administration (Lynch and Carroll, 1999;George et al., 2021). Female mice also exhibit higher levels of conditioned freezing relative to males during fear conditioning (Dubroqua et al., 2011). Sex differences have additionally been identified in Pavlovian conditioning tasks using food rewards (Stringfield et al., 2019). Specifically, female rats exhibited greater levels of sign-tracking behavior (e.g., physical interactions with a lever cue), but there were no sex differences in goal-tracking behavior (e.g., head entries to the food receptacle; Stringfield et al., 2019). By using audio cues in our Pavlovian task, animals were not able to engage in standard sign-tracking behaviors. Regardless, we identified prominent sex differences in this Pavlovian task as females exhibited a higher level of goal-tracking compared with males. In addition to behavioral responses occurring during the cue, we analyzed the post-US head entries into the food port. Both males and females performed more head entries following the delivery of the large reward option. Furthermore, female rats performed more head entries compared with males throughout the post-US period. These findings illustrate previously unappreciated sex differences in behavioral responding following the delivery of rewards. We note that the sex differences observed across some of our behavioral metrics (latency and post-US head entries) could be explained by higher levels of motor activity in females (Scholl et al., 2019;Bishnoi et al., 2021;Knight et al., 2021). However, our measure of conditioned responding is normalized to underlining differences in motor activity as we calculate the change in the rate of the head entries during the 5 s CS relative to the rate of head entries during the preceding 5 s. As such the sex differences in conditioned responding in our task cannot be explained solely by increased activity.
We identified stimulus-specific sex differences in dopamine release using fast-scan cyclic voltammetry. Cueevoked dopamine release did not differ between males and females. In contrast, the reward-evoked dopamine response was lower in females relative to males. Prior research has identified lower levels of dopamine transmission in ovariectomized female rats compared with castrated male rats using microdialysis (Becker and Ramirez, 1981;Castner et al., 1993;Xiao and Becker, 1994). However, one must exercise caution generalizing the findings from ovariectomized females to intact females, as there are no sex differences in basal dopamine levels, as measured by no net flux microdialysis within intact subjects (Xiao and Becker, 1994). Furthermore, a meta-analysis of microdialysis research concludes no basal or drug-induced sex differences in striatal dopamine in rats (Egenrieder et al., 2020). As such, it is unlikely that any potential basal differences in dopamine levels between sexes could account for the stimulusspecific sex differences in rapid dopamine transmission.
Increasing evidence suggests that the estrous stage may contribute to the observed sex differences in dopamine release in mice, rats, and nonhuman primates (Becker and Ramirez, 1981;Castner et al., 1993;Xiao and Becker, 1994;Kritzer and Kohama, 1998;Zhang et al., 2008;Becker et al., 2012;Perez et al., 2014;Calipari et al., 2017;Becker and Chartoff, 2019;Yoest et al., 2019;Zachry et al., 2021). For example, in mice and rats the Research Article: New Research burst firing rate of dopamine neurons in the ventral tegmental area is elevated during estrus compared with other stages of the cycle in females, as well as compared with males (Perez et al., 2014;Calipari et al., 2017). Additional research finds that female mice elicit higher striatal dopamine release during estrus in response to electrical stimulation and cocaine, as measured using fast-scan cyclic voltammetry in anesthetized subjects (Calipari et al., 2017). We did not monitor the stages of the estrous cycle in the current study, so we cannot assess whether these sex differences in reward-evoked dopamine release are because of cycling hormones. However, the observed sex differences were selective to the reward delivery (and not the cue) and were observed across sessions. Together, this evidence suggests that the sex differences in dopamine transmission are not mediated by the estrous cycle and could instead reflect intrinsic differences between males and females. Additionally, we found no difference in the rate of decay of reward-evoked dopamine release between males and females, which suggests that the observed sex difference is likely not because of differences in dopamine clearance. Future studies are needed to identify the source of these sex differences with in vivo dopamine transmission, which may result from anatomic and/or functional differences in the afferent input conveying reward-related information. Recent work identified regional differences in stimulated dopamine release between males and females (Brundage et al., 2021), so it is unclear whether the sex differences in rewardevoked dopamine release are present throughout the striatum and across species. One potential limitation of our study is the lower number of female rats with voltammetry electrodes used. However, we observed robust behavioral effects and changes in dopamine release in females. Regardless, the lower US-evoked dopamine response in females could account for the results from human studies where females exhibit a diminished sensitivity to rewards relative to males (Dhingra et al., 2021).
The magnitude of the dopamine response to the cue presentation and reward delivery have been linked to behavioral outcomes in a variety of Pavlovian conditioning tasks (Darvas et al., 2014;Fonzi et al., 2017;Coddington and Dudman, 2018;Heymann et al., 2020;Lee et al., 2020;Morrens et al., 2020;Stelly et al., 2020Stelly et al., , 2021. Our findings demonstrate a positive relationship between head entries and dopamine release following reward delivery, which is likely mediated by the increase in both post-reward head entries and the post-reward dopamine response across training in females. We also observed an inverse correlation between conditioned responding and reward-evoked dopamine release. While this relationship accounts for a relatively small amount of variation in the data, these results parallel our prior research and is consistent with the findings from studies using optogenetic manipulations of the dopamine system (Heymann et al., 2020;Stelly et al., 2020Stelly et al., , 2021. Cue-evoked dopamine release did not encode differences in reward value during the first six training sessions in the current study and in our prior work (Stelly et al., 2021). Our task design used two audio cues, and the rewards were delivered in the same location between trials, which together could dampen the ability of the subject to discriminate between the cues and their corresponding outcomes. It is possible that cue-evoked dopamine release could reflect differences in reward value during earlier training sessions if we had used cues with different sensory modalities. We find that conditioned responding did not differ by reward size and is not related to cue-evoked dopamine release within the first nine training sessions. However, in animals extensively trained under a similar Pavlovian task, an update in cue-evoked dopamine release can elicit a cue-specific change in conditioned responding (Fonzi et al., 2017). Together, these studies indicate that the initial emergence of conditioned responding is linked to the decrease in reward-evoked dopamine release, whereas updates to cue-evoked dopamine release in well trained animals leads to a corresponding update in conditioned responding.