Research ArticleResearch Article: New Research, Cognition and Behavior

The Serotonin 1B Receptor Modulates Striatal Activity Differentially Based on Behavioral Context

Ka H. Ng, Arati Sharma and Katherine M. Nautiyal
eNeuro 20 January 2026, 13 (2) ENEURO.0413-25.2026; https://doi.org/10.1523/ENEURO.0413-25.2026

Abstract

The dorsomedial striatum (DMS) is critical for both motivating and inhibiting behavioral responses. The region integrates inputs from the cortex, thalamus, and other subcortical structures including midbrain dopamine neurons. Though less studied, serotonin neurons from the dorsal raphe nucleus also richly innervate the DMS, which expresses nearly all 14 serotonin receptor subtypes. Slice electrophysiology shows that the serotonin 1B receptor (5-HT1BR) impacts DMS physiology and plasticity, and behavioral experiments show that 5-HT1BR expression modulates impulsivity and other DMS-dependent reward-related behaviors. In these studies, our goal was to investigate the effects of 5-HT1BR on the DMS in vivo. Using a genetic 5-HT1BR loss-of-function mouse model, we examined calcium activity of individual medium spiny neurons (MSNs) in the DMS of both males and females during operant tasks focusing on responses to actions, reward, and waiting. We found that knock-out of 5-HT1BRs resulted in different effects on MSN calcium activity depending on behavioral state. Specifically, mice lacking 5-HT1BRs showed significantly more inhibition of MSN calcium activity during the rewards, but more cells with excitatory calcium responses during the delay period of the trial. This suggests that serotonin, acting via 5-HT1BRs, may recruit MSN activity in response to reward but inhibit MSN activity during waiting. These results highlight the importance of in vivo studies for understanding the functional role of DMS serotonin in reward-related behavior. Overall our results demonstrate that serotonin can modulate the DMS in a behavioral state-specific manner, potentially providing a mechanism for how serotonin effects on behavior are context dependent.

Significance Statement

Behavioral, electrophysiological, and pharmacological evidence points to a role for serotonin modulation of striatal circuits in the brain. Our studies use in vivo calcium imaging to understand how serotonin signaling influences the dorsomedial striatum (DMS) during action and waiting behaviors. We show that serotonin signaling through the serotonin 1B receptor can have different effects on medium spiny neurons (MSNs) in the DMS depending on the ongoing behavior—increasing MSN activity during rewards and decreasing MSN activity during inhibition. These data show that serotonin effects on striatal circuitry are complex and can be dependent on behavioral context.

Introduction

The dorsomedial striatum (DMS) is critical for learning and executing motivated behavior across species. Its role in action–outcome learning has been extensively studied, and research has demonstrated the importance of dopamine modulation of DMS control of reward-motivated behavior (Dayan and Balleine, 2002). However, the DMS is also richly innervated by serotonin afferents from the dorsal raphe nucleus (DRN), and the majority of the 14 serotonin receptor subtypes are expressed in this region (Soghomonian et al., 1989; Di Matteo et al., 2008; Nair et al., 2020). Additionally, evidence shows that serotonin is critical for goal-directed decision-making, as disruption of serotonin signaling in humans and mice impairs inhibition of responses to devalued rewards and shifts behavior toward more habitual responding (Worbe et al., 2015; Ohmura et al., 2021). This suggests that serotonin may be involved in recruiting DMS-related circuitry to engage in model-based decision-making. However, limited research has investigated how serotonin modulates DMS circuitry in vivo.

Many convergent lines of research support a role for serotonin in the modulation of DMS signaling (Mathur and Lovinger, 2012; Nair et al., 2020). Using a GRAB-5-HT biosensor in vivo, we showed that serotonin encodes an anticipatory value-graded reward signal in the DMS (Spring and Nautiyal, 2024). Additionally, serotonin can modulate the release of dopamine in the DMS (Sershen et al., 2000). More directly, electrophysiology studies show that serotonin increases excitation of cholinergic and fast spiking interneurons in the DMS and reduces local collateral inhibition between medium spiny neurons (MSNs; Blomeley and Bracci, 2009; Virk et al., 2016; Pommer et al., 2021). Finally, serotonin has the potential to influence plasticity in the DMS by inducing long-term depression of cortical and thalamic inputs to MSNs (Mathur et al., 2011; Cavaccini et al., 2018). Interestingly, the majority of these effects of serotonin on DMS physiology have been shown to be mediated through signaling at the serotonin 1B receptor (5-HT1BR).

In addition to the evidence supporting a role for 5-HT1BR modulation of DMS physiology, many studies show that 5-HT1BRs also influence DMS-dependent behavior. Whole-brain 5-HT1BR knock-out (KO) mice display increased incentive motivation, impulsive, and compulsive behavior (Rocha et al., 1998; Nautiyal et al., 2015; Desrochers et al., 2021). However, 5-HT1BRs are expressed on axon terminals widely throughout the brain, and the region and cellular specificity for the effects of 5-HT1BR on these behavioral phenotypes are still being worked out. Tissue-specific knock-out of 5-HT1BRs in the DMS has been associated with altered reward-seeking phenotypes (Virk et al., 2016; Li et al., 2021). Interestingly this may arise through a number of different mechanisms via 5-HT1BR expression on a variety of distinct cell types within the DMS. These include heteroreceptor-mediated inhibition of acetylcholine release from striatal cholinergic interneurons, glutamate release from cortical inputs, and MSN collaterals inhibition within the DMS (Virk et al., 2016; Li et al., 2021; Pommer et al., 2021; Burke and Alvarez, 2022). Additionally, 5-HT1B autoreceptors are located on presynaptic terminals of serotonin inputs to the DMS allowing serotonin to inhibit its own subsequent release. The heterogeneous localization of 5-HT1BR expression within the complex DMS local circuitry highlights the difficulty of understanding serotonin effects on DMS activity, even just through one of its 14 receptors.

In this study, we set out to understand how global deletion of 5-HT1BR expression influences the calcium activity of DMS neurons during action and inhibition. We were specifically interested in understanding if the encoding of action or waiting in the DMS were altered in the absence of 5-HT1BRs. We used a one-photon head-mounted miniature microendoscope to image the calcium activity of MSNs in the DMS neurons in mice during two operant tasks to examine action and inhibition. The first was a time-pressured cue-guided response task in which mice had a limited cue period to respond for a reward. In this task, we found that mice lacking 5-HT1BR expression had less excitatory calcium responses suggesting that serotonin may increase MSN reward-related neural activity via 5-HT1BRs. In a second task, mice were challenged to wait during a delay period, before responding for a reward. Interestingly, during the delay period of this task, mice lacking 5-HT1BRs showed increased recruitment of excitatory calcium activity compared with controls suggesting that 5-HT1BR activation could inhibit MSN activity during behavioral inhibition. The differential effects of 5-HT1BR expression on MSN calcium activity during action and waiting suggest that serotonin can have different physiological effects on a population of MSNs depending on behavioral context. These results establish possible mechanisms of 5-HT1BR-dependent modulation of DMS-dependent reward-related behavioral dysfunction and point to the importance of studying the effect of serotonin in the brain in vivo.

Materials and Methods

Animals

All mice used for these experiments were bred in the institutional Center for Comparative Medical Research. Male and female mice lacking expression of 5-HT1BR (N = 9) and littermate genetic controls (N = 8) were bred by crossing tetO1B+/+::βActin-tTS+/− males to tetO1B+/+::βActin-tTS−/− females as previously reported (Nautiyal et al., 2015). Mice were weaned at postnatal day (P) 21 into cages of 2–4 same-sex littermates and maintained on a 12 h light/dark cycle on ad libitum chow and water until experimental testing began as described. All procedures were approved by the Institutional Animal Care and Use Committee.

Surgical procedures

Mice were anesthetized with isoflurane (1–3%) in 0.8 ml/min oxygen and head fixed into a stereotax (Kopf). The scalp was cleaned, shaved, and opened, and the connective tissue was cleared from the skull. The skull was leveled and a 1.0 mm burr hole was drilled at bregma (AP + 0.8; ML + 1.6 mm). The dura was removed, and the cortical tissue under the craniotomy was aspirated. AAV9.CamKII-GCaMP6f.WPRE.SV40 (2.5 × 1012 GC/ml) was injected into the parenchyma in two separate infusions of 250 nl each at a rate of 50 nl/min at 3.0 and 2.6 mm ventral to dura using a glass microsyringe (Hamilton) driven by a programmable pump (World Precision Instruments). A gradient refractive index (GRIN) lens (1 mm diameter, 4 mm long, Inscopix) was lowered into the brain to a depth of 2.3 mm ventral to dura and affixed to the skull with skull screws and Metabond (Parkell). Mice were individually housed following surgery. Approximately 6–8 weeks later GCaMP fluorescence was visualized with an nVista miniature microscope (Inscopix) and a baseplate was cemented to the headcap with additional Metabond cement.

Behavioral training and testing

The DIY-NAMIC system (Lee et al., 2020) was used for behavioral training and testing. Mice were initially habituated to retrieve water in their homecage from the center noseport. A solenoid delivered 10 μl of water through a blunted 18 gauge needle upon each head entry. During the first 4 d, a reward retrieval training paradigm (P1) was presented in which the cue light in the center port was illuminated and mice received 10 μl of water upon nosepoking into the center port. After water retrieval, a variable intertrial interval (ITI), with an average of 45 s, preceded the next light illumination. Next, mice were trained for 3–4 d on a second paradigm (P2) which required a nosepoke to either side port when illuminated with a cue light. Mice were then trained for 4 d on a third paradigm (P3) to self-initiate trials through nosepoke responses to a blinking light in the center port which indicated trial availability after the ITI. This illuminated both left and right cue lights. A nosepoke response to either side port was rewarded. Mice were then trained for 3 d on a fourth training paradigm (P4) in which only one of the two side ports was illuminated following trial initiation, and a nosepoke only to that port resulted in reward delivery. Incorrect nosepokes to the unlit port terminated the trial without reward and started the ITI. Subsequently, the cue duration was shortened to 5 s (P5) and then 1.5 s (P5b) for 2 d each. One mouse did not reach criteria for behavioral performance during this time-pressured 1.5 s cue and was excluded from analysis for this paradigm. Mice were imaged in the P5b paradigm for 1–2 d to measure the calcium activity of MSNs in the DMS during action and reward consumption. Subsequently, a delay of 3 s was introduced between trial initiation and the cue response in order to examine the calcium activity during waiting behavior. Mice were run on this paradigm (P6.3) for 6 d, with imaging on the first and sixth session. Two mice did not reach criteria for behavioral performance and were therefore excluded from further testing and analysis of calcium imaging data. Throughout training and testing, all port entries were detected with infrared sensors and sent through the Arduino UNO to Processing software on a linked PC. All behavioral programs (P1–P6.3) and processing scripts are available online at https://github.com/DIY-NAMICsystem. Following these training paradigms, the ports were blocked off in their homecage to restrict access to allow for imaging during behavior within a discrete session.

Data acquisition

Imaging was performed in 40 min long sessions by affixing the one-photon Inscopix nVista miniature microscope to the baseplate and removing the access barrier to the ports. Data was recorded using the nVista acquisition software at 20 frames per second with 50 ms exposure. Optimal LED settings (0.5–0.6 mW/mm2 range) were selected for each mouse and remained consistent for each mouse over all imaging sessions. The Arduino UNO that controlled the DIY-NAMIC system initiated the calcium imaging session by sending a 5 V start signal to the Inscopix Data Acquisition Software (IDAS). The Arduino UNO also emitted trains of 5 s on/off 5 V signals collected by an IDAS GPIO port. These pulses also allowed synchronization of the overhead camera video recording used for motion tracking. Additional 5 V signals were also sent to the IDAS GPIO for all DIY-NAMIC infrared port entries to validate signal alignment. Video footage was captured using an overhead camera (ELP-SUSB1080P01-LC1100) at 30 frames/second.

Calcium imaging processing

Raw videos were spatially cropped to remove areas not in the field of view of the lens and spatially downsampled by a factor of 3. The videos were then processed using the python implementation of the Calcium Imaging Analysis package, CaImAn (Giovannucci et al., 2019). Motion correction was performed using the NoRmCorre algorithm (Pnevmatikakis and Giovannucci, 2017). Parameters for motion correction for each video were selected using a grid search approach, where multiple iterations of the model were tested using different combinations of parameter values, and optimized for stability of the video frames over the session. Spatial contours and temporal fluorescence traces for putative cells were identified using the constrained non-negative matrix factorization (CNMF-e) algorithm (Zhou et al., 2018). Parameters for CNMF-e were selected for each video using a grid search approach. The optimal set of parameters were selected based on the following: the dimensions of spatial footprints, their spatial location in the field of view, the correlation of spatial footprints within a session, and the signal-to-noise ratio of the temporal traces. The components obtained using the optimal combination of parameters were then sorted manually by two observers to remove components with signal dynamics or spatial characteristics that are inconsistent with neuronal calcium activity. The data from one mouse in a P5b imaging sessions yielded fewer than 10 cells and was therefore excluded from analysis for that paradigm. Delta F/F videos were generated using a minimum z-projection image of the video. In order to analyze the data as a proxy of the underlying neural activity, calcium transients were extracted from the denoised calcium fluorescence traces using the sparse non-negative deconvolution algorithm (OASIS) implemented in CaImAn (Friedrich et al., 2017). This allowed us to categorize neurons by their activity profiles within discrete time windows of the trial without spillover of signal from adjacent trial epochs due to the relatively slow kinetics of calcium indicators (Sabatini, 2019).

Behavioral data processing

Each trial was categorized as omitted (no response in the cue period), incorrect (a response to the non-cued side port), or correct (a response to the cued port during the response window). Premature responses during the delay period in the waiting task did not influence categorization of the trial as correct or incorrect. The arduino UNO behavioral data was aligned to the Inscopix GPIO data using the 5 s on/off pulse to ensure continued alignment throughout the session. Pixel changes per frame were quantified using ezTrack (Pennington et al., 2019) and then thresholded at a 10 gray scale unit to remove baseline pixel fluctuations. The locomotion signal was then binned into 20 frames per seconds and aligned to the neural and Arduino data. For the action task, locomotion was compared between genotypes over the whole trial using a two-way ANOVA (genotype by time). For the waiting task, locomotion data was compared between genotypes and session across different phases of the trial using mixed effect models using the lmer function in RStats.

Analysis

For each cell, the last 6 s of every ITI (prior to trial availability) was averaged to establish average baseline activity. Cells were assessed for significant changes in calcium event rate during four time epochs of correct trials relative to the baseline period by 10,000 round two-tailed paired permutation. Permutations were chosen because of the non-normal distribution of the event rate data. In the cue-guided operant task, the four epochs were trial initiation (T) which included the 2 s preceding trial initiation; cue presentation (C) which began at trial initiation and lasted through the cue response; reward presentation (R) which consisted of the 3 s following the reward delivery; and the post-reward period (P) which included the period between 3 and 6 s after reward delivery. For the delay task, five epochs were used, with T, R, and P being defined as for the cue-guided operant task. Instead of the cue presentation epoch, a delay period (D) epoch was used which consisted of the 3 s delay period in the trial. p values were adjusted with Bonferroni to control for multiple comparisons across epochs and between excitation/inhibition. Cells that showed a significant change (p < 0.05/8) in one or more epoch relative to the pretrial baseline were assigned to a category based on the epoch(s) (T, C, D, R, P) and direction (+/−) of change. Average calcium event traces were calculated by averaging the traces for each cell across all animals within each genotype and then compared using a two-way repeated-measures ANOVA (genotype × time). Percentages of inhibited, excited, and nonresponsive cells were compared between genotypes using a chi-square test for each trial epoch. While both male and female mice were included in these studies, the groups were not powered to detect any sex differences.

Lens placement

Following all behavioral testing, mice were anesthetized with Ket/Xyl and transcardially perfused with 0.9% PBS followed by 4% paraformaldehyde (PFA). Skulls were removed and postfixed overnight in 4% PFA, before the brain was removed and then postfixed again in 4% PFA overnight. Brains were then cryoprotected in 30% sucrose in PBS and then frozen for sectioning at 40 μm on a cryostat (Leica CM3050S). Sections were stored in PBS and immunostained with chicken anti GFP (ab13970, lot GR3190550-15, Abcam) followed by cy2 anti-chicken (Jackson ImmunoResearch Laboratories, catalog #703-546-155) and DAPI (Thermo Scientific, catalog #62248). Stained sections were mounted onto slides and coverslipped with Krystalon for verification of lens placement and GCaMP6f expression (Fig. 1A). Localization of the lens for all mice with reported data was plotted on the brain atlas sections corresponding to 1.34 and 0.74 mm rostral to bregma (Fig. 1B; Franklin and Paxinos, 2007).

Figure 1.
Figure 1.

Localization of calcium imaging in the DMS. A, Representative photomicrograph showing the location of the GRIN lens track and GCaMP6f expression (green) on a DAPI stained (blue) coronal section. B, Atlas sections show reconstructed localization of the lens placement from each imaged mouse. Blue indicates mice excluded from analysis in the action task, and orange indicates mice excluded from analysis for the waiting task (see Materials and Methods for details).

Results

All mice (N = 6 Controls, N = 9 5-HT1BR KOs) were trained in a cue-guided operant response task (Fig. 2A). Mice performed an average of 45.9 ± 0.7 trials per session with 59 ± 3% of the trials rewarded, 5 ± 1% incorrect, and 37 ± 3% percent omitted (Fig. 2B). The high omission rates are a reflection of the time pressure in this task design which allows a cued response time period of only 1.5 s following trial initiation. There were no significant effects of 5-HT1BR genotype on any behavioral measures in this task, including the total number of trials initiated (t(13) = 1.51, p = 0.155), percentage of correct trials (t(13) = 1.62, p = 0.128), or latency to respond to the cue (avg 0.93 ± 0.3 s; t(13) = 1.59, p = 0.136). There were also no significant differences in body weight (t(13) = 1.46, p = 0.169) or overall locomotor activity during the trial (F(1,13) = 1.90, p = 0.191) between the genotypes. Calcium activity was recorded in 537 and 842 cells in control and 5-HT1BRKO mice, respectively. Activity for each cell was averaged over all rewarded trials within a session and aligned to the operant nose poke response. Cellular calcium activity heatmaps show that this task recruits the DMS in both genotypes, as seen by enriched calcium activity around the operant response during correct trials compared with a baseline period during the ITI (Fig. 2C).

Figure 2.
Figure 2.

DMS calcium activity during an appetitive instrumental task. A, Schematic showing operant trial structure including average percentage of trial types (omission, incorrect, and correct responses) across all mice. B, Behavioral data showing the average number of trials initiated (left) and average latency to respond to the cue on correct trials (right) for controls and mice lacking 5-HT1BRs. C, Heatmaps show the normalized dF/F of all cells averaged over all trials during ITI baselines (left) and correct trials (right) for control (top) and 5-HT1BR KO mice (bottom), sorted by time of peak event rate. D, Normalized event rates are shown averaged for all cells over baseline and trial periods for Control (blue) and 5-HT1BR KO (orange) mice. ***p < 0.0001.

5-HT1BR promotes reward-related increases in neural activity in the DMS

Event rates were extracted from the raw calcium traces and compared across genotypes. During the ITI baseline, there were no effects of 5-HT1BR expression on calcium event rates (Fig. 2D; t(1377) = 0.83, p = 0.41). During the trial, although population traces revealed qualitatively similar patterns of calcium activity between genotypes, there were significant differences in the magnitude of the average calcium event rates. Collapsed across the entire trial, there were significantly higher calcium event rates in cells from control mice compared with those from 5-HT1BR KO mice (t(1377) = 7.96, p < 0.0001). To further investigate the source of the decreased calcium event rates in the population-averaged traces in mice lacking 5-HT1BR across the extent of the trial, individual cells were characterized as excitatory or inhibitory in each of four epochs (trial initiation, cue response, reward consumption, and post reward consumption) of the task compared with the ITI baseline activity (Fig. 3A). This revealed significant effects of 5-HT1BR expression on the balance of excitation and inhibition of DMS calcium activity during this task (Fig. 3B). Specifically, the percentage of cells that showed excitatory responses during the trial was larger in controls compared with 5-HT1BR KO mice in all four epochs of the trial [χ2(1, N = 930) > 6.07, p < 0.013]. Additionally, there was a higher percentage of cells showing significant decreases in calcium event rates in 5-HT1BR KO mice during the reward and post reward periods [χ2(1, N = 930) > 5.62, p < 0.018]. These data suggest that the absence of 5-HT1BR expression results in decreased activity in MSNs in the DMS during instrumental behavior via less excitation and more inhibition.

Figure 3.
Figure 3.

5-HT1BR influences the balance of excitation and inhibition during goal-directed behavior. A, Traces from example cells showing significant responses during different trial epochs compared with baseline. B, The percentage of cells showing no change (gray), increased (green), or decreased (red) calcium activity during each of four trial epochs (trial initiation, cue presentation, reward presentation, post reward consumption) compared with baseline are shown for control (top) and 5-HT1BR KO (bottom). C, Radial plot shows the percentage of responsive cells according to their mixed selectivity across all four trial epochs for control (orange) and 5-HT1BR KO (blue) mice. T, trial initiation period; C, cue period; R, reward period; P, post reward period; +, significant excitatory response; −, significant inhibitory response; *p < 0.05; #p < 0.07.

Since these initial cell categorizations did not consider mixed selectivity over different trial epochs, we further classified cells according to combinations of trial epochs that cells showed significant excitation or inhibition of calcium event rate above baseline ITI levels. Twenty-two categories accounted for over 80% of responsive cells (Fig. 3C), and interestingly, the percentage of cells across categories was different between genotypes (category × genotype interaction: F(21,546) = 1.62, p = 0.039). Control mice showed a significantly higher percentage of cells excited during the trial (T + C + R + P+: p = 0.014 and C + R + P+: p = 0.020), while 5-HT1BR KO mice showed a higher percentage of cells inhibited during the reward epochs of the trial (R−: p = 0.028, R − P−: p = 0.017, and P−: p = 0.035). This suggests that in the absence of 5-HT1BR expression, the decreased excitation may be task related, while the increased inhibition is relatively later in the trial, after the operant response, and potentially specific to the reward-related responses.

When examining reward-related calcium activity, regardless of mixed selectivity, there was a higher proportion of neurons that were reward-responsive (either excited or inhibited) in mice lacking 5-HT1BR compared with controls [Fig. 4A; 39% in controls and 53% in 5-HT1BR KOs; χ2(1, N = 1,379) = 27.99, p < 0.0001]. Expression of 5-HT1BR significantly influenced the excitation–inhibition balance in trial epochs following the operant response to the cue (Fig. 4B; F(5,156) = 6.07, p < 0.0001 for genotype × cell type interaction). This was most notable in the number of cells showing significant excitation during reward receipt and continuing in the post-reward period, which was reduced in mice lacking 5-HT1BRs (R + P+; p = 0.005). Not only did a lower percentage of cells show excitation in 5-HT1BR KO mice, but population calcium traces showed that cells in 5-HT1BRKO mice also had lower calcium event rates on average, especially during the reward epochs (Fig. 4C; R+: p = 0.0014, P+: p = 0.013; R + P+: p = 0.0421). In addition to the decreased excitation, there was also increased inhibition seen following reward consumption in terms of a higher percentage of cells showing inhibitory responses during the post-reward period in 5-HT1BR KO mice (Fig. 4B; P−: p = 0.008). This was not associated with any differences in average amplitude of the inhibition during the reward or post reward epochs (Fig. 4C; P−: p = 0.136). Overall, the calcium data from this instrumental task suggests that 5-HT1BR may mediate a reward-related increase in neural activity in the DMS, which is consistent with our previous data showing that 5-HT1BR expression influences reward-sensitivity phenotypes (Desrochers et al., 2021).

Figure 4.
Figure 4.

5-HT1BR expression influences DMS activity associated with reward. A, Normalized calcium event rates are plotted for cells showing a significant response to reward for control (top) and 5-HT1BR KO (bottom) mice relative to cue response (yellow dashed line). B, The percentage of responsive cells that are responsive to the reward and post reward epochs, regardless of their responses to trial initiation or cue presentation, are shown for both genotypes. C, The average calcium activity is shown for cells classified as excited during the reward period (R+), post reward period (P+), or both (R + P+), or inhibited during the post reward period (P−) for control (blue) and 5-HT1BR KO (orange) mice. *p < 0.05.

5-HT1BR inhibits waiting-related increases in neural activity in the DMS

We used a second operant paradigm to probe the role of serotonin in the DMS during waiting by inserting a 3 s delay following trial initiation before the cue light was illuminated for responding (Fig. 5A). Given that this paradigm was challenging for the mice, mice were rewarded in only 14 ± 3% of the trials during their first session. However, there were no genotype differences in the number of trials initiated (F(1,13) = 0.14, p = 0.712) or in the percentage of rewarded trials (F(1,13) = 0.44, p = 0.520). Additionally, consistent with our previous data showing increased impulsivity in mice lacking 5-HT1BR (Nautiyal et al., 2015; Desrochers et al., 2021), performance in this task, as measured by the percentage of trials with premature responses, was worse in 5-HT1BR KOs compared with controls (main effect of genotype F(1,13) = 6.84, p = 0.021). A total of 334 and 645 cells were analyzed from control and 5-HT1BR KO mice while they performed this waiting task (Fig. 5B). There were no significant genotype differences in the number of cells identified per mouse (F(1,13) = 0.36, p = 0.561), nor in the calcium events for all cells over the trial (F(1,977) = 0.08, p = 0.772).

Figure 5.
Figure 5.

Mice lacking 5-HT1BRs have more active MSNs during the delay period. A, Schematic showing the two-choice serial reaction time task including average percentage of trial types (omission, incorrect, and correct responses) across all mice including sessions with 3 s and 6 s delays. B, The percentage of responsive cells that show increases in calcium event rates during the trial initiation and/or delay window periods for both genotypes. C, Normalized calcium event rates for cells averaged over all correct trials for Control (top) and 5-HT1BR KO mice (bottom), sorted by time of peak event rate. D, Percentage of D+ cells for each mouse plotted against impulsive behavior measured by percentage of premature trials within the imaging session. T+, cells significantly excited during the trial initiation period; D+, cells significantly excited during the delay period; T + D+, cells that are significantly excited during both trial initiation and delay period. ***p < 0.001.

We focused on cells that were responsive during the trial initiation and delay periods of the trial (Fig. 5C) and found a significantly higher percentage of cells that were active during the delay period in mice lacking 5-HT1BR (t(13) = 4.26, p = 0.0009). There were no significant differences in the proportion of cells that increased their calcium event rates during the trial initiation period or through the trial initiation and delay period (t(13) = 0.71, p = 0.50 for T+ and t(13) = 0.68, p = 0.51 for T + D+). The calcium event rates over the trial for each of these categories of cells showed similar patterns of increased calcium activity in mice lacking 5-HT1BR. In the T+ cells, there was no effect of genotype on the average calcium event rates (F(1,54) = 0.018, p = 0.894). For D+ and T + D+ cells, there were small but significant effects of genotype, with 5-HT1BR KOs showing higher calcium event rates during the trial (F(1,126) = 11.58, p = 0.0009 for T + D+; F(1,124) = 4.68, p = 0.033 for D+).

We were also interested to see if the increased recruitment of cells during the delay period in 5-HT1BR KO mice was associated with premature responding in the task (Fig. 5D). Although there was not a significant correlation between the percentage of cells that were active during the delay period and the number of trials with premature responses (r = 0.344; p = 0.21), there was a trend toward a positive association in the 5-HT1BR KO mice (rcontr = 0.049; rKO = 0.442). There were no significant effects of genotype on locomotion activity during the delay period (β = −259.9, SE = 354.2, t(13) = −0.73, p = 0.48), indicating that the increased calcium activity during the delay period did not likely stem from any differences in locomotor activity.

To further analyze the neural activity during the delay window, we trained mice for six sessions on this task to allow for improvement in performance. Both genotypes increased their performance on the task, showing increases in numbers of trials initiated (F(1,13) = 29.41, p = 0.0001) and rewarded (F(1,13) = 7.05. p = 0.0198), though premature responses remained elevated in 5-HT1BR KOs compared with controls (p = 0.013; Fig. 6A). Focusing again on cells that showed increases in calcium event rates during the trial initiation and delay period, we find that there were no effects of session on the proportion of cells in each category (Fig. 6B; F(1,13) = 0.9, p = 0.359). Additionally, the genotype difference in the percentage of cells that are responsive during the delay window was diminished in later sessions (F(1,13) = 1.829, p = 0.199). However, in both genotypes, the average calcium event rates of these cells increased significantly across the sessions (F(1,217) = 19.40, p < 0.0001), with the event rates almost doubling from the first to the sixth session (Fig. 6C). Comparing locomotor activity between the first and sixth sessions shows no significant difference in locomotor activity across sessions (β = −400.8, SE = 312.2, t(28) = −1.28, p = 0.21), matching the lack of change in premature responses, suggesting that the increased calcium activity during the delay period is not likely a consequence of any change in behavior during the task.

Figure 6.
Figure 6.

Training increases amplitude, but not proportion of delay window response neurons. A, Behavioral performance is shown as the percentage of trials with premature responses during the delay window. B, The percentage of responsive cells that show increases in calcium event rates during the trial initiation and/or delay window periods for both genotypes in the first and sixth training sessions of the waiting task. C, Normalized calcium event rates for cells showing significant increases in calcium activity during the delay period, averaged over all correct trials for the first (darker) and sixth (lighter) sessions. D, Scatterplots show correlations of the amplitude of the z normalized calcium event rate for individual cells showing increases in calcium activity during the delay period as a function of the number of impulsive responses made during the delay period for the first (left) and sixth (right) sessions. Lines of best fit are plotted for cells from mice from each genotype separately. *p < 0.05; **p < 0.001.

In order to examine the relationship between delay-period neural activity and behavior, we correlated the percentage of trials with premature responses with the maximum event rate of delay responsive cells for each animal (Fig. 6D). During session 1 there was no correlation in either genotype (r = 0.05; p = 0.8 for controls; r = −0.07, p = 0.49 for 5-HT1BR KOs). However, during session 6, both genotypes showed significant correlations between behavior and calcium event rate, though interestingly in opposite directions. Controls showed a negative association with decreased event rates being associated with more premature responses (r = −0.51, p = 0.012) while 5-HT1BR KOs showed increased event rates associated with more premature responding (r = 0.24, p = 0.048). These data suggest that over 6 sessions the calcium activity in the DMS changes differentially in the absence of 5-HT1BR expression, potentially underlying the decreased inhibition of responding during the delay period.

Discussion

Overall, our results show that 5-HT1BRs influence MSN activity in vivo during appetitive instrumental tasks. While previous work has shown this effect in vitro using electrophysiology, our study was one of the first to examine the effects of 5-HT1BR during freely moving operant behavior using one-photon calcium imaging of MSNs in the DMS. This allowed us to look at the influence of behavioral context on serotonin modulation of the DMS. Interestingly we found that the absence of 5-HT1BR expression resulted in differential effects depending on behavioral task requirements. The divergence is particularly remarkable given that the behavioral tasks were overall very similar—having the same general task structure, reward, and physical context. The difference was limited only to the waiting requirement during a delay period. First, in the task without a delay, we found that the absence of 5-HT1BR expression resulted in less excitatory calcium responses and more inhibitory responses to reward in MSNs. This suggests that serotonin signaling through 5-HT1BRs can promote increased MSN responding to reward. On the other hand, in the second task which included a 3 s delay period in the trial, we found that the absence of 5-HT1BR expression resulted in more cells showing excitation—suggesting that serotonin can inhibit MSN calcium activity via 5-HT1BRs during waiting. These nearly opposite effects of 5-HT1BR expression on MSN activity during responding and waiting were surprising given the overall similarity of the paradigm and trial structure.

Our results are consistent with past work implicating serotonin in regulating both approach and inhibition (Soubrié, 1986; Desrochers et al., 2022). A number of studies suggest that serotonin is necessary for patience. For example, depletion of serotonin increases premature responding, and optogenetic activation of serotonin neurons prolongs waiting time in rodents (Winstanley et al., 2004; Miyazaki et al., 2014, 2018). Interestingly, serotonin is also important for behavioral vigor in the context of operant responding (Bailey et al., 2018; Yoshida et al., 2019). In order to reconcile these two seemingly opposing ideas, one can consider that serotonin may act through different receptors, in different brain regions, or on different timescales in order to elicit the differential outcomes. However, in our studies, we imaged the same brain region in mice lacking a single serotonin receptor and found opposite effects of 5-HT1BR signaling on DMS activity. One potential explanation that has been proposed previously (Roberts et al., 2020) is that serotonin acts differentially based on behavioral task context, therefore having varied effects depending on the state of the DMS influenced by other inputs. In the response activation state, serotonin signaling through 5-HT1BRs may promote DMS activity, while in the behavioral inhibition context, serotonin inhibits DMS excitability. In this way, serotonin acting through a single receptor in one brain region on similar timescales could contribute to promoting both approach and waiting behavior.

Given that we used mice lacking 5-HT1BR expression throughout the brain, our results do not speak to the location of the direct site of action of the effects of serotonin. It is possible that the receptors which influence MSN calcium activity are located on a number of cell types within the DMS or even expressed on neurons at a distal brain region which send projections to MSNs (Mathur et al., 2011; Virk et al., 2016; Cavaccini et al., 2018; Pommer et al., 2021; Burke and Alvarez, 2022). Furthermore, given that we used whole-life knock-outs of 5-HT1BR, it is unclear when the effects of 5-HT1BR are occurring. There could be developmental effects resulting from a lack of 5-HT1BR expression which are the proximate cause of the reported effects on calcium activity. Importantly, we chose to use the tetO1B mouse line in these studies (Nautiyal et al., 2015), so that future work can build upon this using inducible, cell type-specific, tissue-specific, and projection-specific knock-out approaches to investigate the mechanisms through which 5-HT1BRs influence MSN activity in the DMS during reward-related behaviors.

Another limitation of our study is that we did not image cell type-specific calcium activity of D1R- and D2R-expressing subtypes of MSNs in the DMS. We used the CaMKIIa promotor to drive expression of the calcium indicator in the DMS. Although previous characterization of CaMKIIa expression has shown that it is expressed in the majority of MSNs and not expressed in ChAT, PV, NPY, or calretinin interneurons in the DMS (Klug et al., 2012), this does not preclude the possibility of the CaMKIIa promoter driving expression of GCaMP in interneurons as well as MSNs. Given mixed evidence supporting purported differential responses of direct and indirect pathway MSNs during motivated responding, it is unclear if we would expect to see differences in effects between the two cell types. A number of papers show similar calcium responses of D1+ and D2+ cells encoding actions during goal-directed behavior, while others show evidence of the more classic opponent control these cell types in their encoding of action and reward (Kravitz et al., 2012; Cui et al., 2013; Tecuapetla et al., 2016; Shin et al., 2018; Cox and Witten, 2019; Bolkan et al., 2022; Malvaez et al., 2025).

Regardless of how D1+ and D2+ cells may encode motivated behavior differentially, it is still unknown if serotonin preferentially targets or differentially affects the activity of the cell types. However, there is limited evidence to support that signaling through 5-HT1BR expression would differentially influence the activity of D1+ versus D2+ MSNs. First, there is no evidence of differential expression of 5-HT1BRs on the different MSN subtypes (Pommer et al., 2021). Further, assuming that the effects seen are due to the absence of MSN-instrinsic 5-HT1BR expression in the DMS, we would not expect to see any differences on the effect of D1- versus D2-expressing MSNs given evidence that 5-HT1BR reduces lateral inhibition similarly in both cell types (Pommer et al., 2021; Burke and Alvarez, 2022). Alternatively, if the effects of 5-HT1BR expression are via modulation of cortical inputs, we would again expect no cell type-specific effects based on evidence that 5-HT-LTD of MSNs is not pathway specific (Mathur et al., 2011). Finally, we have previously shown that 5-HT1BR KO mice do not have altered release of dopamine in the DMS (Nautiyal et al., 2015), suggesting that 5-HT1BR expression could not differentially influence dopamine signaling on D1 versus D2 receptors in the DMS. Overall, future studies should directly test if there are differential effects of serotonin on direct and indirect pathway signaling.

In conclusion, our work addresses the effects of 5-HT1BRs on MSN activity in the DMS, in vivo. While many elegant electrophysiology studies have shown 5-HT1BR-dependent effects on MSN activity in vitro, examining these effects during behavior is important to understand how these effects are influenced by behavioral context. In particular, the functional role of serotonin signaling during decision-making, goal-directed, habitual, and other reward-related behavior is multifaceted and likely modulated by context. Additionally, given the multiple cell types and projections expressing 5-HT1BRs in the DMS, this is a likely site at which serotonin may differentially influence behavior dependent on the state of the DMS given that inputs are known to vary during approach and inhibition. Overall, our one-photon calcium imaging data in the DMS show that 5-HT1BR modulation of MSN activity is dependent on behavioral state and point to a need to study the effects of serotonin signaling on neural circuits in vivo to understand how context influences serotonin neuromodulation.

Footnotes

  • The authors declare no competing financial interests.

  • We thank Drs. Kyle Smith and Mitch Spring for their helpful comments. This work was supported by the National Institutes of Health (NIH) R00 MH106731, National Institute of Mental Health (NIMH) R01 MH126178, and the William H. Neukom 1964 Institute for Computational Science CompX Faculty Award.

This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International license, which permits unrestricted use, distribution and reproduction in any medium provided that the original work is properly attributed.

References

  1. ↵
    1. Bailey MR, et al.
    (2018) An interaction between serotonin receptor signaling and dopamine enhances goal-directed vigor and persistence in mice. J Neurosci 38:2149–2162. https://doi.org/10.1523/JNEUROSCI.2088-17.2018
    OpenUrlAbstract/FREE Full Text
  2. ↵
    1. Blomeley CP,
    2. Bracci E
    (2009) Serotonin excites fast-spiking interneurons in the striatum. Eur J Neurosci 29:1604–1614. https://doi.org/10.1111/j.1460-9568.2009.06725.x
    OpenUrlCrossRefPubMed
  3. ↵
    1. Bolkan SS, et al.
    (2022) Opponent control of behavior by dorsomedial striatal pathways depends on task demands and internal state. Nat Neurosci 25:345–357. https://doi.org/10.1038/s41593-022-01021-9
    OpenUrlCrossRefPubMed
  4. ↵
    1. Burke DA,
    2. Alvarez VA
    (2022) Serotonin receptors contribute to dopamine depression of lateral inhibition in the nucleus accumbens. Cell Rep 39:110795. https://doi.org/10.1016/j.celrep.2022.110795
    OpenUrlCrossRefPubMed
  5. ↵
    1. Cavaccini A, et al.
    (2018) Serotonergic signaling controls input-specific synaptic plasticity at striatal circuits. Neuron 98:801–816.e7. https://doi.org/10.1016/j.neuron.2018.04.008
    OpenUrlCrossRefPubMed
  6. ↵
    1. Cox J,
    2. Witten IB
    (2019) Striatal circuits for reward learning and decision-making. Nat Rev Neurosci 20:482–494. https://doi.org/10.1038/s41583-019-0189-2
    OpenUrlCrossRefPubMed
  7. ↵
    1. Cui G,
    2. Jun SB,
    3. Jin X,
    4. Pham MD,
    5. Vogel SS,
    6. Lovinger DM,
    7. Costa RM
    (2013) Concurrent activation of striatal direct and indirect pathways during action initiation. Nature 494:238–242. https://doi.org/10.1038/nature11846
    OpenUrlCrossRefPubMed
  8. ↵
    1. Dayan P,
    2. Balleine BW
    (2002) Reward, motivation, and reinforcement learning. Neuron 36:285–298. https://doi.org/10.1016/S0896-6273(02)00963-7
    OpenUrlCrossRefPubMed
  9. ↵
    1. Desrochers SS,
    2. Lesko EK,
    3. Magalong VM,
    4. Balsam PD,
    5. Nautiyal KM
    (2021) A role for reward valuation in the serotonergic modulation of impulsivity. Psychopharmacology 238:3293–3309. https://doi.org/10.1007/s00213-021-05944-2
    OpenUrlCrossRefPubMed
  10. ↵
    1. Desrochers SS,
    2. Spring MG,
    3. Nautiyal KM
    (2022) A role for serotonin in modulating opposing drive and brake circuits of impulsivity. Front Behav Neurosci 16:791749. https://doi.org/10.3389/fnbeh.2022.791749
    OpenUrlCrossRefPubMed
  11. ↵
    1. Di Matteo V,
    2. Pierucci M,
    3. Esposito E,
    4. Crescimanno G,
    5. Benigno A,
    6. Di Giovanni G
    (2008) Serotonin modulation of the basal ganglia circuitry: therapeutic implication for Parkinson’s disease and other motor disorders. Prog Brain Res 172:423–463. https://doi.org/10.1016/S0079-6123(08)00921-7
    OpenUrlCrossRefPubMed
  12. ↵
    1. Franklin KBJ,
    2. Paxinos G
    (2007) The mouse brain in stereotaxic coordinates. San Diego: Elsevier.
  13. ↵
    1. Friedrich J,
    2. Zhou P,
    3. Paninski L
    (2017) Fast online deconvolution of calcium imaging data. PLoS Comput Biol 13:e1005423. https://doi.org/10.1371/journal.pcbi.1005423
    OpenUrlCrossRefPubMed
  14. ↵
    1. Giovannucci A, et al.
    (2019) Caiman an open source tool for scalable calcium imaging data analysis. Elife 8:e38173. https://doi.org/10.7554/eLife.38173
    OpenUrlCrossRefPubMed
  15. ↵
    1. Klug JR, et al.
    (2012) Genetic inhibition of CaMKII in dorsal striatal medium spiny neurons reduces functional excitatory synapses and enhances intrinsic excitability. PLoS One 7:e45323. https://doi.org/10.1371/journal.pone.0045323
    OpenUrlCrossRefPubMed
  16. ↵
    1. Kravitz AV,
    2. Tye LD,
    3. Kreitzer AC
    (2012) Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat Neurosci 15:816–818. https://doi.org/10.1038/nn.3100
    OpenUrlCrossRefPubMed
  17. ↵
    1. Lee JH,
    2. Capan S,
    3. Lacefield C,
    4. Shea YM,
    5. Nautiyal KM
    (2020) DIY-NAMIC behavior: a high-throughput method to measure complex phenotypes in the homecage. eNeuro 7:ENEURO.0160-20.2020. https://doi.org/10.1523/ENEURO.0160-20.2020
    OpenUrl
  18. ↵
    1. Li Y,
    2. Simmler LD,
    3. Van Zessen R,
    4. Flakowski J,
    5. Wan J-X,
    6. Deng F,
    7. Li Y-L,
    8. Nautiyal KM,
    9. Pascoli V,
    10. Lüscher C
    (2021) Synaptic mechanism underlying serotonin modulation of transition to cocaine addiction. Science 373:1252–1256. https://doi.org/10.1126/science.abi9086
    OpenUrlCrossRefPubMed
  19. ↵
    1. Malvaez M, et al.
    (2025) Striatal cell-type specific stability and reorganization underlying agency and habit (p. 2025.01.26.634924). bioRxiv. https://doi.org/10.1101/2025.01.26.634924
  20. ↵
    1. Mathur BN,
    2. Lovinger DM
    (2012) Serotonergic action on dorsal striatal function. Parkinsonism Relat Disord 18:S129–S131. https://doi.org/10.1016/S1353-8020(11)70040-2
    OpenUrlCrossRefPubMed
  21. ↵
    1. Mathur BN,
    2. Capik NA,
    3. Alvarez VA,
    4. Lovinger DM
    (2011) Serotonin induces long-term depression at corticostriatal synapses. J Neurosci 31(20):7402–7411. https://doi.org/10.1523/JNEUROSCI.6250-10.2011
    OpenUrl
  22. ↵
    1. Miyazaki K,
    2. Miyazaki KW,
    3. Yamanaka A,
    4. Tokuda T,
    5. Tanaka KF,
    6. Doya K
    (2018) Reward probability and timing uncertainty alter the effect of dorsal raphe serotonin neurons on patience. Nat Commun 9:2048. https://doi.org/10.1038/s41467-018-04496-y
    OpenUrlCrossRefPubMed
  23. ↵
    1. Miyazaki KW,
    2. Miyazaki K,
    3. Tanaka KF,
    4. Yamanaka A,
    5. Takahashi A,
    6. Tabuchi S,
    7. Doya K
    (2014) Optogenetic activation of dorsal raphe serotonin neurons enhances patience for future rewards. Curr Biol 24:2033–2040. https://doi.org/10.1016/j.cub.2014.07.041
    OpenUrlCrossRefPubMed
  24. ↵
    1. Nair SG,
    2. Estabrook MM,
    3. Neumaier JF
    (2020) Chapter 18—serotonin regulation of striatal function. In: Handbook of behavioral neuroscience (Müller CP, Cunningham KA, eds), Vol. 31, pp 321–335. London: Elsevier.
    OpenUrl
  25. ↵
    1. Nautiyal KM,
    2. Tanaka KF,
    3. Barr MM,
    4. Tritschler L,
    5. Le Dantec Y,
    6. David DJ,
    7. Gardier AM,
    8. Blanco C,
    9. Hen R,
    10. Ahmari SE
    (2015) Distinct circuits underlie the effects of 5-HT1B receptors on aggression and impulsivity. Neuron 86:813–826. https://doi.org/10.1016/j.neuron.2015.03.041
    OpenUrlCrossRefPubMed
  26. ↵
    1. Ohmura Y,
    2. Iwami K,
    3. Chowdhury S,
    4. Sasamori H,
    5. Sugiura C,
    6. Bouchekioua Y,
    7. Nishitani N,
    8. Yamanaka A,
    9. Yoshioka M
    (2021) Disruption of model-based decision making by silencing of serotonin neurons in the dorsal raphe nucleus. Curr Biol 31:2446–2454.e5. https://doi.org/10.1016/j.cub.2021.03.048
    OpenUrlCrossRefPubMed
  27. ↵
    1. Pennington ZT,
    2. Dong Z,
    3. Feng Y,
    4. Vetere LM,
    5. Page-Harley L,
    6. Shuman T,
    7. Cai DJ
    (2019) Eztrack: an open-source video analysis pipeline for the investigation of animal behavior. Sci Rep 9:19979. https://doi.org/10.1038/s41598-019-56408-9
    OpenUrlCrossRefPubMed
  28. ↵
    1. Pnevmatikakis EA,
    2. Giovannucci A
    (2017) NoRMCorre: an online algorithm for piecewise rigid motion correction of calcium imaging data. J Neurosci Methods 291:83–94. https://doi.org/10.1016/j.jneumeth.2017.07.031
    OpenUrlCrossRefPubMed
  29. ↵
    1. Pommer S,
    2. Akamine Y,
    3. Schiffmann SN,
    4. de Kerchove d'Exaerde A,
    5. Wickens JR
    (2021) The effect of serotonin receptor 5-HT1B on lateral inhibition between spiny projection neurons in the mouse striatum. J Neurosci 41:7831–7847. https://doi.org/10.1523/JNEUROSCI.1037-20.2021
    OpenUrlAbstract/FREE Full Text
  30. ↵
    1. Roberts C,
    2. Sahakian BJ,
    3. Robbins TW
    (2020) Psychological mechanisms and functions of 5-HT and SSRIs in potential therapeutic change: lessons from the serotonergic modulation of action selection, learning, affect, and social cognition. Neurosci Biobehav Rev 119:138–167. https://doi.org/10.1016/j.neubiorev.2020.09.001
    OpenUrlCrossRefPubMed
  31. ↵
    1. Rocha BA,
    2. Scearce-Levie K,
    3. Lucas JJ,
    4. Hiroi N,
    5. Castanon N,
    6. Crabbe JC,
    7. Nestler EJ,
    8. Hen R
    (1998) Increased vulnerability to cocaine in mice lacking the serotonin-1B receptor. Nature 393:175–178. https://doi.org/10.1038/30259
    OpenUrlCrossRefPubMed
  32. ↵
    1. Sabatini BL
    (2019) The impact of reporter kinetics on the interpretation of data gathered with fluorescent reporters. bioRxiv. https://doi.org/10.1101/834895
  33. ↵
    1. Sershen H,
    2. Hashim A,
    3. Lajtha A
    (2000) Serotonin-mediated striatal dopamine release involves the dopamine uptake site and the serotonin receptor. Brain Res Bull 53:353–357. https://doi.org/10.1016/S0361-9230(00)00358-0
    OpenUrlCrossRefPubMed
  34. ↵
    1. Shin JH,
    2. Kim D,
    3. Jung MW
    (2018) Differential coding of reward and movement information in the dorsomedial striatal direct and indirect pathways. Nat Commun 9:404. https://doi.org/10.1038/s41467-017-02817-1
    OpenUrlCrossRefPubMed
  35. ↵
    1. Soghomonian JJ,
    2. Descarries L,
    3. Watkins KC
    (1989) Serotonin innervation in adult rat neostriatum. II. Ultrastructural features: a radioautographic and immunocytochemical study. Brain Res 481:67–86. https://doi.org/10.1016/0006-8993(89)90486-1
    OpenUrlCrossRefPubMed
  36. ↵
    1. Soubrié P
    (1986) Reconciling the role of central serotonin neurons in human and animal behavior. Behav Brain Sci 9:319–335. https://doi.org/10.1017/S0140525X00022871
    OpenUrlCrossRef
  37. ↵
    1. Spring MG,
    2. Nautiyal KM
    (2024) Striatal serotonin release signals reward value. J Neurosci 44:e0602242024. https://doi.org/10.1523/JNEUROSCI.0602-24.2024
    OpenUrlAbstract/FREE Full Text
  38. ↵
    1. Tecuapetla F,
    2. Jin X,
    3. Lima SQ,
    4. Costa RM
    (2016) Complementary contributions of striatal projection pathways to action initiation and execution. Cell 166:703–715. https://doi.org/10.1016/j.cell.2016.06.032
    OpenUrlCrossRefPubMed
  39. ↵
    1. Virk MS,
    2. Sagi Y,
    3. Medrihan L,
    4. Leung J,
    5. Kaplitt MG,
    6. Greengard P
    (2016) Opposing roles for serotonin in cholinergic neurons of the ventral and dorsal striatum. Proc Natl Acad Sci U S A 113:734–739. https://doi.org/10.1073/pnas.1524183113
    OpenUrlAbstract/FREE Full Text
  40. ↵
    1. Winstanley CA,
    2. Dalley JW,
    3. Theobald DEH,
    4. Robbins TW
    (2004) Fractionating impulsivity: contrasting effects of central 5-HT depletion on different measures of impulsive behavior. Neuropsychopharmacology 29:1331–1343. https://doi.org/10.1038/sj.npp.1300434
    OpenUrlCrossRefPubMed
  41. ↵
    1. Worbe Y,
    2. Savulich G,
    3. de Wit S,
    4. Fernandez-Egea E,
    5. Robbins TW
    (2015) Tryptophan depletion promotes habitual over goal-directed control of appetitive responding in humans. Int J Neuropsychopharmacol 18:pyv013. https://doi.org/10.1093/ijnp/pyv013
    OpenUrlCrossRefPubMed
  42. ↵
    1. Yoshida K,
    2. Drew MR,
    3. Mimura M,
    4. Tanaka KF
    (2019) Serotonin-mediated inhibition of ventral hippocampus is required for sustained goal-directed behavior. Nat Neurosci 22:770–777. https://doi.org/10.1038/s41593-019-0376-5
    OpenUrlCrossRefPubMed
  43. ↵
    1. Zhou P, et al.
    (2018) Efficient and accurate extraction of in vivo calcium signals from microendoscopic video data. Elife 7:e28728. https://doi.org/10.7554/eLife.28728
    OpenUrlCrossRefPubMed

Synthesis

Reviewing Editor: Jibran Khokhar, Western University Department of Anatomy and Cell Biology

Decisions are customarily a result of the Reviewing Editor and the peer reviewers coming together and discussing their recommendations until a consensus is reached. When revisions are invited, a fact-based synthesis statement explaining their decision and outlining what is needed to prepare a revision will be listed below. The following reviewer(s) agreed to reveal their identity: Brian Mathur, Munir Kutlu.

This is an impactful and timely study examining how 5-HT1B receptor deletion alters dorsomedial striatal neuronal activity across reward-related behavioral states. The experimental design is strong, the data are novel, and the conclusions are generally well supported. Several aspects of the single-cell calcium imaging analyses, classification methods, and data presentation need clarification, and addressing these will substantially strengthen the manuscript.

Major Concerns

The criteria used to classify cells as excited, inhibited, or unchanged, and the definitions of R+, P+, R+P+, T+, D+, and T+D+, are not adequately explained. The thresholding rules, rationale, and examples should be explicitly provided. Because CaMKIIα-driven expression is not Cre-dependent, the recorded signals likely represent a mixed neuron population; this important caveat should be acknowledged. The Klug (2012) citation does not eliminate the possibility of CaMKII promoter activity in interneurons through other isoforms, and this interpretation should be revised.

The decision to rely solely on calcium event rates, rather than showing raw or normalized ΔF/F traces, needs justification. Providing continuous ΔF/F signals would allow readers to assess baseline stability, signal-to-noise, and temporal structure. Event-rate-only metrics may obscure meaningful differences, especially given the sparse, variable firing of MSNs. In several cases, significant genotype effects appear inconsistent with the visual similarity of traces. For example, in Fig. 5E, T+, D+, and T+D+ traces appear nearly identical in shape, peak, and area despite reported significant effects. Additional clarification is needed on what drives these differences and whether averaging event rates over a 12-second window may distort trial-specific dynamics.

There are inconsistencies in the degrees of freedom for T+, D+, and T+D+ analyses (54, 126, 124) that require clarification, as it appears the same pool of cells is being compared across trial phases. The rationale for performing premature-response correlations in Fig. 6, where behavior does not differ, but not in Fig. 5, where differences may be more relevant, should be addressed. The manuscript should also reference foundational work showing that 5-HT1BR activation depresses GABAergic input onto MSNs (PMID: 21593324) in the discussion.

Several figures need clearer presentation. Some axis labels are missing or use experimenter shorthand. Text sizes and axis numbers are sometimes too small or inconsistent. Abbreviations such as T+, D+, and T+D+ should be defined clearly in both the legend and main text. Figure 2d (right) and Fig. 6D require significance markings. Figure 6C is not shown and should be included.

Minor Concerns:

The inclusion of both sexes is appropriate; indicate whether sex differences were tested, absent, or underpowered to detect. In Fig. 3B, the KO color in the legend does not match the plotted data and may challenge color-blind readers. Ensure consistent p-value formatting (e.g., use "p = 0.018" rather than "p < 0.018" unless warranted). The term "goal-directed" should be used only if supported by explicit behavioral assessments. Clarify whether premature responses were counted as incorrect responses. Standardize spacing in genotype descriptors (e.g., "5-HT1BR KO" used consistently). Indicate statistical significance clearly in Fig. 6D.

Author Response

Dear Dr. Khokar and Reviewers, Thank you very much for your review of our manuscript "The serotonin 1B receptor modulates striatal activity differentially based on behavioral context". We appreciate your helpful comments. We have responded to each concern below and revised our manuscript accordingly.

Thank you again for your time.

The criteria used to classify cells as excited, inhibited, or unchanged, and the definitions of R+, P+, R+P+, T+, D+, and T+D+, are not adequately explained. The thresholding rules, rationale, and examples should be explicitly provided.

We have added example traces of categorized cells in Figure 3A shown as normalized ΔF/F traces, and the methods have been edited to include more details and reads as follows:

For each cell, the last 6s of every ITI (prior to trial availability) was averaged establish average baseline activity. Cells were assessed for significant changes in calcium event rate during trial epochs of relative to the baseline period by 10,000 round two-tailed paired permutation. Permutations were chosen because of the non-normal distribution of the event rate data. In the cue-guided operant task, the four epochs were trial initiation (T) which included the 2s preceding trial initiation; cue presentation (C) which began at trial initiation and lasted through the cue response; reward presentation (R) which consisted of the 3s following the reward delivery, and the post-reward period (P) which included the period between 3 and 6s after reward delivery. For the delay task, four epochs were also used, with T, R, P being defined as for the cue-guided operant task. Instead of the cue presentation epoch, a delay period (D) epoch was used which consisted of the 3s delay period in the trial. P-values were adjusted with Bonferroni to control for 8 comparisons among the 4 epochs and between excitation/inhibition. Cells that showed a significant change (p< 0.05/8) in one or more epoch relative to the pre-trial baseline were assigned to a category based on the epoch(s) (T, C, D, R, P) and direction (+/-) of change.

Because CaMKIIα-driven expression is not Cre-dependent, the recorded signals likely represent a mixed neuron population; this important caveat should be acknowledged. The Klug (2012) citation does not eliminate the possibility of CaMKII promoter activity in interneurons through other isoforms, and this interpretation should be revised.

We have added this caveat to the discussion to include the possibility that the CaMKII promoter could drive expression of GCaMP in interneurons in addition to MSNs.

The decision to rely solely on calcium event rates, rather than showing raw or normalized ΔF/F traces, needs justification. Providing continuous ΔF/F signals would allow readers to assess baseline stability, signal-to-noise, and temporal structure. Event-rate-only metrics may obscure meaningful differences, especially given the sparse, variable firing of MSNs.

We have added panels showing continuous dF/F signals in Fig 2C and Fig 3A. We have also added our rationale to the manuscript in the methods describing our deconvolution approach. Using calcium transients allowed us to categorize the activity of neurons by discrete trial epochs without spillover of the calcium signal from adjacent trial epochs due to the relatively slow kinetics of the calcium indicator. This allowed us to "recover the impulse-like biological events" that might be a more similar approximation of the underlying biological firing patterns (Sabatini, 2019).

In several cases, significant genotype effects appear inconsistent with the visual similarity of traces. For example, in Fig. 5E, T+, D+, and T+D+ traces appear nearly identical in shape, peak, and area despite reported significant effects. Additional clarification is needed on what drives these differences and whether averaging event rates over a 12-second window may distort trial-specific dynamics.

We have omitted the traces in Fig 5E because we agree with the reviewers that although there is a significant difference when tested over the whole trial period, this amplifies the significance of relatively small effect sizes. Additionally, these observations are secondary to our primary result that there is a change in the number of cells recruited to respond to the delay period. The significant results are still reported in the results section, but the relatively small effect size is noted.

There are inconsistencies in the degrees of freedom for T+, D+, and T+D+ analyses (54, 126, 124) that require clarification, as it appears the same pool of cells is being compared across trial phases.

The differences in the degrees of freedom for each of the subsets of cells are accurate because each group does include different sets of cells. The sets of cells (T+, D+, T+D+) are determined by their categorization as significantly responsive during that epoch of the trial (as determined by criteria explained in the first point above).

The rationale for performing premature-response correlations in Fig. 6, where behavior does not differ, but not in Fig. 5, where differences may be more relevant, should be addressed.

We have analyzed the correlations between premature responses and the percentage of D+ cells to the results and now show the data in Fig 5D.

The manuscript should also reference foundational work showing that 5-HT1BR activation depresses GABAergic input onto MSNs (PMID: 21593324) in the discussion.

We agree that this Mathur et al, 2011 paper which shows 5-HT1BR mediated LTD of glutamatergic input to MSNs is foundational. We have added additional references to this work in the discussion given that our citations of this work were previously limited to the introduction.

Several figures need clearer presentation. Some axis labels are missing or use experimenter shorthand. Text sizes and axis numbers are sometimes too small or inconsistent. Abbreviations such as T+, D+, and T+D+ should be defined clearly in both the legend and main text. Figure 2d (right) and Fig. 6D require significance markings. Figure 6C is not shown and should be included.

-Axes labels and text size in figures have been increased and standardized where relevant.

-Abbreviations for cell categories are now included in the legends and the main text.

-Significance markings have been added to Figure 2D and 6D.

-Figure 6C is now labeled.

Minor Concerns:

The inclusion of both sexes is appropriate; indicate whether sex differences were tested, absent, or underpowered to detect.

The groups were not powered to detect any sex differences, and this has now been explicitly stated in the analysis methods.

In Fig. 3B, the KO color in the legend does not match the plotted data and may challenge color-blind readers.

This has been corrected.

Ensure consistent p-value formatting (e.g., use "p = 0.018" rather than "p < 0.018" unless warranted).

Precise p values are reported except if they are <0.0001, reporting grouped post-hoc comparisons, or stated in figures.

The term "goal-directed" should be used only if supported by explicit behavioral assessments. 'Goal-directed' has been replaced by more accurate descriptors when appropriate.

Clarify whether premature responses were counted as incorrect responses.

Premature responses were not counted as incorrect. Only responses into the incorrect port were considered incorrect. This information has been added to the methods.

Standardize spacing in genotype descriptors (e.g., "5-HT1BR KO" used consistently).

These have been standardized in the manuscript and figures.

Indicate statistical significance clearly in Fig. 6D.

Statistical significance is now indicated for Figs 6C and 6D.

Back to top

In this issue

Email
Print
View Full Page PDF
Citation Tools
Respond to this article
The Serotonin 1B Receptor Modulates Striatal Activity Differentially Based on Behavioral Context
Ka H. Ng, Arati Sharma, Katherine M. Nautiyal
eNeuro 20 January 2026, 13 (2) ENEURO.0413-25.2026; DOI: 10.1523/ENEURO.0413-25.2026
Twitter logo Facebook logo Mendeley logo

Jump to section

Keywords

Responses to this article

Respond to this article

Jump to comment:

No eLetters have been published for this article.

Subjects