Pupil Correlates of Decision Variables in Mice Playing a Competitive Mixed-Strategy Game

Performance of head-fixed mice in a matching pennies game. A, A schematic illustration of the competitive game. Head-fixed mouse makes a left or right choice by licking the spouts. A computer monitors the mouse’s past choices and outcomes, generating its own left or right choice every trial. B, The payoff matrix of the game. The mouse receives a water reward if it chooses the same choice as the computer does in the same trial. C, Trial timing: the mouse waits for a go cue, licks a spout to indicate its response, and the outcome is delivered immediately. A random ITI follows the outcome. D, An example session. The reward rate for this session was 52.1%. Top, The mouse’s choices and outcomes. Bottom, The computer’s choices. Blue and red bars indicate right (R) and left choices (L), respectively. Black bars indicate rewards. E, Cumulative number of different three-choice patterns detected as the mouse progressed in the session shown in D. F, Summary from 81 sessions. Left, The average trials performed each session is 513 ± 13. Middle, The average entropy of the three-choice sequences is 2.87 ± 0.02. Right, The average reward rate is 44.0 ± 0.5%. G, The histogram of the ITI durations for all trials. ITI would range from 4 to 8 s long if the mouse did not lick to trigger additions to the ITI. H, The response times for trials in which the mouse chose left (left) or right (right).

Computational modeling of the animals’ behavior during matching pennies. A, Schematic of the hybrid FQ_RPE_CK model. QL,QR denote the action values for left and right choices; KL,KR denote the choice kernels for left and right choices. The agent chooses based on a weighted sum of action-value and choice-kernel differences (ΔQ, ΔK). The chosen action is used to compute the CKE, which is used to update the choice kernels. The outcome is used to calculate the RPE, which is used to update the action values. B, Model comparison using BIC. C, An example of the time course of the latent variables and predicted behavior. Top, Long red and blue bars indicate rewarded left and right choices; short red and blue bars indicate unrewarded left and right choices. Gray line shows the observed probability to choose left, smoothed by a Gaussian kernel. Black line shows the probability to choose left predicted by the hybrid model. Middle, The action values of left (red) and right (blue) choices estimated by the hybrid model. Bottom, The choice kernels of left (red) and right (blue) choices estimated by the hybrid model.

Comparison between matching pennies and two-armed bandit: behavioral results, model fitting, and computer simulations. A, A schematic diagram of the two-armed bandit task. Top, Head-fixed mouse makes a left or right choice by licking the spouts. The trials are separated into different blocks based on the high (0.7) or low (0.1) reward probability assigned to each of the two choices. Middle, The assigned reward probabilities for the left (red) and right (blue) choices in an example session. Bottom, The choices and outcomes in the same session. The reward rate was 52.8%. Blue and red bars indicate right and left choices. Black bars indicate rewards. B, Summary from 26 sessions. Top, The average trials performed each session is 702 ± 26. Bottom, The average reward rate is 43.5 ± 1.1%. C, Model comparison using BIC. D, Learning parameters extracted from fitting the hybrid model to the matching pennies data. Left, The relative weight of choice kernel, β_K/(β + β_K) = 0.75 ± 0.04. Middle, Sum of inverse temperature, β + β_K = 2.15 ± 0.07. Right, Relative learning rate of the choice kernel. α_K/α = 0.22 ± 0.04. E, Psychometric curve based on the whole dataset. Black histograms indicate the distribution of trials according to the weighted sum of the difference of action values (ΔQ) and the difference of choice kernels (ΔK). Dashed purple line shows the predicted probability to choose left according to the softmax equation used by the hybrid model; purple dots show the observed probability to choose left. F, The performance of a computational agent playing the game, where β_K/(β + β_K) was varied, while the learning rates and (β + β_K) were fixed and set to be the median of the fitted values based on animal data. The solid and open dots indicate the median β_K/(β + β_K) value fitted based on animal data. G–I, Same as D–F for the two-armed bandit task. See also Extended Data Figure 3-1.

Effects of choices and outcomes on pupil responses during matching pennies. A, A schematic illustration of the pupillometry set up. B, An example still frame from a video showing both human labeling (dot) and DLC labeling (cross) for five labels. Scale bar: 50 pixels. C, The pupil response at any time during a trial (−1–5 s from the cue) is the z score at the corresponding time subtracted by the baseline, which is the mean z score between −2 and −1 s before the cue onset. D, A schematic diagram of the multiple linear regression model, i.e., Equation 9, that was fit to the pupil response in each 100-ms time bin. E, The fraction of sessions with significant regression coefficient for choice in the next trial c_n+1, choice in the current trial c_n, choice in the previous trial c_n–1, and choice in the trial before the previous trial c_n–2. Red shading indicates the p-value from the χ² test, without correcting for multiple comparison. F, Same as E for trial outcomes. G, Same as E for the interactions of choice and outcome. H, Same as E for recent reward rate, calculated as a moving average over last 20 trials, and the cumulative reward from start of session to current trial. I, The mean regression coefficients of several predictors: choice of the current trial (c_n), reward of the current trial (r_n). Shading indicates the 95% confidence interval estimated by bootstrap. See also Extended Data Figure 4-1.

Effects of decision variables for action selection on pupil responses during matching pennies. A, A schematic diagram of the multiple linear regression model, i.e., Equation 12. B, The fraction of sessions with significant regression coefficient for the choice c_n, outcome r_n, and the interaction c_n x r_n in the current trial. Red shading indicates the p-value from the χ² test, without correcting for multiple comparison. C, Same as B for the choice, outcome, and interaction in the previous trial. D, Same as B for the difference in action values (QnL−QnR ), the action value of the chosen action (Qnchosen ), and the difference in the choice kernel (KnL−KnR ). E, Same as B for choice kernel of the chosen action (Knchosen ), moving-average reward rate (rnMA¯ ), and cumulative reward (rnCum. ).

Effects of decision variables for value updating on pupil responses during matching pennies. A, A schematic diagram of the multiple linear regression model, i.e., Equation 13. B, The fraction of sessions with significant regression coefficient for choice in the current trial c_n, choice in the previous trial c_n–1, and outcome in the previous trial r_n–1. Red shading indicates the p-value from the χ² test, without correcting for multiple comparison. C, Same as B for difference in action values (QnL−QnR ), the RPE, and the difference in choice kernels (KnL−KnR ). D, Same as B for the choice kernel error (CKE ), moving-average reward rate (rnMA¯) , and cumulative reward (rnCum.) . E, The mean regression coefficients for RPE (quantified separately for trials with positive or negative RPE) and CKE. Shading indicates the 95% confidence interval estimated by bootstrap.

Multiple linear regression analyses of factors influencing pupil responses during two-armed bandit. A, The fraction of sessions with significant coefficient for choice in the next trial c_n+1, choice in the current trial c_n, choice in the previous trial c_n–1, outcome in the next trial r_n+1, outcome in the current trial r_n, and outcome in the previous trial r_n–1. The results were obtained by fitting Equation 9 as we did in Figure 4 but onto pupil responses from two-armed bandit task. Red shading indicates the p-value from the χ² test, without correcting for multiple comparison. B, Same as A but for the difference in action values (QnL−QnR ), the action value of the chosen action (Qnchosen ), the difference in the choice kernel (KnL−KnR ), and the choice kernel of the chosen action (Knchosen ). The results were obtained by fitting Equation 12 as we did in Figure 5 but onto pupil responses from two-armed bandit task. C, Same as A but for the RPE and the CKE. The results were obtained by fitting Equation 13 as we did in Figure 6 but onto pupil responses from two-armed bandit task.