Markerless Mouse Tracking for Social Experiments

The pipeline of the proposed algorithm for MT and feature detection after training is complete. Traditional segmentation (details in Extended Data Fig. 2-1) can be optionally used to reduce computational cost. User can select to bypass this and use Mask R-CNN exclusively. Our approach generates body masks and snout and tail-base coordinates. It can optionally be ensembled with deep-learning-based pose estimation techniques such as DLC or SLEAP for further improvement of snout and tail-base detection.

Training and evaluation of the Mask R-CNN. a, Example images with labels taken from the auto-training set. b, Example images with human annotations taken from the manual-training set. Extended Data Figure 3-1 compares human annotation required for all approaches. c, Mask, bounding box, and classification losses for fivefold cross-validation on the auto-training set in which the dataset was split into five sets, the model trained on four splits, and validated on the left-out split. d, Average precision metrics (AP, AP₇₅, and AP₅₀) of test data for three splits of training data vs. number of training images corresponding to 0, 20, 40, 60, and 80% of the manual-training set. e, Kernel density estimate of AP of the auto-trained model and dual-trained model groups on test data for the first split of the training set which account for 20–80% of the manual-training set. f, Visualization of the outputs of the auto-trained model and dual-trained model groups with 20, 40, 60, and 80% training fraction from the manual-training set (red) and human annotation (yellow). Each row shows performance on a different frame. Number pairs are predicted mask confidence and IoU.

Percentage of frames with segmentation errors over 12 videos in 4 categories in the MT dataset. The auto-trained model is built using the auto-training set that required no human effort and does not have mice in close proximity. The dual-trained model is a fine-tuned version of the auto-trained model incorporating manually segmented images of closely interacting mice from the manual-training set. Manual-trained model is trained only by the manual-training set.

Comparison of our approaches—MD and ensemble—with DLC and SLEAP. The upper panel shows average MOTA and the lower panel shows total instances of switched identities across all 12 videos in the 4 categories.

Performance across 12 videos in 4 categories. a, Fraction of frames with the mean distance between model predictions and human annotations below a varying threshold. b, Boxplots showing errors in MD, DLC, and ensemble models. Plots show median, 25th and 75th percentile, and outliers defined as >75th percentile +1.5 times the inter-quartile range. Text above outliers show number of outliers and average outlier within parentheses. Results for individual videos are shown in Extended Data Figures 6-1–6-3.

Examples of a, good and b, visibly inaccurate snout and tail-base detection in a group of three mice. c, Snout and tail-base trajectories for each animal along with a distribution of frame-to-frame movement in x and y coordinates.

Application of the markerless MT approach showing that loss of olfaction through ablation of the main olfactory epithelium impairs social recognition behaviors in mice. a, Mice were housed in same sex pairs; one mouse per pair was a control (intranasal saline-treated) or anosmic (intranasal ZnSO₄-treated). b, Experimental paradigm for social recognition test: anosmic/control mice acted as “observers” (Obs) and were presented with either a familiar demonstrator (Dem) or an unfamiliar demonstrator. c, Left: behavioral ethograms of control (top) and anosmic (bottom) observers while interacting with a familiar or unfamiliar demonstrator. Each color indicates when the snout of the observer was directed toward the anogenital (blue), body (orange), or head/face (green) region of the demonstrator. Right: cumulative distributions of each of the social investigative behaviors for control (top) and anosmic (bottom) observers toward familiar (solid line) or unfamiliar (dotted line) demonstrators. d, Total number of social investigation events by control (top) and anosmic (bottom) observers directed toward the anogenital, body, or head/face region of familiar versus unfamiliar demonstrators. e, Average duration of each social investigation event. f, Interval between social investigation events. *p < 0.05; **p < 0.01; ***p < 0.001. Paired t-test comparing social investigation toward a familiar versus unfamiliar demonstrator. Analysis of demonstrator mice behavior in Extended Data Figure 8-1.

Application of the markerless MT approach showing that anosmic mice spend more time in contact (non-snout-directed contact) with unfamiliar versus familiar demonstrators. a, Left: behavioral ethograms of control (top) and anosmic (bottom) observers while interacting with a familiar or unfamiliar demonstrator. Each brown bar indicates when the mice were in contact (touching) that was not as a result of snout-directed investigation by the observer or demonstrator. Right: cumulative distributions of touching in pairs of control (top) or anosmic (bottom) observers with familiar (solid line) or unfamiliar (dotted line) demonstrators. Bar graph shows total time spent touching during the first minute of the 5-min period. b, Total number of touching events between control (left) or anosmic (right) observers, with familiar or unfamiliar demonstrators. c, The average duration of each touching event. d, The inter-event interval between touching events. *p < 0.05. Paired t-test comparing touching between observers and familiar versus unfamiliar demonstrators.