Article Figures & Data
Figures
- Figure 1.
A multistep pipeline processed the input videos into masks using semi-supervised learning and FLHO for postprocessing. A, Examples of preprocessed video frames, intermediate SNR representation frames, model output, and final masks obtained from our pipeline. B, Schematic of our semi-supervised training pipeline that used ensemble learning to predict active neurons in unlabeled frames. We trained three different shallow U-Nets on labeled frames. The titles above the U-Nets represent the number of channels in each level of the decoder, starting with the deepest level, and “c” denotes a concatenation. We then passed unlabeled frames through each model and averaged the resulting probability maps to create pseudolabels. We retrained and fine-tuned one of the three U-Nets with these pseudolabels and then fine-tuned this network with a final round of training on the labeled frames. See Extended Data Figures 1-1, 1-2, 1-3A,B, and 1-6 for more details. C, FLHO found optimal hyperparameters for the postprocessing pipeline that processed frames of probability maps to predict individual neurons. We first thresholded the probability maps from the CNN (p_thresh). We then segmented ROIs in each frame and removed ROIs that were smaller than min_area. We then merged ROIs across frames by their relative spatial locations (centroid_dist) and removed any ROIs that were not active for enough consecutive frames (min_consecutive). * denotes hyperparameters. See Extended Data Figures 1-3C, 1-4, and 1-5, Table 1-1, and Table 1-2 for more details.
- Figure 2.
SAND outperformed other pipelines on low numbers of ground truth labels when processing the ABO 275 µm and ABO 175 µm datasets. A, Example segmentations from ABO 275 µm video 539670003. Masks generated by SAND were more accurate than those of other methods, even when trained on only 10 frames. Yellow boxes indicate region isolated in panel B. Scale bar, 50 µm. See Extended Data Figure 2-1A and Table 2-2 for more details. B, Example neurons zoomed from boxed regions in panel A. When trained on only 10 frames, SUNS identified many false-positive masks, whereas SAND accurately identified neurons. CaImAn and Suite2p both failed to find some ground truth neurons and Suite2p in particular had irregularly shaped masks. Scale bar, 25 µm. SAND had higher accuracy than other methods with low number of ground truth labels on both the (C) ABO 275 µm and (D) ABO 175 µm datasets. Dots represent the average F1 score for each model when processing the nine test videos. Lines represent the mean F1 scores averaged over bins grouped by the number of training labels; bins spanned 0–50 labels, 50–100 labels, etc. Shaded regions represent standard error. Horizontal lines are the average F1 scores of Suite2p and CaImAn. SAND generally did not improve recall but improved precision for the ABO datasets. The red line (SAND) represents ensemble learning and hyperparameter optimization with FLHO. The blue line represents single-model supervised learning and hyperparameter optimization with FLHO. The orange line (SUNS) represents single-model supervised learning and grid search hyperparameter optimization. See Extended Data Figures 2-2, 2-3, and 2-4 and Table 2-1 for more details.
- Figure 3.
SAND outperformed other pipelines on low numbers of ground truth labels using the Neurofinder datasets. A, Example segmentations from Neurofinder video 4.00. Masks generated by SAND were more accurate than those of other methods, even when trained on only 10 frames. Yellow boxes indicate region isolated in panel B. Scale bar, 50 µm. See Extended Data Figure 2-1B and Table 2-3 for more details. B, Example neurons zoomed from boxed regions in panel A. When trained on only 10 frames, SAND correctly identified more masks than CaImAn and Suite2p. Scale bar, 25 µm. C, SAND generally had higher accuracy than other methods when trained on a low number of ground truth labels. Dots represent the average F1 score for each model when processing the test video(s). Lines represent the mean F1 scores averaged over bins grouped by the number of training labels; bins spanned 0–50 labels, 50–100 labels, etc. Shaded regions represent standard error. Horizontal lines are the average F1 scores of Suite2p and CaImAn. More than half of Neurofinder videos did not have >250 neurons, so we did not include trials with >250 neurons in comparisons and binned results. The red line (SAND) represents ensemble learning and hyperparameter optimization with FLHO. The blue line represents single-model supervised learning and hyperparameter optimization with FLHO. The orange line (SUNS) represents single-model supervised learning and grid search hyperparameter optimization. See Extended Data Figure 3-1 and Table 3-1 for more details.
- Figure 4.
SAND outperformed other pipelines on low numbers of ground truth labels using the CaImAn datasets. A, When trained with low numbers of ground truth neurons, SAND outperformed all other methods on the K53 video. SAND had the highest F1 and precision of all methods. See Extended Data Figure 2-1C and Table 2-4 for more details. B, When trained with low numbers of ground truth neurons, SAND outperformed all other methods on the J115 video. SAND outperformed CaImAn and Suite2p, but not SUNS on the (C) J123 and (D) YST videos when trained on low numbers of ground truth neurons. Dots represent the average F1 score for each model when processing the test video(s). Lines represent the mean F1 scores averaged over bins grouped by the number of training labels; bins spanned 0–50 labels, 50–100 labels, etc. Shaded regions represent standard error. Horizontal lines are the average F1 scores of Suite2p and CaImAn. The red line (SAND) represents ensemble learning and hyperparameter optimization with FLHO. The blue line represents single-model supervised learning and hyperparameter optimization with FLHO. The orange line (SUNS) represents single-model supervised learning and grid search hyperparameter optimization. See Extended Data Figures 4-1, 4-2, 4-3, 4-4, and 4-5, Table 4-1, and Table 4-2 for more details.
Extended Data
Figure 1-1
Multiple neural network architectures generated the pseudolabels and final predictions. We used three U-Net architectures with the same encoder but varying decoders. The labels above the U-Nets represent the number of channels in each level of the decoder, starting with the deepest layer, and ‘c' denotes a concatenation. The numbers above each block represent the number of channels at each layer and the variables to the left represent the dimensions of the image at each step, given an input frame with dimensions m × n. We used a dropout rate of 0.1 for the first two depths and a rate of 0.2 for the deepest depth. The total number of trained parameters for each architecture was ∼5000 (blue), ∼6000 (green), and ∼7500 (purple). Download Figure 1-1, TIF file.
Figure 1-2
Pseudolabel training helped reduce and stabilize training loss. (A) Example learning curves for training and validation sets using different amounts of training labels on ABO 275 µm data. The first round of supervised training (green) lasted for 200 epochs. We then trained models of pseudolabels (orange) for 25 epochs. A final round of training on the labeled frames (purple) lasted for another 200 epochs. (Insets) Corresponding learning curves on pseudolabels, which used binary cross entropy loss. (B) Breakdown of training time for SAND and SUNS for each step of training using different amounts of ground truth labels. Total training time for SAND and SUNS increased as the number of training labels increased. Even when trained on 500 frames, both SAND and SUNS could be trained in less than an hour. Bars represent the average training time for 10 different models. Error bars represent the standard deviation for the total training time (n = 10 models). Download Figure 1-2, TIF file.
Figure 1-3
Pseudolabels and final probability maps closely aligned with ground truth temporal masks. (A) Example pseudolabels for two different ABO 275 μm frames generated by ensembles trained on different numbers of labeled frames. Pseudolabels were generally consistent across different numbers of training frames. The scale bar is 50 μm. (B) Pseudolabels were strongly correlated with the ground truth temporal masks. Bars represent the median correlation values between 1000 pseudolabels and ground truth temporal masks (100 labels from 10 different models). Error bars represent standard deviation. Data points represent the median correlation values for each model. (C) Example probability maps used for FLHO. Probability maps closely aligned with the ground truth temporal mask. Thresholding with the 25th percentile value helped retain lower confidence neurons. The scale bar is 50 μm. Download Figure 1-3, TIF file.
Figure 1-4
Few label hyperparameter optimization accurately estimated SAND hyperparameters and was invariant to optimization choices. (A) Grid searches for hyperparameters with small numbers of ground truth labels consistently underestimated the optimal p_thresh value. We performed a traditional grid search for all hyperparameters on 20 models, each trained on randomly selected sets of 10 frames from the ABO 275 µm dataset (2 models per video). We also used grid search for all hyperparameters using all labels from all frames to generate the ‘optimal' hyperparameters. p_thresh tuned on 10 frames was consistently lower than the optimal p_thresh. We then calculated the hyperparameters for the 20 models using our new hyperparameter optimization method, FLHO. This method produced p_thresh values that were similar to the optimal p_thresh. p_thresh is listed as a grayscale pixel value from 0 to 255 (e.g. a value of 205 corresponds to a probability of 80%). Parentheses show the mean ± standard deviation of the number of labeled neurons used in hyperparameter optimization. (B) The accuracy of our method was robust to small changes in intermediate p_thresh. We performed our hyperparameter optimization pipeline on 20 models using four different percentiles for intermediate p_thresh, used during step 2 of the pipeline. Each model was trained on 10 labeled frames from the ABO 275 µm dataset (73 ± 22 labeled neurons). A one-way Kruskal-Wallis test did not find a significant difference in performance between the four choices of p_thresh (p = 0.20). Bars and error bars respectively represent mean and standard error. (C) The values of centroid_dist and min_area determined by the grid search were robust to changes in the p_thresh percentile used in step 2 of FLHO. Bars represent the median values of centroid_dist and min_area for the models in B. A one-way Kruskal-Wallis test did not find a significant difference in the grid search selection between the four choices of p_thresh for min_area or centroid_dist (n = 20 models, p = 0.45 and p = 0.46, respectively). (D) The accuracy of our method was robust to changes in the index of max_continuous used to calculate min_consecutive. We performed our hyperparameter optimization pipeline on 20 models using three different conditions for min_consecutive for each model. Each model was trained on 10 labeled frames from the ABO 275 µm dataset (73 ± 22 labeled neurons). A one-way Kruskal-Wallis test did not find a significant difference between the three conditions (p = 0.43). Bars and error bars respectively represent mean and standard error. Download Figure 1-4, TIF file.
Figure 1-5
The Few Label Hyperparameter Optimization pipeline had four main steps. (A) We determined the 25th percentile and median values for p_thresh. The 25th percentile value was used in Step 2 and the median value was used in steps 3 and 4, which included neuron prediction. (B) We used a grid search to find the min_area and centroid_dist values that maximized F1 on labeled frames. (C) We found the maximum number of continuous frames for each ground truth neuron. (D) We determined the final hyperparameters. We placed an upper bound of 80% on p_thresh. To determine min_consecutive, we found the second smallest number of continuous frames from Step 3. We placed an upper bound of 8 frames on this value. Download Figure 1-5, TIF file.
Figure 1-6
We adjusted the number of training labels by randomly sampling different numbers of training frames. A scatter plot of the number of labeled frames vs. number of labeled neurons for each dataset shows that the number of training labels (neurons) increased as we increased the number of training frames. Each point represents a unique model. Download Figure 1-6, TIF file.
Table 1-1
The grid search values for post-processing hyperparameters were generally consistent between datasets. Values are listed as Begin:Step:End. Download Table 1-1, DOCX file.
Table 1-2
The datasets in this study covered multiple brain regions and imaging conditions. V1 was primary visual cortex, PPC was posterior parietal cortex, S1 was primary somatosensory cortex, vS1 was vibrissal primary somatosensory cortex. Download Table 1-2, DOCX file.
Supplementary 1
Download Suppl 1, ZIP file.
Figure 2-1
SAND had higher quality masks than competing methods. (A) Masks identified by SAND were more consistently shaped like neuron somas than masks identified by other methods on the ABO datasets. SUNS and SAND were both trained on 10 frames. (B) Masks identified by SAND were more consistently shaped like neuron somas than masks identified by other methods on the Neurofinder dataset. SUNS and SAND were both trained on 10 frames. (C) Masks identified by SAND were more consistently shaped like neuron somas than masks identified by other methods on the CaImAn datasets. SUNS and SAND were both trained on 10 frames for J115, K53, and YST. SUNS and SAND were both trained on 100 frames for J123. Bars represent average ratio values and error bars represent standard deviation. *** indicates p < 0.001 (Tables 2-2, 2-3, 2-4). Download Figure 2-1, TIF file.
Figure 2-2
SAND outperforms other methods on the ABO dataset when trained on fewer labeled frames. SAND had higher accuracy than other methods with low number of labeled frames on both the ABO 275 μm and ABO 175 μm datasets. Dots represent the average F1 score for each model when processing the nine test videos. Lines represent the mean F1 scores averaged over different numbers of training frames. Shaded regions represent standard error. Horizontal lines are the average F1 scores of Suite2p and CaImAn. Download Figure 2-2, TIF file.
Figure 2-3
Neuron recall with SAND outperformed unsupervised algorithms. We calculated recall as a function of neuron pSNR for each method on the ABO 275 μm dataset. SAND trained on 10 frames could more reliability detect neurons, specifically in the low pSNR regime, compared to CaImAn and Suite2p. SUNS trained on 10 frames had higher recall than SAND in the low pSNR regime, but also had lower precision than all other methods (Figure 2C). Lines represent the average recall across all 10 models for all 10 videos. Neurons were grouped by their pSNR in bins with a width of 3 (n = 3016 neurons). Download Figure 2-3, TIF file.
Figure 2-4
SAND achieved the same processing speed as SUNS on the ABO 275 μm dataset. Segmentation with SAND and SUNS consisted of three steps: pre-processing, CNN inference, and post-processing. For all steps, SAND and SUNS achieved comparable speeds. The total processing speed was an order of magnitude faster than the video’s frame rate (30 Hz). Bars represent the average processing speeds for 10 different models over 9 videos. Error bars represent the standard deviation. Download Figure 2-4, TIF file.
Table 2-1
SAND significantly outperformed SUNS on the ABO 275 µm and ABO 175 µm datasets when both trained on a small number of labels. SAND also significantly outperformed SUNS when trained on many labels. "Neuron #" indicates the range of labeled neurons used to train our models over many trials, grouped as a bin; the average number of training labels for each of these trials followed in parentheses. "Neuron #" for CaImAn and Suite2p is listed as NA because these methods were unsupervised; however, hyperparameter optimization for these unsupervised methods was done using all ground truth labels. nx shows the number of trials (models) in that bin. F1 is the median F1 value in that bin. We used a two-sided Wilcoxon rank-sum test to evaluate significance. *, **, ***, and n.s. represent p < 0.05, 0.01, 0.001, and not significant, respectively. Download Table 2-1, DOCX file.
Table 2-2
SAND had significantly higher quality masks than competing methods on the ABO datasets. We measured quality as the ratio of the mask’s area to the area of the mask's convex hull. SAND and SUNS used 10-fold cross validation to test performance; each model trained on a single video and was tested on the 9 remaining videos in that dataset (i.e. 9 applied models per video). CaImAn and Suite2p did not use cross validation (1 applied model per video). We evaluated SAND and SUNS based on models trained on 10 labeled frames. "# Training Frames" for Suite2p and CaImAn are N/A because these methods were unsupervised. We compared methods using a two-tailed Wilcoxon rank-sum test on all the masks generated by each model across all test videos. Download Table 2-2, DOCX file.
Table 2-3
SAND had significantly higher quality masks than competing methods on the Neurofinder dataset. We measured quality as the ratio of the mask's area to the area of the mask’s convex hull. SAND and SUNS used Train 1 Test 1 cross validation to test performance; each model was trained on 1 video and tested on a corresponding video (i.e. 1 applied model per video). CaImAn and Suite2p did not use cross validation (1 applied model per video). We evaluated SAND and SUNS based on models trained on 10 labeled frames. "# Training Frames" for Suite2p and CaImAn are N/A because these methods were unsupervised. We compared methods using a two-tailed Wilcoxon rank-sum test on all the masks generated by each model across all test videos. Download Table 2-3, DOCX file.
Table 2-4
SAND had significantly higher quality masks than competing methods on the CaImAn datasets. We measured quality as the ratio of the mask's area to the area of the mask's convex hull. SAND and SUNS trained on 1 video for the K53 and J115 datasets and tested on the remaining 3 videos (i.e. 3 applied models per video). SAND and SUNS trained on 3 videos for the J123 and YST datasets and tested on the held-out video (1 applied model per video). "# Training Frames" for Suite2p and CaImAn are N/A because these methods were unsupervised. We compared methods using a two-tailed Wilcoxon rank-sum test on all the masks generated by each model across all test videos. Download Table 2-4, DOCX file.
Figure 3-1
SAND outperforms SUNS on the Neurofinder dataset when trained on fewer labeled frames. SAND had higher precision and recall than SUNS when trained on only 10 labeled frames. Dots represent the average F1 score for each model when processing the nine test videos. Lines represent the mean F1 scores averaged over different numbers of training frames. Shaded regions represent standard error. Horizontal lines are the average F1 scores of Suite2p and CaImAn. Download Figure 3-1, TIF file.
Table 3-1
SAND significantly outperformed SUNS on the Neurofinder dataset when both trained on a small number of labels. SAND trained on 0-50 labels did not significantly differ from SUNS trained on 200-250 labels. "Neuron #" indicates the range of labeled neurons used to train our models over many trials, grouped as a bin; the average number of training labels for each of these trials followed in parentheses. "Neuron #" for CaImAn and Suite2p is listed as NA because these methods were unsupervised; however, hyperparameter optimization for these unsupervised methods was done using all ground truth labels. nx shows the number of trials (models) in that bin. F1 is the median F1 value in that bin. We used a two-sided Wilcoxon rank-sum test to evaluate significance. *, **, ***, and n.s. represent p < 0.05, 0.01, 0.001, and not significant, respectively. Download Table 3-1, DOCX file.
Figure 4-1
SAND outperforms SUNS on K53 and J115 when trained on fewer labeled frames. SAND had higher precision than SUNS when trained on only 10 labeled frames from the K53 and J115 videos. Dots represent the average F1 score for each model when processing the nine test videos. Lines represent the mean F1 scores averaged over different numbers of training frames. Shaded regions represent standard error. Horizontal lines are the average F1 scores of Suite2p and CaImAn. Download Figure 4-1, TIF file.
Figure 4-2
Masks generated by SAND closely matched ground truth masks on the K53 and J115 videos. (A) Example segmentations from a K53 sub-video. Masks generated by SAND were more accurate than those of other methods, even when trained on only 10 frames. The scale bar is 10 μm. (B) Example segmentations from a J115 sub-video. Masks generated by SAND were more accurate than those of other methods, even when trained on only 10 frames. For both videos, SUNS’s predictions included many false positives, while CaImAn and Suite2p had many false negatives. The scale bar is 10 μm. Download Figure 4-2, TIF file.
Figure 4-3
SAND and SUNS predicted similar neuron masks from the J123 and YST videos. (A) Example segmentations from a J123 sub-video. The scale bar is 25 μm. (B) Example segmentations from a YST sub-video. The scale bar is 10 μm. Download Figure 4-3, TIF file.
Figure 4-4
SAND greatly outperformed SUNS on datasets with high average pSNR or low variability of pSNR. Scatter plot of average vs standard error of pSNR for all neurons in each video. Dot size indicates the difference in median F1 between SAND and SUNS when trained on 0-50 neurons. Green dots indicate datasets where SAND significantly outperformed SUNS on low numbers of ground truth labels. Download Figure 4-4, TIF file.
Figure 4-5
SAND generalized well to videos with similar imaging conditions as the training data. We trained SAND (red) and SUNS (orange) on the ABO 175 μm dataset and tested the performance of those models on a dataset with similar imaging conditions (ABO 275 μm) and a dataset with different imaging conditions (K53). We evaluated these models against CaImAn (green) and Suite2p (purple) using the optimal hyperparameters for each test dataset. We also evaluated the models against SAND models that were trained on videos with the same imaging conditions as the test data (gray). SAND generalized well to datasets with similar imaging conditions to the training data. SAND outperformed unsupervised methods when generalizing to data with different imaging conditions. However, SAND performed best when training and testing videos had the same imaging conditions. Download Figure 4-5, TIF file.
Table 4-1
SAND significantly outperformed SUNS on the K53 and J115 videos when both trained on a small number of labels. SAND also significantly outperformed SUNS when trained on many labels. "Neuron #" indicates the range of labeled neurons used to train our models over many trials, grouped as a bin; the average number of training labels for each of these trials followed in parentheses. "Neuron #" for CaImAn and Suite2p is listed as NA because these methods were unsupervised; however, hyperparameter optimization for these unsupervised methods was done using all ground truth labels. nx shows the number of trials (models) in that bin. F1 is the median F1 value in that bin. We used a two-sided Wilcoxon rank-sum test to evaluate significance. *, **, ***, and n.s. represent p < 0.05, 0.01, 0.001, and not significant, respectively. Download Table 4-1, DOCX file.
Table 4-2
Calcium sensor SNR have improved over multiple generations of development. SNR values are the average fold increase of SNR relative to GCaMP3. Values are estimated from (Y. Zhang et al., 2023) and (Chen et al., 2013) and are based on single action potential results in vitro. Download Table 4-2, DOCX file.
In this issue
