Abstract
Advances in in vivo Ca2+ imaging using miniature microscopes have enabled researchers to study single-neuron activity in freely moving animals. Tools such as Minian and CalmAn have been developed to convert Ca2+ visual signals to numerical data, collectively referred to as CalV2N. However, substantial challenges remain in analyzing the large datasets generated by CalV2N, particularly in integrating data streams, evaluating CalV2N output quality, and reliably and efficiently identifying Ca2+ transients. In this study, we introduce CalTrig, an open-source graphical user interface (GUI) tool designed to address these challenges at the post-CalV2N stage of data processing collected from C57BL/6J mice. CalTrig integrates multiple data streams, including Ca2+ imaging, neuronal footprints, Ca2+ traces, and behavioral tracking, and offers capabilities for evaluating the quality of CalV2N outputs. It enables synchronized visualization and efficient Ca2+ transient identification. We evaluated four machine learning models (i.e., GRU, LSTM, Transformer, and Local Transformer) for Ca2+ transient detection. Our results indicate that the GRU model offers the highest predictability and computational efficiency, achieving stable performance across training sessions, different animals, and even among different brain regions. The integration of manual, parameter-based, and machine learning-based detection methods in CalTrig provides flexibility and accuracy for various research applications. The user-friendly interface and low computing demands of CalTrig make it accessible to neuroscientists without programming expertise. We further conclude that CalTrig enables deeper exploration of brain function, supports hypothesis generation about neuronal mechanisms, and opens new avenues for understanding neurological disorders and developing treatments.
Significance Statement
Understanding neuronal activity at the single-cell level is critical for unraveling brain mechanisms underlying behavior and neurological disorders. CalTrig, a novel GUI-based tool, integrates machine learning with Ca2+ imaging analysis to address key challenges in processing complex, large-scale neural data from freely moving rodents. By providing synchronized visualization and flexible detection of Ca2+ transients, CalTrig enables researchers without programming expertise to extract meaningful insights into brain function. This tool's adaptability and computational efficiency support diverse research applications, from behavioral neuroscience to translational studies. In brief, CalTrig enhances the precision and scalability of Ca2+ transient analysis, paving the way for deeper exploration of neuronal dynamics and facilitating advancements in neuroscience research.
Introduction
Neurons within the same brain region are diverse in type, connectivity, and activity, responding to stimuli with high temporal precision. One of the key advances in modern neuroscience is the shift from studying brain regions as functional units to focusing on individual neurons. Monitoring single-neuron activity in freely moving animals allows for deeper insights into brain function and dysfunction. Recent advances in in vivo imaging and fluorescent Ca2+ indicators, coupled with miniature microscopes (miniScope), have revolutionized the study of neural dynamics in freely moving animals. Tools have been developed to convert Ca2+ visual signals (i.e., Ca2+ imaging video files) to numerical data (e.g., changes in fluorescent signal intensity indicating the Ca2+ influx for each neuron), denoted CalV2N. Early tools, such as Suite2P (Pachitariu et al., 2016), SIMA (Kaifosh et al., 2014), STNeuronNet (Soltanian-Zadeh et al., 2019), and CalmAn (Giovannucci et al., 2019), were created to process two-photon imaging data. Due to the low resolution and poor signal-to-noise ratio (SNR) in one-photon imaging data, more specific CalV2N tools were created including MIN1PIPE (Lu et al., 2018) and Minian (Zhou et al., 2018). The algorithms used in CalV2N tools started with principal-component analysis/independent component analysis (PCA/ICA; Mukamel et al., 2009), and then upgraded to constrained non-negative matrix factorization (CNMF), and its various derivations such as CNMF-E (Zhou et al., 2018). Due to better reliability in demixing the activities of overlapping cells, the computational efficiency and accessibility of parameter adjustment, CNMF-based Minian has become a reliable choice and selected as the CalV2N tool to extract Ca2+ traces in this study.
Despite the availability of CalV2N tools, significant challenges remain in extracting meaningful insights from Ca2+ imaging data. Key issues include the lack of synchronized visualization for multiple data streams, inconsistent results due to global parameter application, and the uncertainty in verifying detected cells and Ca2+ traces. Current tools, like Minian, process data in discrete sections, limiting integrative analysis. Additionally, there are no efficient and reliable methods for identifying miniScope Ca2+ transients. These challenges are compounded by the complexity of the data, which feature high temporal (10–60 Hz) and spatial (0.8–1.0 µm pixel size) resolution, high cell throughput (50–200 neurons per animal), and the integration of behavioral tracking, resulting in vast, multifaceted datasets requiring advanced processing tools. To address these challenges at the post-CalV2N stage, we developed the Ca2+ transient identifier GUI (CalTrig), a graphical user interface (GUI) using the Python package PyQT5. CalTrig integrates all outputs from Minian (including Ca2+ imaging original processed videos, Ca2+ traces, cell footprints, and behavioral videos) to facilitate synchronized visualization, evaluate the performance of CalV2N, and identify Ca2+ transients.
In summary, CalTrig is developed to bridge the gap between CalV2N tools and final analysis stages, streamlining workflows and extracting meaningful biological insights as illustrated in Figure 1. Although the potential readership is broad, this research article primarily aims to assist neurobiologists in processing in vivo Ca2+ imaging data collected using a single-photon miniScope. We would like to give a brief introduction to the article's organization, which differs from a typical neurobiological research paper. The Materials and Methods section not only reiterates previously established procedures for collecting Ca2+ imaging data from freely moving mice and extracting the Ca2+ traces, as we reported earlier (Fang et al., 2023; Chen et al., 2024), but also provides detailed information about the newly developed tool, CalTrig. The latter includes data loading from CalV2N, data visualization, cell verification, CalV2N evaluation, Ca2+ transient identification, and exporting figures or data for statistical analysis, ready for publication. We present three strategies for Ca2+ transient identification (i.e., parameter-based, manual, and machine learning-based detection), including their procedures and applications. In the Results section, we compare the performance of multiple machine learning models, evaluate whether the established machine learning model can be used to identify transients across datasets collected at different time points, from different animals, and in various brain regions. Finally, the Discussion highlights the advantages of using machine learning models, the integrative visual exploration interface, and the unique features of CalTrig. A list of key abbreviations is provided in Table 1 to facilitate readability.
Flowchart showing the procedures of an in vivo Ca2+ study and highlighting the role of CalTrig in addressing gaps in data processing and analysis.
Materials and Methods
In vivo Ca2+ imaging data collection
Experimental animals
All in vivo procedures on laboratory animals were performed in accordance with the United States Public Health Service Guide for Care and Use of Laboratory Animals and were approved by the Institutional Animal Care and Use Committee at Indiana University School of Medicine. Ten male C57BL/6J mice, bred in-house using breeders originally derived from the Jackson Laboratory, were used in this study. The mice were group housed, except for those with GRIN lens implants, which were singly housed to prevent cage mates from damaging the implanted lens. All mice had ad libitum access to chow and water in home cage, maintained on a 12 h light/dark cycle (light on at 7:00 A.M. and off at 7:00 P.M.).
Surgical procedures
Microinjection of AAV
Mice were anesthetized with 2.5% isoflurane for induction and maintained with ∼1.2%. A 28-gauge injection needle was used to unilaterally inject the AAV1-Syn-jGCaMP8f-WPRE or AAV1-CaMKIIa-jGCaMP8f-WPRE solution (0.5 µl/site, 0.1 µl/min) via a Hamilton syringe into the secondary motor cortex (M2; coordinates in mm: AP, +1.80; ML, ±0.60; DV, −1.30) or the medial prefrontal cortex (mPFC; coordinates in mm: AP, +2.05; ML, ±0.3; DV, −2.45) using a Pump 11 Elite Syringe Pumps (Harvard Apparatus). Injection needles were left in place for 5 min following injection.
Lens implantation
A couple of minutes after withdrawing the AAV injection needle, a unilateral Gradient Refractive Index (GRIN) lens (Inscopix, #1050-004595; diameter, 1.0 mm; length, ∼4.0 mm; working distance, 200 µm) was lowered through the cranial window to 200 µm above the center of the virus injection site. The open space between the lens' side and the skull opening was sealed with surgical silicone (Kwik-Sil) and secured by dental cement (C&B Metabond). The exposed part of the lens above the skull was further coated with black cement (Lang Dental).
Base plating
Three weeks later, mice were anesthetized with isoflurane again. The cement on top of the GRIN lens was carefully removed using drill bits until the lens was exposed. The top of the lens was then cleaned using lens paper and a cleaning solution. A metal baseplate was mounted onto the skull over the lens using Loctite super glue gel, guided by a MiniScope V4.4 (Open Ephys) for optimal field of view. Once the baseplate was securely mounted, the MiniScope was removed. A protective cap was attached to the baseplate, and mice were returned to home cage.
Verification of AAV expression and the lens location
After the completion of the in vivo Ca2+ imaging, M2 or mPFC-containing coronal slices were prepared as described previously (Huang and Ma, 2023) and then fixed in 4% paraformaldehyde for no less than a couple of hours. After a brief rinse with Phosphate-Buffered Saline, slices were mounted with ProLong Gold antifade mounting reagent with DAPI (Invitrogen, catalog #P36931). Confocal imaging was performed using a Zeiss LSM 800 confocal microscope. The criteria for animal inclusion in this study were (1) highly enriched AAV expression in mostly pyramidal neurons, within-M2 or mPFC viral injection site, and (2) the footprint of the GRIN lens tip at the top of the targeting brain area.
In vivo Ca2+ imaging recording
Mice were habituated to the in vivo Ca2+ recording procedure by mounting the miniScope V4.4 (Open Ephys) to the preanchored baseplate and recording for 5 min per day at the home cage for 3 d before starting the 1 h daily recordings in an operant chamber (Med Associates). Data Acquisition box, supported by an open-source, C++ and Open Computer Vision libraries-based software, was used to collect both Ca2+ and behavioral video streams simultaneously controlled by the operant chamber software, MED-PC (Med Associates) via a TTL adaptor. The sampling frequency was 30 Hz.
Extraction of Ca2+ transient traces from the raw videos
Among multiple types of computational tools established previously to extract Ca2+ transients from raw videos, a Python-based analysis pipeline, Minian (Dong et al., 2021), was used in our data analyses due to its low memory requirement and user-friendly parameter options. In brief, there were five steps in the pipeline. First, multiple raw videos were batch loaded and subjected to a PREPROCESSING stage, where sensor noise and background fluorescence from scattered light were removed. Second, rigid brain motions were corrected by MOTION CORRECTION. Third, the initial spatial and temporal matrices for later steps were generated by a seed-based approach, called SEEDS INITIALIZATION. Fourth, the spatial footprints of cells were further refined. Fifth, the temporal signals of cells were also refined. The last two steps, the SPATIAL UPDATE, and the TEMPORAL UPDATE, as the core computational components based on CNMF algorithm, were repeated at least one more time.
CalTrig (Ca2+ transient identification GUI)
CalTrig is a Python-based open-source code with GUI interface. The repository contains documentation, demos, and a message/discussion board. The code, which is compatible with Python 3.10, uses several open-source libraries including Xarray, Numpy, Pandas, PyQt5, and Pyqtgraph.
Computer system specifications for CalTrig development and testing: The development and testing of CalTrig were conducted on a system with the following specifications: CPU, Ryzen 9 7900X; RAM, 64GB; GPU, Nvidia RTX 4080; Operating System, Windows 11.
Data loading
The front page of CalTrig serves as a hub for loading data (Fig. 2A). Data can be loaded by directly specifying the file path, or indirectly by loading the pregenerated INI (Initialization) files or JSON (JavaScript Object Notation) files.
Data loading. A, The front page of CalTrig serves as a hub for loading data. B, Each set of data can be visualized in the window by their cell footprint and are arranged in a grid related to the experiment design details, such as animal ID, day, and session stage.
The INI file uses a flat format with itemized information to detail a single dataset. It incorporates both basic experimental design details (e.g., Animal ID, day, session stage) and multiple temporally synchronized data lines, including (1) the original and CNMF processed Ca2+ image video; (2) extracted Ca2+ trace data (i.e., the CNMF output); (3) behavioral data (i.e., operant behavioral such as active lever press, ALP; inactive level press, ILP; and Reinforcement, RNF); and (4) behavioral video tracking. The variables included in each CNMF dataset are listed in Table 2 and included in Parameter list in CalTrig interface (Fig. 3). The INI file can be created by following the format provided in demo INI files, ensuring that both the experimental design and data streams are properly synchronized for exploration and analysis.
Parameter list. In top panel of the Trace Toolbox, raw signal, C signal, S signal, ΔF/F, Noise, Signal to noise ratio (SNR), as well as other behavioral readouts (e.g., active lever press, ALP; ALP during timeout; inactive lever press; and reinforcement, RNF) are listed as items to be selected to show in the Ca2+ trace window. The parameters of Savitzky–Golay filter (SavGol), used to estimate the SNR, are listed in the bottom panel.
Key abbreviations
CNMF variable
The JSON file, in contrast, uses a hierarchical format, representing the experimental design. Using data from one of our addiction projects as an example, the JSON file can collect information from multiple INI files, outlining details such as treatments for each animal, Ca2+ imaging days, and session stages. The JSON file can be generated by saving data after loading multiple INI files, either for longitudinal recordings in a single animal or for a group of animals assigned to the same experimental group.
Data visualization
As shown in Figure 2B, each set of data can be visualized in the window by their cell footprint and are arranged in a grid related to the experiment design details, such as animal ID, day, and session stage. Select the dataset to be explored by clicking the corresponding window of the footprint, and then click the “cell exploration” button to open a separate window that is subdivided into five main components (Fig. 4), including the following:
Ca2+ imaging video: toggle between original versus CNMF processed videos can be zoomed in to see more details about the cellular signal.
Behavioral video tracking, showing behaviors of the freely moving animal during Ca2+ imaging recording.
Cell list, including individual cell number identified by CNMF.
Ca2+ trace window.
Trace toolbox.
Five windows in CalTrig interface.
The following tasks can be done by the cross talk between windows:
Footprint of cells of interest: The footprint of the detected cells can be visualized by (1) clicking on the projection area of a specific cell or selecting an image region containing multiple cells in the Ca2+ imaging video or (2) choosing the cell number from the cell list. This action superimposes its corresponding footprint onto the Ca2+ imaging video. The footprint can be displayed as a solid patch, a contour or by dimming the noncell intensity.
Ca2+ traces of cells of interest: Display the Ca2+ transient traces of the cells of interest, which can be selected by clicking on the footprint area in Ca2+ imaging video, or selection of the cell number in the cell list.
Variable readouts of Ca2+ traces: Ca2+ traces can be visualized with different temporal readouts as listed in Trace toolbox, including the variables directly imported from CNMF (i.e., C signal and S signal), the variables calculated by CalTrig (i.e., ΔF/F, raw signal, SavGol filter of ΔF/F, noise, SNR).
Time stamp of behavioral events or external stimuli on Ca2+ traces: Ca2+ traces can be segmented by the behavioral readout listed in the Trace toolbox, such as ALP, ILP, and RNF.
Standardized window size of the Ca2+ trace window: The window size for the x-axis in the unit of frame, and the scale range for the y-axis, can be defined in this box. When clicking “Reset view” button, the magnification of the signal is adjusted according to the y-axis range, and the window is set up with the predefined size by anchoring the first visible frame in the Ca2+ trace window. This is considerably helpful when manually identifying or verifying Ca2+ transients.
Cell verification: Cells identified by CNMF can be individually reviewed and categorized into “Approved cells,” “Rejected Cells,” and Missed Cells” as detailed below by cross talk between multiple windows and illustrated in Figure 5.
Quality evaluation of CNMF analysis: CalTrig allows user to verify detected cells, manually identify missing cells, and refine transient detection (see details below, Cell verification and CalV2N evaluation).
Ca2+ transient identification: this can be done by parameter-based autodetection, manual detection, machine learning-based detection, or combination of multiple strategies we have developed in CalTrig.
Use CalTrig to validate and filter cells identified by CalV2N. This flowchart illustrates the workflow for validating and filtering cells detected by CalV2N using CalTrig. The process begins with in vivo calcium imaging data collection, followed by CalV2N analysis, which performs cell identification and calcium transient extraction. To assess the quality of detected cells, a visual exploration tool is used, allowing interactive inspection of calcium video, footprint visualization, behavioral video, and temporal traces, including raw signal, C, S, ΔF/F, noise, and signal-to-noise ratio. Users determine whether to accept or reject a detected cell based on visual confirmation of its footprint, calcium image, and calcium transients. If a cell is accepted, it proceeds to further processing for transient event confirmation. If rejected, the reasons for rejection are evaluated, which may include poor signal quality, suspicious footprint, or statistical outliers. The rejection process also considers whether the issue stems from suboptimal CalV2N performance, in which case parameters for calcium transient extraction can be optimized. If CalV2N errors persist, improvements in raw imaging data quality should be considered. See more details in Materials and Methods, Cell verification and CalV2N evaluation.
Cell verification and CalV2N evaluation
Cell verification.
Verified Cell: CalTrig allows user to inspect individual cells detected by CalV2N by interactive exploration of the Ca2+ imaging window and Ca2+ trace window. A cell becomes verified after a visual confirmation of its footprint, Ca2+ image, and Ca2+ trace, ensuring identifiable Ca2+ transients are present (Fig. 6). Once verified, the cell can be analyzed using tools developed for Ca2+ transient identification, as outlined in Figure 7.
Rejected Cell: Cells can be assigned to the rejection column for different reasons, including no identifiable Ca2+ transients, suspicious footprint, statistically identified as an outlier of the detected Ca2+ transients, etc. CalTrig provides the option to provide a written justification for each cell rejection, which can be used to provide feedback and potentially rerun the CNMF pipeline.
Missing Cell: Automatic time-series-based algorithms may not be sensitive enough to detect cells with low to medium SNR, or those constitutively active or inactive with minimal dynamic changes. While reviewing the Ca2+ imaging video, one may notice a potentially missed cell by CalV2N. CalTrig allows user to manually draw the contour of the suspected cell in the Ca2+ imaging window, creating its footprint (Fig. 8). The corresponding temporal trace of signal intensity for the selected area is then calculated by averaging pixel intensities and displayed in the Ca2+ trace window. If the Ca2+ traces meet the criteria for accepted cells, the manually identified cell can be added to the list of missing cells.
Evaluation of CalV2N performance
Although many CalV2N tools have been developed, there is no evaluation system available to assess the quality of cell identification and Ca2+ transient extraction. One of our initial motivations for developing CalTrig was to address this gap by creating a platform to evaluate CalV2N outputs (Fig. 5). Cells identified by CalV2N can be accepted or rejected (Fig. 9A,B, example traces), and missing cells can be manually added as described above. Further refinement of cell quality evaluation is possible after Ca2+ transient detection, using metrics such as rise time, peak amplitude, and inter-transient interval, etc., to review transient kinetics as show in Figure 9C–F. If the rate of missing cells is too high (e.g., 10% or higher) and/or the rate of acceptable cells is too low (e.g., 90% or lower), further optimization of CalV2N parameters should be considered. For example, Minian pipeline can be optimized by adjusting the following parameters (Dong et al., 2021).
ksize: Controls the denoising step, filtering out electronic noise while preserving important cell details.
wnd: Regulates the background removal step, subtracting unwanted light to isolate signals from cells.
dl_wnd: Used in the spatial update to focus on nearby cells and adjust the shape of cell signals.
sparse_penal: In the spatial update, it controls how detailed or simplified the shapes of the detected cells are. In the temporal update, it balances how much detail is included in the signals from the cells. A higher value makes the signals simpler and less detailed. The best value for this needs to be found through experimentation, as it is hard to estimate in advance.
sparsity: Ensures only the most meaningful signals are captured during the temporal update, minimizing noise.
Cell verification. CalTrig allows user to inspect individual cells detected by CalV2N by interactive exploration of the Ca2+ imaging window and Ca2+ trace window. A cell becomes verified after a visual confirmation of its footprint, Ca2+ image, and Ca2+ trace, ensuring identifiable Ca2+ transients are present. The verified cells are highlighted in green.
Use CalTrig to identify Ca2+ transients through an integrative strategy that combines manual detection, parameter-based autodetection, and machine learning-based detection. Circled numbers (1–4) mark the corresponding steps for the four applications discussed in the main text.
Add missing cell. While reviewing the Ca2+ imaging video, one may notice a potentially missed cell by CalV2N. CalTrig allows user to manually draw the contour of the suspected cell in the Ca2+ imaging window, creating its footprint. The corresponding temporal trace of signal intensity for the selected area is then calculated by averaging pixel intensities and displayed in the Ca2+ trace window. If the Ca2+ traces meet the criteria for accepted cells, the manually identified cell can be added to the list of missing cells.
Ca2+ transient dynamics in accepted and rejected cells. A, Example Ca2+ transient traces from a rejected cell. B, Example Ca2+ transient traces from an accepted cell. a → b, the dash-framed trace in a was horizontally enlarged to illustrate the measurement of ITI. b → c, the dash-framed trace in b was further horizontally enlarged to illustrate the rise and decay of a Ca2+ transient. c → d, the dash-framed trace in c was further horizontally enlarged to illustrate the measurements of peak amplitude, the rise AUC, and the rise time. C, The individual rise AUC values of Ca2+ transients from 22 accepted cells and 6 rejected cells in M2 randomly selected from a mouse, and the summary data showing the different cumulative probability between accepted versus rejected cells, indicating that accepted cells exhibit more Ca2+ transients with larger rise AUC values compared with rejected cells. D, The individual peak amplitude values of Ca2+ transients from 22 accepted cells and 6 rejected cells in M2 randomly selected from a mouse, and the summary data showing the different cumulative probability between accepted versus rejected cells, indicating that accepted cells exhibit more Ca2+ transients with larger peak amplitude values compared with rejected cells. E, The individual ITI values of Ca2+ transients from 22 accepted cells and 6 rejected cells in M2 randomly selected from a mouse, and the summary data showing the similar cumulative probability between accepted versus rejected cells. F, The individual rise time values of Ca2+ transients from 22 accepted cells and 6 rejected cells in M2 randomly selected from a mouse, and the summary data showing the different cumulative probability between accepted versus rejected cells, indicating that accepted cells exhibit less Ca2+ transients with larger long rise time compared with rejected cells.
Low SNR may also indicate poor raw data quality, which can be addressed by optimizing AAV preparation (e.g., selecting appropriate subtypes, adjusting titer and volume), improving surgical techniques (slower needle insertion/withdrawal, slower AAV delivery, minimizing bleeding), and enhancing data collection methods (adjusting focal levels, securing MiniScope anchoring, preventing cable twists via sensitive commutator, and ensuring reliable hardware connections).
Ca2+ transient identification
Transient confirmation can be done in three ways, including manually selecting the start and end points of the transients, autofiltering with specified kinetic parameters or specifying frame number ranges, or running a trained machine learning model.
Manual identification
Method: Ca2+ transients can be directly identified by manually selecting the start and end points of the rising section of Ca2+ transients (Fig. 10). Efforts have been made to improve the efficiency of manual identification. For example, the window size of the Ca2+ can be preset at a desired size (e.g., 1,000 frames per window width in the x-axis, −1 to 20 as the signal range in the y-axis), which will allow the visual impression of Ca2+ at different time stage, different animals are comparable, assisting a consistent decision-making on Ca2+ manual identification. We also set up the shortcut keys to switch the screen window in the Ca2+ trace panel. Specifically, clicking A and F allows to jump to the first or the last window, respectively, and clicking S and D allows to jump to the previous or the next screen window, respectively. To account for inaccuracies resulting from human interaction, the selected points would automatically position themselves within a 20-frame window to the local maxima or minima, dependent on their order.
Manual identification of Ca2+ transients. Ca2+ transients can be directly identified by manually selecting the start and end points of the rising section of Ca2+ transients.
Applications for manual identification: First, directly label the Ca2+ transient at the beginning stage of the project when no reference in setting up the parameters for autodetection or no dataset are available to training the machine learning model. It takes ∼5 min to go through a 15 min Ca2+ traces. Second, correct, add, or remove the transient spikes after running the parameter-based autodetection to establish the ground truth for training the machine learning model. This would primarily apply to problematic spikes around the specified parameter threshold or those whose rising part is too slow to be confidently identified as a transient event. It takes ∼1 min to go through a 15 min Ca2+ traces. Third, to further improve the Ca2+ transient identification after processed by an established machine learning model. We expect this reviewing and correction process can be done on 2–4 cells per min. Fourth, when we need to extend the application of a well-trained machine learning model to a different brain region, a different cell type, etc., the machine learning model may provide a compromised predictability in detecting Ca2+ transients. Adding a few traces with the manually identified Ca2+ transient will set up a new dataset as ground truth for establishing an updated machine learning model to be used in a new task. These four applications of manual identification are illustrated in Figure 7, as indicated by the circled numbers 1–4.
Parameter-based automatic identification
Although the variability in Ca2+ transient characteristics is well recognized, the fundamental dynamic parameters of Ca2+ transients should fall within a certain range. Thus, establishing key dynamic parameters as a digital filter can streamline the Ca2+ transient identification process, improving both accuracy and efficiency.
Three Parameters. “Peak Threshold (ΔF/F)”: First, let us define ΔF/F, which represents the relative change in fluorescence intensity, where:
F0 is the background fluorescence intensity, which is calculated using a moving percentile provided by Caiman (Giovannucci et al., 2019).
ΔF is the difference between the observed fluorescence intensity (F) at a given time and the background fluorescence intensity (F0).
The formula is as follows:
“Interval Threshold”: This specifies the minimum inter-transient interval (ITI), measured as the frame distance between the initial fames of two adjacent Ca2+ transients. If this distance is shorter than the threshold, the transients are merged, with the lower peak combined into the higher peak. However, a peak occurring within the prolonged decay slope of a preceding transient but exceeding the interval threshold is counted as an independent event. Its amplitude is calculated as the difference between its peak ΔF/F and the ΔF/F at its rising phase onset.
“SNR Threshold”: This sets the minimal Signal-to-Noise ratio (SNR). Using the Savitzky–Golay filter (Steinier et al., 1972; Dai et al., 2017) to smooth ΔF/F signals, noise is calculated as the difference between the original and filtered ΔF/F, further smoothed by a rolling window strategy. Then SNR is computed by dividing the smoothed ΔF/F by the estimated noise. See more details below and in Figure 11.
Savitzky–Golay filter. This sets the minimal signal-to-noise ratio (SNR). Using the Savitzky–Golay filter (Steinier et al., 1972; Dai et al., 2017) to smooth ΔF/F signals, noise is calculated as the difference between the original and filtered ΔF/F, further smoothed by a rolling window strategy. Then SNR is computed by dividing the smoothed ΔF/F by the estimated noise.
Signal-to-noise estimation
While verifying transient events, occasional spikes were detected due to background fluctuations rather than specific cellular activity, and in rare cases, noise was misinterpreted as calcium signals. Contemporary methods often use z-scores to estimate noise, but this global noise estimation lacks robustness when noise fluctuates over time, potentially leading to the rejection of valid transients during low-noise periods.
To address this concern, we apply the Savitzky–Golay (SavGol) filter (Fig. 11), which smooths noisy data while preserving important features such as peaks and edges (Steinier et al., 1972; Dai et al., 2017). This filter fits successive polynomial functions to adjacent data points, making it ideal for time-series data where preserving peak shape is crucial. In our pilot studies, where we compared the original ΔF/F with the smoothed ΔF/F using CalTrig's data exploration tools, the Ca2+ transients were well preserved. Therefore, the SavGol filter was chosen as the primary smoothing filter.
Noise estimation for each image frame is calculated by the difference between the original and SavGol-filtered ΔF/F:
The parameters for the SavGol filter and noise smoothing can be adjusted under the “SavGol” and “Noise” subtabs in the Trace toolbox window. Visualization options for the SavGol-filtered ΔF/F, noise, and SNR are available under the “Params” subtab for easy analysis in the Ca2+ trace window.
At the initial experimental stage, parameter settings are determined using manual identification on a limited number of cells or demo data. As more cells are manually identified, these parameters can be refined to enhance the effectiveness of parameter-based autodetection.
Method: Through manual identification, the C signal from CNMF was found to reliably predict Ca2+ transients in most cases but prone to false positives. To enhance detection accuracy, the parameter-based autodetection algorithm starts by including all C peaks in a candidate pool, then filtering down to a valid subset based on three predefined parameters: “Peak Threshold (ΔF/F),” “Interval Threshold,” and “SNR Threshold” (Fig. 12). This is achieved through the following steps.
Initial peak detection through local maxima calculation of the C signal.
Refine peak selection from left to right by looping the following steps.
Remove erroneous peaks if the S signal is zero.
Allocate overlapping, continuous S signal to the current peak.
Check if the distance to the next peak is shorter than the predefined “Interval Threshold.”
If so, merge the current and subsequent peaks, selecting the taller peak.
If not, use the S signal to determine the start and end of the transient.
Accept the transient selection if in the defined boundaries, the values surpass both the predefined “Peak Threshold (ΔF/F)” and “SNR Threshold.”
Automatic detection. To enhance detection accuracy, the parameter-based autodetection algorithm starts by including all C peaks in a candidate pool and then filtering down to a valid subset based on three predefined parameters: “Peak Threshold (ΔF/F),” “Interval Threshold,” and “SNR Threshold.”
Application: Parameter-based Ca2+ transient detection is faster than manual detection, but less predictive than using a well-trained machine learning model. It is particularly feasible after manual identification of a few cells, enabling parameter tuning. Parameter-based detection can then serve as a predetection step to accelerate further manual identification, eventually creating a ground-truth dataset for training machine learning models. This method also helps refine or update machine learning models for different experimental conditions (Figs. 15, 18⇓–20).
Detection using machine learning model
Feature selection for machine learning model: In selecting the appropriate architecture for the machine learning model, we aimed to incorporate key signals gained from the “ground truth” Ca2+ transient verification by manual detection process. We identified two signals, i.e., C and ΔF/F, as sufficient for decision-making. The determination of whether a given time-step contains a Ca2+ transient can be inferred from a 100-frame window centered by the frame of interest. A corresponding dataset, denoted E, was created by marking the initial and peaking frames within the context of a dynamic rise in Ca2+ transients.
Selection of machine learning models: Ca2+ transient data are inherently time-series data with a long sequence of many neurons firing at different times, where the signal at any given frame is highly relevant to past and upcoming activity. Thus, Recurrent Neural Networks (RNNs) (Durstewitz et al., 2023) and Transformers (Chandra et al., 2023; Bian et al., 2024) are selected as the candidate machine learning models for Ca2+ transient detection due to their ability or potential to handle the temporal dynamics of Ca2+ signals in neurons (Fig. 13).
Flowchart demonstrating Ca2+ transient validation via machine learning models. A, Application of RNNs in predicting Ca2+ transients. The input consists of the C signal and ΔF/F traces. A sequence of GRUs or LSTM modules processes the temporal dynamics of the Ca2+ transient. The network includes multiple stacked layers, with fully connected layers (FNN) at the final stage to produce the predicted Ca2+ transient. B, Application of Transformer models for predicting Ca2+ transients. The input is first expanded through a dimensional transformation step, followed by repeated attention-based self-mapping in either a standard Transformer or a local Transformer architecture. The processed features are then passed through a feedforward network (FNN) to generate the predicted Ca2+ transient. C, D, Diagrams showing the internal operations of two variants of RNN, i.e., GRU module (C) and LST module (D). The GRU uses two gates: the update gate, which controls the amount of information passed to the next step, and the reset gate, which determines how much of the previous information to forget. These gates modify the hidden state H(t) at the current time-step based on the input X(t) and the previous hidden state H(t−1), leading to a new hidden state H(t). The LSTM module architecture is depicted, showcasing its cell structure. It uses three gates: forget, input, and output gates to control the flow of information. The input X(t), the previous cell state C(t−1), and the previous hidden state H(t−1) are processed through these gates to update the cell state C(t) and produce the new hidden state H(t). E, F, Diagram showing the Transform (E) and local Transformer (F) mechanisms. Multiple columns represent the same set of data. Nontranslucent colored regions highlight the specific values being processed, while translucent colors represent comparison values within the self-attention mechanism. No color denotes values excluded from evaluation, enforcing global attention to all time points. Local Transformer mechanism. Unlike the standard Transformer, the local Transformer focuses only on a subset of adjacent time points, restricting the attention scope to a local temporal window. This reduces computational complexity while maintaining essential local dependencies in Ca2+ transient predictions. Data input format: The models receive sequential time-series data of C signals and ΔF/F traces as input. Data output format: The models predict Ca2+ transients by generating the time stamp of the initial and ending frame for each Ca2+ transients, which will be saved in E. FNN, Feedforward neural network.
The core of its architecture lies in the RNN cells and its gating mechanisms, which processes inputs sequentially, creating a representation of the data in the context of the preceding timestamps. In the case of Ca2+ transient data, two features (i.e., C, ΔF/F) used for training will be incorporated into the hidden state (H) and passed to the subsequent cell (C). We tested two RNN variants: Long Short-Term Memory (LSTM; Jiang et al., 2020; Alfattni et al., 2021) and GRU (Wang et al., 2020; Merkelbach et al., 2023). Given the importance of both forward and backward context in detecting Ca2+ transients, we employed a bidirectional RNN. The outputs from both directions are concatenated and passed through a feedforward layer to generate the final prediction.
One significant limitation of the RNN is due to the fixed size of its hidden state and the sequential nature of the architecture, causing early time-step bias. To address this, we trained a Transformer, which uses self-attention to compare the relationship between all input values simultaneously, rather than processing sequentially (Choi and Lee, 2023; Liu et al., 2023). This approach allows the model to attend to both low- and high-intensity values that occur at different temporal distances from the Ca2+ transient, improving prediction accuracy. Recent advances in Natural Language Processing have introduced the Local Transformer, which limits attention to a subset of nearby inputs (Boffi and Vanden-Eijnden, 2024; Zhao et al., 2024). This model could help reduce errors by focusing on local context and ignoring distant irrelevant signals. However, the self-attention mechanism in the Transformer obscures positional information; hence, we need positional embeddings. The trainable embeddings were used in our Transformer. Additionally, we observed that the small dimensionality of the original input data negatively affected training, likely due to the model's inability to effectively encode positional information in low-dimensional data. To address this, we introduced a preliminary dimension expansion layer, which projects the input into a higher-dimensional space before passing it through the encoding layers.
Both RNN and Transformer architectures were implemented in CalTrig using PyTorch (Lam et al., 2022; Orken et al., 2022). For the Transformer, we used the Transformer Encoder module, and for the Local Transformer, we adapted a modified version of the model (Wang, 2020). Parameters for machine learning model training are listed in Table 3.
Parameters for machine learning model training
Data Preprocessing: The input data consisted of two signals (i.e., C, ΔF/F) representing 15 min time intervals of 27,000 frames per cell, captured at a sampling rate of 30 Hz. Each segment was normalized relative to the highest value within the cell, per data type. We tested two data generation and loading approaches:
Discrete Sample Chunking: We defined three variables—sequence length, slack, and rolling parameter. The sequence length specifies the number of frames the model is trained on to make predictions, typically set to ∼200 frames (∼6.6 s) in our testing. Considering the slow dynamics of Ca2+ transients, which may last 10–15 s, increasing the segment length could be explored in future optimizations to further improve transient detection. The slack variable determines the extra context length on either side of a given sequence, providing necessary context for making predictions without being used for the predictions themselves. This slack is usually set between 50 and 100 frames. For sequences at the edges of the 27,000-frame segment, zero padding is applied to match the slack length and maintain consistency. Lastly, the rolling parameter defines the windowing approach for generating data, allowing overlapping sequences to be extracted for model training.
Ca2+ transient events account for only 2–3% of the overall data. To avoid biasing the model toward detecting non-Ca2+ transients, we initially applied class weighting, which resulted in a poor precision score. We determined that it was due to the underemphasizing of noisy data within training, whose signal characteristics were more similar to transient activity rather than an empty signal. This resulted in a model that considered any activity including noise to be a transient event. We opted instead to implement stratification where we ensured that only samples containing ground-truth transient events or positive values from the C array were included. The C array was used as a reference to give the model insight into problematic spikes identified by the CNMF process but deemed invalid by the verifier. During classification, we average all outputs for a single time-step to address overlapping predictions.
Key metrics for validation: We used Precision, Recall, F1, and macro F1 as key metrics to evaluate the performance of machine learning model in predicting Ca2+ transients versus non-Ca2+ transients (Fig. 14). There are four types of predictions as shown in Tables 4 and 5: true positive (TP) predicting the positive as positive, true negative (TN) predicting the negative as negative, false positive (FP) predicting the negative as positive, and false negative (FN) predicting the positive as negative.
Machine learning evaluation. We used Precision, Recall, F1, and macro F1 as key metrics to evaluate the performance of machine learning model in predicting Ca2+ transients versus non-Ca2+ transients.
Event to be predicted: Ca2+ transients
Event to be predicted: non-Ca2+ transients
Precision measures how many of the positive predictions (TP + FP) are actually positive (TP). (It focuses on the quality of the positive predictions.)
GRU module has the best predictability of Ca2+ transients. A, Shared strategy for assigning data to training, validation, and testing across four machine learning models. B, C, Precision varied significantly in Ca2+ transient prediction (B, F(3,36) = 14.2, p < 0.01), but remained similar in no Ca2+ transient prediction (C, F(3,36) = 0.4, p = 0.78) in four machine learning models. D, E, Recall remained similar in Ca2+ transient prediction (D, F(3,36) = 0.6, p = 0.63) but varied in Ca2+ transient prediction (E, F(3,36) = 9.4, p < 0.01) in four machine learning models. F,G, Significant differences in F1 scores of Ca2+ transient prediction (F, F(3,36) = 9.3, p < 0.01) and no Ca2+ transient prediction (G, F(3,36) = 4.8, p < 0.01) in four machine learning models. H, Significant differences in macro F1 scores (F(3,36) = 9.3, p < 0.01) in four machine learning models. Each machine learning model was trained using data from 226 cells from 4 mice, with 28 cells allocated to the validation set and 28 cells to the testing set. Each experimental group consisted of 10 tests. Data were analyzed by one-way ANOVA, followed by Bonferroni post hoc test. *p < 0.05; **p < 0.01.
Data export
All the detected Ca2+ transient information can be extracted. A few examples are listed below. This will directly aid in statistical analyses and figure preparation for sharing and publishing.
Animal-wide data export
A data table is generated, with each row representing one of the CalV2N-detected cells and each column providing specific readout as a general description of that cell (Fig. 16A). From left to right, the columns include the following: Cell ID, Cell Size (number of pixels), Footprint Location (x, y), Total Ca2+ Transient Count, Frequency (Hz), Average Amplitude (ΔF/F), Average Rising (# of frames), Average Rising Time (seconds), Average Interval (seconds), Standard Deviation (denoted Std, ΔF/F), Mean Absolute Deviation (denoted MAD, ΔF/F), Average Peak Amplitude (ΔF/F), and Category (Verified, Rejected, Missing). The data can be easily copied to the clipboard for further processing in other applications like Excel (Microsoft 365) or Prism (GraphPad). Figures, such as box plots showing the 25, 50, and 75% values for metrics like Average Peak Amplitude or ITI (Fig. 16B,C), can be directly generated and saved as editable SVG files for publication purposes.
Animal-wide data export. A, A data table is generated, with each row representing one of the CalS2N-detected cells and each column providing specific readout as a general description of that cell. B, C, Average peak amplitude (B) and inter-transient interval (ITI, C) can be directly generated and saved as edit able SVG files for publication purposes.
Cell-wide data export
A data table is generated for a selected cell, with the top row displaying column titles and the subsequent rows representing individual Ca2+ transients detected by CalTrig, one transient per row (Fig. 17A). The columns include the following: Rising Start (frames), Rising Stop (frames), Total Rising Frames, Rising Start (seconds), Rising Stop (seconds), Total Rising Time (seconds), Interval with Previous Transient (frames), Peak Amplitude (ΔF/F), and Total Amplitude (ΔF/F, calculated as the area under the rising slope). The column title can be selected to sort the data based on the information in the selected column (Fig. 17B). This table can be copied to the clipboard for further processing in applications like Excel (Microsoft 365) or Prism (GraphPad). Figures such as Amplitude Distribution or ITI Frequency Histograms (Fig. 17C,D) can be created directly from this data and saved as editable SVG files for publication.
Cell-wide data export. A, A data table is generated for a selected cell, with the top row displaying column titles and the subsequent rows representing individual Ca2+ transients detected by CalTrig, one transient per row. B, The column title (e.g., Total Amplitude) can be selected to sort the data based on the information in the selected column. C, D, Figures such as amplitude distribution (C) and ITI frequency histograms (D) can be created directly from this data and saved as editable SVG files for publication.
Maximum projection image
An editable image file, including the footprints of detected or verified cells for an animal, can be created for data sharing and publication purposes.
Statistical analysis
Data were collected from 10 mice in vivo, shown as mean ± SEM in curve graphs (Figs. 19B,C,F,G,J,K,N, 20B,C,F,G,J,K,N) or the quartile in violin plots (Figs. 15, 18, 19D,E,H,I,L,M,O, 20D,E,H,I,L,M,O). Using GraphPad Prism 10, statistical significance was assessed by one-way ANOVA or two-way ANOVA, followed by Bonferroni post hoc tests. Statistical significance was considered to be achieved if p < 0.05. All raw data for TN, TP, FN, and FP are available in an Excel file, which can be downloaded via the following link: https://github.com/cci-MaLab/Calcium-Transient-Analysis/raw/refs/heads/main/ml_training/caltrig_results/TN%20TP%20FN%20FP%20Raw%20data.xlsx
Code/software
CalTrig is available in our lab Github station (https://github.com/cci-MaLab/Calcium-Transient-Analysis).
Results
Comparison of the four machine learning models
Our goal was to find an efficient machine learning model with high predictability in detecting the Ca2+ transients while minimizing computational demands. To achieve this, we compared the performance of two RNN variants, GRU and LSTM, alongside two Transformer models, the standard Transformer and local Transformer. We used a dataset of 226 cells with manually confirmed Ca2+ transient as “ground” truth (see more details in Materials and Methods; Fig. 15A). Our data showed that the macro F1 score for predicting Ca2+ transients and no Ca2+ transients was higher with RNN (GRU, 0.948 ± 0.002; LSTM, 0.946 ± 0.004; mean ± SEM, same below) compared with the standard Transformer (0.919 ± 0.005), which was partially improved by local Transformer (0.935 ± 0.005; Fig. 15H).
Specifically, for Ca2+ transient prediction, the F1 score was higher with RNN (GRU, 0.900 ± 0.005; LSTM, 0.894 ± 0.008) compared with the standard Transformer (0.841 ± 0.010), which was partially improved by local Transformer (0.873 ± 0.010; Fig. 15F). In terms of Precision, both GRU (0.873 ± 0.008) and LSTM (0.860 ± 0.010) models outperformed the standard Transformer (0.771 ± 0.015) and the local Transformer (0.835 ± 0.013; Fig. 15B). However, the Recall scores across four machine learning models were similar (GRU: 0.926 ± 0.008; LSTM: 0.931 ± 0.010; standard Transformer: 0.929 ± 0.009; local Transformer: 0.916 ± 0.010; Fig. 15D), indicating that the F1 score differences were mainly driven by Precision.
For predicting non-Ca2+ transients, the precision, recall, and F1 scores were all near perfect (>0.990), due to the consistent nature of non-Ca2+ transient signals. When analyzing F1 scores specifically for non-Ca2+ transient prediction, similar trends as the macro F1 scores were observed across the four machine learning models (Fig. 15G), with differences primarily driven by Recall (Fig. 15E) rather than Precision (Fig. 15C).
Compared with Transformer models, RNN models exhibited higher Precision in detecting Ca2+ transients and better Recall for non-Ca2+ transients, both attributable to RNN's ability to minimize misclassification of the non-Ca2+ transient signal as Ca2+ transient signals. The consistent good Recall for Ca2+ transients and Precision values for non-Ca2+ transients across all machine learning models indicates that they have a low likelihood of misclassifying Ca2+ transients as non-Ca2+ transients. The primary challenge in identifying Ca2+ transients lies in reducing errors where non-Ca2+ transients are mistakenly classified as transients. RNN models outperformed Transformer in addressing this issue, which was partially improved when using the local Transformer. Given its lower variability and faster processing time, GRU is recommended over LSTM. Therefore, all subsequent analyses were conducted using the GRU model.
Optimization of the cell number of the training dataset
To determine the optimal number of cells to be used as the training dataset, we tested the predictability of GRU model trained by varying numbers of cells, ranging from 1 to 20. When the testing datasets were randomly sampled from the same session stage (i.e., the first 15 min or the last 15 min) in the same 1 h daily session on either Day 1 or Day 5, we observed a significant improvement in the macro F1 score when 5–20 cells were sampled, compared with GRU models trained with only 1 or 2 cells. However, no differences were observed among models trained with 5–20 cells (macro F1 score: 1 cell, 0.817 ± 0.014; 2 cells, 0.885 ± 0.008; 5 cells, 0.926 ± 0.005; 10 cells, 0.945 ± 0.004; 15 cells, 0.947 ± 0.004; 20 cells, 0.946 ± 0.004; Fig. 18H). This cell number-dependent predictability is primarily attributable to the model's predictability of the Ca2+ transient indicated by its Precision (Fig. 18B), Recall (Fig. 18D), and F1 scores (Fig. 18F) for Ca2+ transients, as the Precision (Fig. 18C), Recall (Fig. 18E), and F1 scores (Fig. 18G) for non-Ca2+ transient were all near perfect regardless of the cell number of the training dataset. We further explored the effects of cell number when testing cells were randomly sampled from the different session stages within the same 1 h daily session, the same session stage on a different daily session, and the different sessions stage on a different daily session of the same mouse (Fig. 19A). In all of these three cases, we found significant improvements of both Ca2+ transient prediction (Fig. 19B,F,J,N) and the non-Ca2+ transient predictions, though the latter were near perfect in most instances (Fig. 19C,G,K,O), when five or more cells were sampled for training, compared with the models trained with only 1 or 2 cells. Finally, we extended further by randomly sampling the training datasets from different mice (Fig. 20A). We found significant improvements of Ca2+ transient prediction (Fig. 20B,F,J,N) and the non-Ca2+ transient prediction, though the latter was near perfect in most instances (Fig. 20C,G,K,O), when using five or more cells, regardless of whether the testing cells were from the same or different brain regions in mice distinct from those used for training. In conclusion, the number of cells used for training is crucial for accurately predicting Ca2+ transients. A minimum of 10 cells appears to be sufficient for achieving macro F1 scores above 0.900 when training the GRU model.
Prediction of Ca2+ transients or no Ca2+ transients across different numbers of cells in machine learning training. A, Strategy of randomly picking up cells for training, validation, and testing within the same session stage on the same day. B,C, Increasing the number of cells in machine learning training model significantly improved the precision in predicting Ca2+ transients (B, F(5,234) = 11.0, p < 0.01) and no Ca2+ transients (C, F(5,234) = 12.8, p < 0.01). D, E, Increasing the number of cells in machine learning training model significantly improved the recall in predicting Ca2+ transients (D, F(5,234) = 40.9, p < 0.01) and no Ca2+ transients (E, F(5,234) = 2.8, p = 0.02). F,G, Increasing the number of cells in machine learning training model significantly improved the F1 scores in predicting Ca2+ transients (F, F(5,234) = 46.7, p < 0.01) and no Ca2+ transients (G, F(5,234) = 12.0, p < 0.01). H, Increasing the number of cells in machine learning training model significantly improved the macro F1 scores in predicting Ca2+ transients and no Ca2+ transients (H, F(5,234) = 46.6, p < 0.01). Data were obtained using the GRU model. Each experimental group consisted of 10 tests per mouse, with 4 mice per group, resulting in a total of 40 data points per experimental group. Data were analyzed by one-way ANOVA, followed by Bonferroni post hoc test. *p < 0.05; **p < 0.01, compared with the machine learning model trained by 1 cell. #p < 0.05; ##0.01, compared with the machine learning model trained by 2 cells. 40 testing cells in each group.
Prediction of Ca2+ transients or no Ca2+ transients within the same mice. A, Four strategies for sourcing cells within the M2 area from a mouse for machine learning model training, validation, and testing. B–E, The training cell number, and the testing cell source, but not their interactions, significantly affected the Precision in predicting Ca2+ transients (B, training cell # F(5,936) = 21.4, p < 0.01; testing cell source F(3,936) = 10.0, p < 0.01; training cell number × testing cell source interaction F(15,936) = 0.8, p = 0.64). The training cell number, but not the testing cell source or their interactions, affected the Precision in predicting no Ca2+ transients (C, training cell number F(5,936) = 71.7, p < 0.01; testing cell source F(3,936) = 1.1, p = 0.36; training cell number × testing cell source interaction F(15,936) = 0.3, p = 0.99). When 20 cells were included in the machine learning training model, the Precision in predicting Ca2+ transients (D, F(3,156) = 2.7, p = 0.046), but not in predicting no Ca2+ transients (E, F(3,156) = 0.2, p = 0.93), in testing cells from different session stage on different day reduced. F–I, The training cell number, but not the testing cell source or their interactions, significantly affected the Recall in predicting Ca2+ transients (F, training cell number F(5,936) = 126.0, p < 0.01; testing cell source F(3,936) = 0.8, p = 0.49; training cell number × testing cell source interaction F(15,936) = 0.3, p = 0.99). The number of cells, the testing cell source, but not their interactions, significantly affected the Recall in predicting no Ca2+ transients (G, training cell number F(5,936) = 2.7, p = 0.02; testing cell source F(3,936) = 11.4, p < 0.01; training cell number × testing cell source interaction F(15,936) = 1.5, p = 0.10). When 20 cells were included in the machine learning training model, the Recall in predicting no Ca2+ transients (I, F(3,156) = 3.3, p = 0.02), but not in predicting the Ca2+ transients (H, F(3,156) = 0.4, p = 0.78), in testing cells from different session stage on different day reduced. J–M, The training cell number and the testing cell source, but not their interactions, significantly affected the F1 scores in predicting Ca2+ transients (J, training cell number F(5,936) = 130.0, p < 0.01; testing cell source F(3,936) = 3.0, p = 0.03; training cell number × testing cell source interaction F(15,936) = 0.7, p = 0.74). Similarly, the number of cells and the testing cell source, but not their interactions, significantly affected the F1 scores in predicting no Ca2+ transients (K, training cell number F(5,936) = 32.3, p < 0.01; testing cell source F(3,936) = 8.8, p < 0.01; training cell number × testing cell source interaction F(15,936) = 1.2, p = 0.28). When 20 cells were included in the machine learning training model, the Recall in predicting Ca2+ transients (L, F(3,156) = 2.7, p = 0.048) and no Ca2+ transients (M, F(3,156) = 2.8, p = 0.040) in testing cells from different session stage on different day reduced. N, O, The training cell number, and the testing cell source, but not their interactions, significantly affected the macro F1 scores in predicting Ca2+ transients and no Ca2+ transients (N, training cell number F(5,936) = 128.5, p < 0.01; testing cell source F(3,936) = 3.1, p = 0.03; training cell number × testing cell source interaction F(15,936) = 0.8, p = 0.73). When 20 cells were included in the machine learning training model, the macro F1 scores in predicting Ca2+ transients and no Ca2+ transients (O, F(3,156) = 2.8, p = 0.041) in testing cells from different session stage on different day reduced. P, Legends showing the color-coded abbreviations for four experimental groups. Data were obtained using the GRU model. Each experimental group consisted of 10 tests per mouse, with 4 mice per group, resulting in a total of 40 data points per experimental group. Data were analyzed by two-way ANOVA (B,C,F,G,J,K,N) or one-way ANOVA (D,E,H,I,L,M,O), followed by Bonferroni post hoc test. *p < 0.05; **p < 0.01. 40 testing cells in each group.
Prediction of Ca2+ transients or no Ca2+ transients in different mice. A, Three strategies for sourcing testing cells (smSS::smDay, between mice, different regions, see more details in panel P). B–E, The training cell number, and the testing cell source, but not their interactions, significantly affected the Precision in predicting Ca2+ transients (B, training cell number F(5,942) = 24.4, p < 0.01; testing cell source F(2,942) = 19.2, p < 0.01; training cell number × testing cell source interaction F(10,942) = 1.1, p = 0.37) and the no Ca2+ transients (C, training cell number F(5,942) = 74.1, p < 0.01; testing cell source F(2,942) = 11.6, p < 0.01; training cell number × testing cell source interaction F(10,942) = 1.0, p = 0.43). When 20 cells were included in the machine learning training model, the Precision in predicting Ca2+ transients (D, F(2,157) = 4.7, p = 0.01) and no Ca2+ transients (E, F(2,157) = 6.2, p < 0.001) in testing cells from different regions in different mice reduced. F–I, The training cell number, but not the testing cell source or their interactions, significantly affected the Recall in predicting Ca2+ transients (F, training cell number F(5,942) = 145.6, p < 0.01; testing cell source F(2,942) = 3.7, p = 0.02; training cell number × testing cell source interaction F(10,942) = 0.7, p = 0.68). The training cell number and the testing cell source, but not their interactions, significantly affected the Recall in predicting no Ca2+ transients (G, training cell number F(5,942) = 8.7, p < 0.01; testing cell source F(2,942) = 22.9, p < 0.01; training cell number × testing cell source interaction F(10,942) = 0.9, p = 0.58). When 20 cells were included in the machine learning training model, the Recall in predicting Ca2+ transients (H, F(2,157) = 4.4, p = 0.01) and no Ca2+ transients (I, F(2,157) = 5.3, p < 0.01) in testing cells from different regions in different mice reduced. J–M, The training cell number, and the testing cell source, but not their interactions, significantly affected the F1 scores in predicting Ca2+ transients (J, training cell number F(5,942) = 165.7, p < 0.01; testing cell source F(2,942) = 17.6, p < 0.01; training cell number × testing cell source interaction F(10,942) = 0.3, p = 0.98) and the no Ca2+ transients (K, training cell number F(5,942) = 55.8, p < 0.01; testing cell source F(2,942) = 40.4, p < 0.01; training cell number × testing cell source interaction F(10,942) = 1.3, p = 0.23). When 20 cells were included in the machine learning training model, the F1 scores in predicting Ca2+ transients (L, F(2,157) = 8.9, p < 0.01) and no Ca2+ transients (M, F(2,157) = 9.3, p < 0.01) in testing cells from different regions in different mice reduced. N, O, The training cell number, and the testing cell source, but not their interactions, significantly affected the macro F1 scores in predicting Ca2+ transients and no Ca2+ transients (N, training cell number F(5,942) = 164.6, p < 0.01; testing cell source F(2,942) = 18.3, p < 0.01; training cell number × testing cell source interaction F(10,942) = 0.3, p = 0.98). When 20 cells were included in the machine learning training model, the macro F1 scores in predicting Ca2+ transients and no Ca2+ transients (O, F(2,157) = 9.0, p < 0.01) in testing cells from different regions in different mice reduced. P, Legends showing the color-coded abbreviations for four experimental groups. Data were obtained using the GRU model. Each experimental group consisted of 10 tests per mouse, with 4 mice for smSS::smDay, and 6 mice for smRegion::dfMouse and dfRegion::dfMouse, resulting in a total of 40 data points for smSS::smDay and 60 data points for smRegion::dfMouse and dfRegion::dfMouse. Data were analyzed by two-way ANOVA (B,C,F,G,J,K,N) or one-way ANOVA (D,E,H,I,L,M,O), followed by Bonferroni post hoc test. *p < 0.05; **p < 0.01. 40 testing cells in each group.
Sharing the machine learning model for predicting testing cells across different time windows in the same mouse
To assess how well a GRU model trained on data from a section of a 1 h daily session can predict Ca2+ transients from any other recordings in the same mouse, we randomly sampled testing cells under four conditions: (1) the same session stage in the same daily session (smSS::smDay), (2) a different session stage in the same daily session (dfSS::smDay), (3) the same session stage in a different daily session (smSS::dfDay), and (4) a different session stage in a different daily session (dfSS::dfDay; Fig. 19A). When 20 cells were sampled, the macro F1 scores were 0.946 ± 0.004 for smSS::smDay, 0.936 ± 0.006 for dfSS::smDay, 0.934 ± 0.005 for smSS::dfDay, and 0.927 ± 0.008 for dfSS::dfDay (Fig. 19N). Thus overall, the predictability remained high when testing cells were sampled from the same mouse used for GRU model training, though the macro F1 score for predicting both Ca2+ transients and non-Ca2+ transients was affected statistically, but mildly, by the source of testing cells (Fig. 19N,O).
Specifically, for predicting Ca2+ transients, the source of testing cells affected the F1 scores (Fig. 19J,L), particularly the Precision (Fig. 19B,D) rather than the Recall (Fig. 19F,H). When smSS::smDay was set as the gold standard with 20 cells as the benchmark, similar predictability was observed when testing cells were sampled from dfSS::smDay or smSS::dfDay. However, lower Precision and F1 score, but comparable Recall, were noted when testing cells were sampled from dfSS::dfDay.
For predicting non-Ca2+ transients, the source also affected the F1 scores (Fig. 19K,M), primarily impacting the Recall (Fig. 19G,I), while the Precision remained largely unaffected (Fig. 19C,E). Similarly, as what was observed for predicting Ca2+ transients, we found the non-Ca2+ transient prediction was comparable when testing cells were sampled from smSS::smDay, dfSS::smDay, or smSS::dfDay. Lower Recall and F1 score, but similar Precision, for non-Ca2+ transient prediction was detected when testing cells were sampled from dfSS::dfDay.
In conclusion, the GRU model can effectively predict Ca2+ transients in cells from the same mouse used for training. However, the risk of misclassifying non-Ca2+ transients as Ca2+ transients increases, though within a narrow range, when testing cells are sampled from a different session stage and a different day.
Sharing the machine learning model for predicting testing cells from different mice
To evaluate how the GRU model can predict the Ca2+ transients from a different mouse, either sharing the same brain region or switching to a different brain region, we used the smSS::smDay sampling strategy as the benchmark and added two more strategies in sampling testing cells: (1) the same brain region in a different mouse (denoted smRegion::dfMouse) and (2) a different brain region in a different mouse (denoted dfRegion::dfMouse). When 20 cells were sampled, the macro F1 scores were 0.946 ± 0.004 for smDay::smSS, 0.947 ± 0.003 for smRegion::dfMouse and 0.927 ± 0.005 dfRegion::dfMouse. Thus, regardless of the brain region specificity, overall predictability remained high when testing cells were sampled from mice different from the one used for GRU training, although the macro F1 score for predicting both Ca2+ and non-Ca2+ transients was statistically reduced when the testing cells were sampled in dfRegion::dfMouse (Fig. 20N,O).
Specifically, for predicting Ca2+ transients in different mice, the source of testing cells affected the F1 scores (Fig. 20J,L) by impacting both the Precision (Fig. 20B,D) and the Recall (Fig. 20F,H). Lower Precision, Recall, and F1 score were noted when testing cells were sampled from dfRegion::dfMouse, relative to either smSS::smDay or dfRegion::dfMouse.
For predicting non-Ca2+ transients, the source also affected the F1 scores (Fig. 20K,M), by impacting both the Precision (Fig. 19C,E) and the Recall (Fig. 19G,I). Lower, although still near perfect, Precision, Recall, and F1 score were noted when testing cells were sampled from dfRegion::dfMouse, relative to either smSS::smDay or dfRegion::dfMouse.
In conclusion, the GRU model can identify Ca2+ transients in cells from different mice with a macro F1 beyond 0.900. However, the risk of misclassifying non-Ca2+ transients as Ca2+ transients or Ca2+ transients as non-Ca2+ transients increases, though within a narrow range, when testing cells are sampled from a different brain region of a different mouse. There appears to be no significant impact on the F1 score when the testing cells are from the same region of a different mouse.
Discussion
Selection of machine learning model
In recent years, several studies using different machine learning methods have been developed utilizing some combination of the CNN, attention-based architectures for cell identification, and subsequent cell extraction (Denis et al., 2020; Bao et al., 2021; Sita et al., 2022). However, they require a significant amount of labeled data as ground truth to achieve adequate predictive performance. This raise concerns in both time investment to train a single model as well as, due to the black box nature of these models, their ability to maintain performance given a substantial change in the input data (such as changes to different experimental animals, another brain region of interest, a different type of neuron).
In this study, we evaluated four machine learning models (GRU, LSTM, Transformer, and Local Transformer) for Ca2+ transient detection. Both predictability performance (i.e., precision, recall, and F1 scores) and time efficiency should be considered. First, GRU model is the most time efficient, with a computation time of 287.1 s/epoch, slightly faster than LSTM (290.59 s/epoch) due to its simpler architecture and fewer parameters, making it computationally less expensive. GRU also exhibits strong overall performance in terms of precision, recall, and F1 scores, making it an excellent choice when considering both predictability and computation time. Second, LSTM takes marginally longer than GRU but performs slightly worse across multiple predictability metrics for Ca2+ transient detection. While very close to GRU in time efficiency, its more complex gating mechanism results in slightly longer computation times, without a significant performance advantage. Third, standard Transformer requires significantly more computation time at 708.8 s/epoch. Although transformers can be easily parallelized, their global self-attention mechanism and large parameter space demand much higher computational power. This increased time is not justified by any improvement in Ca2+ transient predictability, which actually suffers due to interruptions from distal frames when global attention is used in decision-making. Fourth, Local Transformer is the least time efficient, taking 1,744.4 s/epoch, almost 6× slower than GRU or LSTM, and 2.5× slower than the standard Transformer. This can be attributed to the increased complexity of managing localized attention windows and maintaining context between them. Additionally, it loses parallelization efficiency when processing the entire window. However, this increased computation time is compensated by improved predictability, as attention is focused more effectively on local frames, leading to better decision-making. In conclusion, RNNs outperform transformers in both time efficiency and predictability for Ca2+ transient detection. Between two RNN variants, GRU offers a better performance in both time efficiency and predictability, although LSTM remains a viable alternative. When the GRU performance is challenged by dataset size, computational resources, and the complexity of the transient dynamics, other models (e.g., LSTM) could be explored directly in the CalTrig.
Integrative visual exploration
The synchronization of Ca2+ imaging, cellular footprint, behavioral tracking, and Ca2+ transient trace windows, adjustable via a time bar, allows for a comprehensive and temporally aligned visualization of neuronal activity and behavior. This integration of multiple data streams provides several key benefits for neuroscience research. First, enhanced understanding of brain–behavior relationships: By synchronizing Ca2+ traces with behavioral data, researchers can directly observe how specific neuronal populations respond during behaviors or external stimuli. This real-time insight deepens our understanding of how brain activity drives behavior, such as the timing of Ca2+ transients relative to actions like lever presses. Second, technical flexibility and real-time adjustments: The adjustable time window enables researchers to zoom in on specific segments of data, closely examining how neural activity changes just before or after a behavioral event, which is critical for understanding causal relationships between stimuli and neural responses. Third, hypothesis proposing and testing: The integration of data streams facilitates hypothesis generation, which can be rapidly tested by determining the relationships between neural signals and behaviors. Researchers can observe whether certain Ca2+ transients precede or follow behavioral events, helping to refine hypotheses in real time. Fourth, visualization of micro-network dynamics: Viewing neural footprints while tracking behavior enables the identification of micro-networks at the neuronal level, distinct from traditional brain region-level network, during specific tasks or stimuli. This is critical for understanding neuronal network dynamics in healthy and diseased states, such as in learning, memory, or addiction studies. Fifth, categorization of neurons: Visualizing Ca2+ transients enable clustering neurons by activity patterns, revealing subpopulations and enhancing understanding of neural network heterogeneity in health and disease. Sixth, facilitating longitudinal studies: In long-term studies, where animals undergo repeated trials, integrative visualization tools allow researchers to track changes in both neuronal activity and behavior over time, aiding in the study of neuroplasticity and how neuronal responses evolve with experience. Seventh, improved data interpretation and collaboration: Simultaneous visualization of multiple data layers simplifies analysis, reveals patterns, and enhances cross-disciplinary communication. In conclusion, integrative visual exploration enables deeper exploration of brain function, supports hypothesis generation about neuronal mechanisms, and opens new avenues for understanding neurological disorders and developing treatments.
Featured value of CalTrig
The usability of the CalTrig tool is highlighted by its integrated visual experience, which provides users with a comprehensive view of Ca2+ images, neuron footprints, and Ca2+ transient traces in one unified interface. The tool features multilayered toolboxes that cater to various analytical needs, allowing users to work efficiently at both the cellular and Ca2+ transient levels. Manual identification of Ca2+ transients is made efficient with features such as keyboard shortcuts (ASDF keys), hide/show traces, and unified scales for transient identification. This allows users to manually identify transients in ∼1–2 min per 15 min Ca2+ trace. Furthermore, manual identification can be assisted by autoidentification processes based on kinetic parameters or pre-established machine learning models. CalTrig is a self-looped tool, designed for continuous functional improvement, ensuring it remains adaptable and upgradable.
In terms of accessibility, CalTrig is a GUI-based open-source tool, making it readily shareable and open for further updates. It supports all CNMF or CNMF-E-processed imaging data and operates with limited computing demands, allowing for efficient performance even on less powerful systems. The GUI of CalTrig is designed to be user-friendly, ensuring that non-programmers can easily interact with it through intuitive buttons, menus, and forms. The tool presents a professional appearance, which enhances its suitability for presentations, collaborations, and publications. Real-time feedback is provided during parameter adjustments in Ca2+ transient identification, allowing users to fine-tune the process interactively. CalTrig can be downloaded as an independent application that runs in a Python environment, meaning users do not need additional software installations. Additionally, the tool is highly customizable, allowing users to tailor it to specific workflows and improving productivity for different use cases. Its expandability ensures that it can accommodate future updates and feature integrations.
Our research demonstrates that the GRU model provides high predictability when applied to testing cells from the same or different session stages, across various training days, brain regions, and mice. The GRU model has proven to be efficient in training, as shown in Figure 15, even with a limited number of cells (Figs. 18–20), and the “ground truth” of Ca2+ transients can be established with relative ease, as detailed in the Methods and Results sections. This makes it highly feasible to extend testing to longer training sessions, such as the 6 h sessions commonly used in studies on learning, memory, motivation, and addiction, or across broader time spans covering days, weeks, or months. The tool we developed here will be instrumental in evaluating the feasibility of in vivo Ca2+ recordings during extended sessions over prolonged recording periods.
The high predictability of the GRU model in detecting Ca2+ transients also indicates that the basic properties of Ca2+ transients, such as rise slope, amplitude, and signal-to-noise ratio, remained stable during 1 h recording sessions, across five recording days, and across different brain regions in several mice. Notably, the neurons detected in two brain regions, M2 and mPFC, are most likely pyramidal neurons, which may explain the model's high level of expandability. A future direction for research could be to test the expandability of the model to different types of neurons, e.g., interneurons.
Limitations and future directions
There are several areas which the current study has not yet explored. For instance, the trained machine learning model has not been tested across different animal lines or species, such as from mice to rats. Additionally, while this study used GCaMP8f as the Ca2+ indicator, comparing its dynamics and fluorescent persistence with other genetically encoded Ca2+ indicators could provide valuable insights. Detecting Ca2+ transients is often just the starting point; challenges remain in selecting appropriate time windows for longitudinal analyses and linking behavior-associated transients to specific brain functions. Another intriguing area is the influence of neuronal spatial distribution on brain function, particularly under different physiological or pathological conditions. Our ongoing tool development aims to integrate these aspects into CalTrig, advancing the understanding of Ca2+ transients in brain output coding.
Classification of Ca2+ transients was performed using denoised and demixed calcium traces (“C” output) from the CNMF process implemented in Minian. Consequently, applying the provided machine learning models directly may limit their effectiveness to data processed using CNMF or similar algorithms. Performance could vary when utilizing calcium traces obtained from alternative signal extraction methods. However, if a new dataset obtained from non-CNMF processes, the user could still try to integrate it into the GUI. We recommend to look at each of the signals and see, which ones adhere most closely to the CNMF data structures. If there is no corresponding signal, then user may try to generate arrays of the same shape as the CNMF data. However, this could lead to some issues with a few of the features in the GUI. At the very least, the user should have the equivalent A, C, DFF, and a video file. If only a single video file is available, then the user could copy it for the other expected video files. We encourage users to open issues on the GitHub page.
Footnotes
The authors declare no competing financial interests.
This work was supported by National Institutes of Health grants (R01AG072897, R01AA025784, and R01DA059548). We thank Dr. Denise Cai and her team for developing Minian as a powerful tool used to extract Ca2+ traces.
This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International license, which permits unrestricted use, distribution and reproduction in any medium provided that the original work is properly attributed.
References
In this issue
