Article Figures & Data
Figures
- Figure 1.
The validation and establishment of the IST and DST with objects and odors. A, A picture of an example object set-up is shown. Objects are displayed in six positions in a white-corrugated plastic box. B, A picture of an example odor set-up is shown. Odors are displayed in six positions in a white-corrugated plastic box. C, An example of an object stimuli. D, Example of an odor stimuli. E, Object interaction was measured using DRs to evaluate novelty preference using 3-objects and 6-objects. Male rats explore the novel object significantly more than the familiar objects in the IST and DST with both 3-objects and 6-objects. No differences in novelty preference or exploration times are seen between the IST and DST, or between 3-object and 6-object versions. F, Odor interaction was also measured using DRs to evaluate novelty preference using 3-odors and 6-odors. Male rats explore the novel odor significantly more than the familiar odors in the IST and DST with both 3-odor and 6-odor. No differences in novelty preference or exploration times are seen between the IST and DST, or between the 3-odor and 6-odor versions. Data are represented as mean ± SEM.
- Figure 2.
Experimental overview for acute Cannabis exposure and behavioral classifier training. A, Schematic representation of the experimental design. Male Long–Evans rats (n = 48) were used for this study. Using a repeated measures experimental design, each rat was exposed to high-THC Cannabis smoke, low-THC Cannabis smoke, and an Air Control condition. Male rats were exposed 20 min before the start of behavioral testing. Each male rat either underwent the 6-object IST and 6-object DST, or the 6-odor IST and 6-odor DST. The order in which the IST and DST were performed was randomized. Rat behavior was quantified using traditional stopwatch scoring and by automated supervised machine learning-based behavioral analysis. Suboptimal supervised machine learning predictions were replaced by stopwatch scoring, constituting a hybrid scoring approach. B, Illustration of the point-of-interest configuration used for pose-estimation analysis. We chose the number and position of points in accordance with the SimBA eight-point configuration. SimBA requires a standardized and specific position (and number) of points. Users should decide what SimBA configuration will be used (single animal, multianimal, point number) before network training with DeepLabCut. C, Visualization of the relative feature importance of the four features clusters. In short, the 40 most important features were systematically categorized into distinct clusters, then we summed the feature importance’s of individual features within each cluster. The raw features importance log is included under “assessment + logs” for each classifier within our GitHub repository. D, Classifier performance metrics for the object (top) and odor (bottom) models. Test frames were randomly extracted from the dataset (20% test, 80% train). E, Classifier performance metrics for the object (top) and odor (bottom) models. Test bouts were randomly extracted from the dataset (20% test, 80% train). See Extended Data Figures 2-1, 2-2, 2-3, and 2-4 for more information regarding the supervised machine learning approach and validation. This figure was created using BioRender.
- Figure 3.
Comparison between human stopwatch and supervised machine-learning generated output. A, Correlation matrix between methods of quantifying rat-object interaction. This comparison was made between supervised machine-learning (SML), human-stopwatch (HS), and region-of-interest (ROI)-generated interaction times. Interaction times by object was quantified using each scoring method, then the correlation between interaction DRs was assessed. B, Correlation matrix between methods of quantifying rat-odor interaction. Interaction times by odor was quantified using each scoring method, then the correlation between interaction DRs was assessed. C, Criteria used to rank automated classification. Each video was manually viewed for accurate classification, where a verification rank was assigned based on objective criteria. D, Frequency of verification rank assignment by type of stimuli. Videos with a verification rank less than three were excluded from final analysis and replaced by human stopwatch scoring. Approximately 80% of object videos and 60% of odor videos met inclusion criteria, respectively. E, Correlation between human stopwatch and SML-generated DRs on object videos meeting inclusion criteria, indicating a moderate-to-high correlation (r(109) = 0.83, p < 0.0001). F, Correlation between human stopwatch and SML-generated DRs on odor videos meeting inclusion criteria, indicating a moderate-to-high correlation (r(77) = 0.87, p < 0.0001). See Extended Data Figures 3-1 and 3-2 for additional information regarding the scoring and the ranking of videos by Cannabis treatment.
- Figure 4.
High-THC Cannabis smoke exposure impacts novelty preference under high-memory (DST) loads using object stimuli, with no impact on distance traveled, frequency of item visitation, or approach latencies. A, An example IST with objects is visualized, showing six identical objects in the sample phase, with a novel object introduced after a 1-min delay in the test phase. B, A DST with objects variation is shown, with an identical test progression, but instead starts with six different objects in the sample phase. C, Interaction measured as time spent with an object was generated using the human-machine hybrid scoring approach and visualized using a discrimination ratio for both variations using object stimuli. No difference in treatment groups is seen in the 6-object IST (n = 64). In the 6-object DST (n = 66), a significant decrease in novelty preference is seen in the SW group in contrast to the AC group (p = 0.04). D, The mean novel approach latency in the 6-object IST (n = 72) and 6-object DST (n = 69) variations is shown to be consistent between treatment groups. E, To illustrate the frequency of visitations to the novel object in comparison to the familiar objects, bout counts are visualized using a discrimination ratio. A preference for novel visitations is seen in the 6-object IST (n = 65) AC and SW groups, with no difference in item visitations in the 6-object DST (n = 66). F, The distance traveled (cm) in the 6-object IST (n = 72) and 6-object DST (n = 69) variations is comparable across treatment groups. Data represents mean ± SEM, *p < 0.05. Abbreviations: high-THC Cannabis smoke (SW); high-CBD Cannabis smoke (TI); Air Control (AC). This figure was created using BioRender.
- Figure 5.
High-THC Cannabis smoke exposure impacts novelty preference under high-memory (DST) and low-memory (IST) loads using odor stimuli, with no impact on distance traveled, frequency of item visitation, or approach latencies. A, Example IST with odors is visualized, showing six identical items in the sample phase, with a novel odor introduced after a 1-min delay in the test phase. B, A DST with odors variation is shown, with an identical task progression, but instead starts with six different odors in the sample phase. C Interaction measured as time spent with an odor was generated using the human-machine hybrid scoring approach and visualized using a discrimination ratio for both variations using odor stimuli. In the 6-odor IST (n = 75), a significant decrease in novelty preference is seen in the AC group in comparison to the SW group (p = 0.046). Whereas in the 6-odor DST (n = 73), a significant decrease in novelty preference is seen in the SW group from both the AC (p = 0.023) and TI (p = 0.046) groups. D, The mean novel approach latency in the 6-odor IST (n = 79) and 6-odor DST (n = 73) variations is shown to be consistent between treatment groups. E, To illustrate the frequency of visitations to the novel odor in comparison to the familiar odors, bout counts are visualized using a discrimination ratio. No differences between treatment groups or 6-odor IST (n = 79) and 6-odor DST (n = 73) are seen. F, Distance traveled (cm) in the 6-odor IST (n = 79) and 6-odor DST (n = 73) variations is comparable across treatment groups. Data represents mean ± SEM, *p < 0.05. Abbreviations: high-THC Cannabis smoke (SW); high-CBD Cannabis smoke (TI); Air Control (AC). This figure was created using BioRender.
- Figure 6.
Boli count following smoke exposure treatment. A significant increase in the number of boli recorded was observed following Cannabis smoke exposure in comparison to the Air Control (AC) condition. However, no difference between Skywalker (SW) or Treasure Island (TI) groups was recorded. ****p < 0.001. high-THC Cannabis smoke (SW); high-CBD Cannabis smoke (TI); Air Control (AC).
Tables
- Table 1
Summary of all interaction times for validation of the tests summarized in Figure 1
Object IST Object DST Odor IST Odor DST Sample* Test* Sample* Test* Sample Test Sample Test 3 Item 71.45 ± 12.1 47.98 ± 6.5 68.43 ± 13.4 104.43 ± 18.9 31.99# ±7.3 58.23 ± 5.3 35.69 ± 8.4 54.74 ± 5.9 6 Item 63.50 ± 5.4 34.30 ± 4.1 47.06 ± 5. 50.39 ± 6.9 38.14# ±7.6 38.59 ± 5.2 33.83 ± 6.3 38.20 ± 3.6 The mean (±SEM) for the total interaction time seen with stimuli is recorded for each sample and test phase in the IST and DST with objects or odors. * Significant main effect of Phase on object IST and DST (p < 0.05). # Significant effect of Item Count on exploration times in the sample phase of the odor IST (p = 0.047).
- Table 2
Summary of all interaction times for tests with Cannabis summarized in Figures 2-5
Object IST Object DST Odor IST Odor DST Sample* Test* Sample# Test# Sample& Test& Sample% Test% Air Control 36.21 ± 2.9 42.93 ± 4.0 35.61 ± 3.2 39.23 ± 3.4 37.75 ± 2.8 47.78 ± 5.8 39.16 ± 3.1 50.12 ± 5.6 High-THC 36.01 ± 3.7 46.90 ± 4.1 39.65 ± 3.5 49.72 ± 4.6 34.27 ± 3.1 57.94 ± 4.8 35.29 ± 2.8 55.27 ± 6.5 High-CBD 30.09 ± 3.0 33.97 ± 2.7 33.9 ± 3.1 46.96 ± 4.2 31.54 ± 2 36.93 ± 5.5 40.54 ± 3.4 48.03 ± 6.1 The mean (±SEM) for the total interaction time seen with stimuli is recorded for the sample and test phases in the different 6-object and 6-odor IST and DST across the Air Control, high-THC, and high-CBD treatment groups. * Significant effect of Treatment (p = 0.019) and of Phase (p = 0.012) on object IST. # Significant effect of Phase (p = 0.0058) on object DST. & Significant effect of Treatment (p = 0.025) and Phase (p = 0.0004) on odor IST. % Significant effect of Phase (p = 0.0019) on odor DST.
- Table 3
Summary of the effect sizes (Cohen’s d) and corresponding p-values for Figures 4C and 5C
AC-SW
Cohen’s dAC-SW
p valueAC-TI
Cohen’s dAC-TI
p value6-Object IST −0.25 [95.0%CI −0.856, 0.357] 0.409 0.291 [95.0%CI −0.323, 0.872] 0.319 6-Object DST −0.655 [95.0%CI −1.27, −0.035] 0.03* 0.118 [95.0%CI −0.507, 0.716] 0.7 6-Odor IST −0.783 [95.0%CI −1.41, −0.194] 0.0058** 0.0239 [95.0%CI −0.539, 0.637] 0.936 6-Odor DST −0.874 [95.0%CI −1.47, −0.228] 0.0042** −0.172 [95.0%CI −0.727, 0.413] 0.544 The unpaired Cohen’s d [confidence interval lower bound, upper bound] for interaction times seen between novel and familiar stimuli is recorded for the test phases in the 6-object and 6-odor IST and DST across the Air Control, high-THC, and high-CBD treatment groups. *p < 0.05, **p < 0.01, ***p < 0.001.
Extended Data Figure 2-1
Mean tracking confidence for each point-of-interest, by video. To calculate the mean tracking confidence for each video, the average of the likelihood column associated with each point of interest was calculated. Download Figure 2-1, TIF file.
Extended Data Figure 2-2
Model hyperparameters used for classifier training. A metadata csv file is included under “assessment + logs” for each classifier within our GitHub repository. Previous studies have shown that creating a balanced dataset by using the model hyperparameters of “random under sampling” or “random over sampling” lead to better classifier performance; however, we found that using these features dramatically decreased classifier performance and lead to equal classifier predictions across the data frame. Therefore, we chose to not use these hyperparameters for analysis, and accounted for the unbalanced dataset by setting a relatively low discrimination threshold. For both classifiers, a discrimination threshold of 0.35 and a minimum bout duration of 50 ms was used (Extended Data Fig. 2-3). Download Figure 2-2, TIF file.
Extended Data Figure 2-3
Representative plot of classifier predictions across a complete video (9000 frames, 5-min video). We chose a discrimination threshold of 0.35 as it corresponds to the middle segment of obvious probability spikes and excludes the majority of noise below 0.2. We assessed model performance in two ways, both of which are integrated in the SimBA GUI (Extended Data Fig. 2-2). First, we generated performance metrics (precision, recall, F1) by randomly splitting the aggregate training set (all human-annotated frames from all videos within the project) into 80% training frames and 20% test frames. Said differently, for a given behavioral video, a fraction of interaction-containing frames was used for model training, then a smaller fraction of frames was used for testing whether the model can accurately predict whether rat-stimulus interaction occurs in each test frame. As shown below, we found that both the object and odor classifiers generated excellent performance metrics when assessed in this manner. However, a fundamental problem with this assessment method is that for a given interaction bout, there may be both test and training frames, so the model is predicting interaction between two known sub-bouts of interaction (visualized- 1 = known interaction, test = test frame that the model must make a prediction on: 1-1-1-1-1-test-1-1-1-1). Therefore, to assess performance without the confound of intrabout test frames, we segregated the aggregate training into interaction bouts, then split the segregated training set into 80% training bouts and 20% test bouts. We found that the performance of the object classifier changed marginally with this change, but performance metrics for the odor classifier significantly decreased when assessed in this manner. While we content that assessing classifier performance by-bout is a more conservative and representative method, an important caveat is that classifier performance on a completely model-naive video is not assessed by either of these methods. This is important to consider because researchers will typically implement this analysis method to automatically quantify behavior for a large dataset, where only a fraction of this dataset is used for training. We did not include a by-video classifier analysis as this is not integrated into SimBA, but we contend that future research and software development should implement this performance assessment method to capture the accuracy of classifier predictions most accurately on model naive behavioral videos. Download Figure 2-3, TIF file.
Extended Data Figure 2-4
Precision recall curve visualizing changes in precision, recall, and F1 with classifier training. Raw data is included under “assessment + logs” for each classifier within our GitHub repository. Recall, precision, and by extension the F1 score are calculated from the entries of a confusion matrix. A confusion matrix tells us, given a set of observations belonging to at least two different classes and a classifier that attempts to label each, how many and what type of errors were made. The diagonal of the confusion matrix is the correct observations, the off diagonal are the errors. For a binary classifier, we are generally focused on one class over the other, thus the metrics we derive are chosen to represent how we did for the most important class. In our case “interaction” is the class we care about. In quantifying how our classifier for “interaction” did, we calculate the recall and precision. Recall is the proportion of all the possible “interaction” observations that our classifier predicted correctly. That is, the number of true positives (TP) divided by the total number of “interaction” observations (note the maximum number of true positives is all the “interaction” observations, in which case the recall equals 1, so a classifier that always predicts interaction will have perfect recall). Now there are many other metrics that could be computed, but the next most natural is the precision. Precision is the proportion of predicted “interaction” observations that were actual “interactions.” Or mathematically, the number of true positives divided by the total number of times our classifier predicted “interaction” (note it is not so easy to get perfect precision). Now we have two perfectly good numbers that quantify how our classifier did, the proportion of overall “interactions” that were recovered (recall) and the proportion of times our classifier predicted “interaction” and was correct (precision). It is not clear which is more important, so we combined the two as the F1 score as the harmonic mean of recall and precision. Why harmonic mean? We want an average of some kind, and the harmonic mean is the smallest of the 3 Pythagorean means (arithmetic mean, geometric mean, and harmonic mean). So, to have a high F1 score you must have high precision and recall, either one will drag the F1 score down nonlinearly. Download Figure 2-4, TIF file.
Extended Data Figure 3-1
Inter-rater variability analysis between human scorers of varying experience levels. In short, 20 behavioral videos (counterbalanced for IST/DST and objects/odors) were scored for rat-stimulus interaction by three independent scorers of differing experience levels (master, experienced, beginner). We found a strong correlation between scorers of all experience levels, but a comparatively weaker correlation between experienced and beginner scorers. Download Figure 3-1, TIF file.
Extended Data Figure 3-2
Proportion of excluded videos from verification ranks 4 and 5 as described in Figure 3C,D. The proportion of videos excluded did not differ significantly when grouped by treatment (A) or stimuli type (B). Download Figure 3-2, TIF file.
In this issue
