SaLSa: A Combinatory Approach of Semi-Automatic Labeling and Long Short-Term Memory to Classify Behavioral Syllables

Semi-automatic labeling. A, A frame with tracked body parts. Although 10 body parts were tracked, the mid-tail and tail tip were excluded for quantitative analyses. B, The categories of extracted features. C, A chunk of normalized feature values. Each feature was Z-scored for a visualization purpose. Dotted lines separate different feature categories. The feature category number corresponds to the number indicated in B. D, Cumulative distribution of explained variance across principal components (PCs). The threshold for including PCs was set at 85% variance explained. E, UMAP representation of the entire video sequences and example frame sequences of labeled behavioral syllables. Data reduced by principal component analysis (PCA) was further reduced to two dimensions by UMAP. A spectral clustering algorithm was applied. By removing fragmented (<0.25 s) sequences, each cluster was manually annotated. The example sequences indicate a down-sampled sequence of each labeled behavioral syllable. Please note that although the mid-tail and tail tip were shown in sample frames, because the tail shape did not reflect behavioral syllables, the mid-tail and tail tip were excluded from all quantitative analyses including the UMAP analysis.

LSTM training data and performance. A, The proportion of each behavioral syllable for LSTM training. B, Receiver operating characteristic curves for each behavioral syllable based on independently labeled 10 videos. FA, false alarm. C, The area under the curve (AUC) values across behavioral syllables. In Extended Data Figure 3-1, a range of parameters in the LSTM model were systematically assessed. The evaluation performance of a multiclass support vector machine is shown in Extended Data Figure 3-2.

Benchmark testing of SaLSa. A, Confusion matrix between manually labeled data (y-axis) and SaLSa outputs (x-axis). The values indicate what fraction of manually labeled data were classified as each syllable by SaLSa. B, Confusion matrix between manually labeled data and keypoint-MoSeq (kpms) outputs. Syllable 16 is a subthreshold (0.5% cutoff) class. C, Confusion matrix between outputs from SaLSa and kpms. The values indicate what fraction of frames classified by SaLSa were classified by kpms. D, Confusion matrix between manually labeled data and re-assigned kpms syllables. The original kpms syllables were re-assigned to one of six syllables based on comparison to manually labeled data (B). E, Confusion matrix between SaLSa output and the re-assigned kpms syllables.

Quantification of behavioral syllables. A, The median speed per episode of each video across behavioral syllables. B, The median turning angle per episode of each video across behavioral syllables. C, The median compactness per episode of each video across behavioral syllables. D, The fraction of each behavioral syllable. E, The average episode duration of each video across behavioral syllables. Inset, p-values of post hoc pair-wise comparisons (n = 43, two-way ANOVA with post hoc Tukey’s HSD test). L, locomotion; T, turning; R, rearing; G, grooming; P, pause.