Article Figures & Data
Figures
- Figure 1.
A, This panel illustrates the levels of abstraction with which acoustic features depict different tone sequences. Furthermore, it highlights the impact of the acoustic detail the acoustic onset, envelope, and mel-spectrogram capture. In the first tone sequence, onsets reveal the timing of sound occurrence, while the envelope extends this information by showing both the timing and duration of the sound; the mel-spectrogram further adds details about the frequency content. For the second tone sequence, the limitation of onsets becomes apparent, as they do not convey any information about amplitude changes. In the third sequence, the frequency changes are distinctly and exclusively captured by the mel-spectrogram, with the onset and envelope methods remaining indifferent to these frequency variations. The fourth sequence underscores a similar limitation of the envelope, which fails to distinguish frequency variations in a continuous tone. The lower section of this panel demonstrates the crucial role of meta-information. For instance, understanding which sound in an experiment requires a behavioral response cannot be discerned from the acoustic features alone. Meta-information thus complements the acoustic analysis by providing essential contextual and functional information. B, This panel showcases acoustic feature representations—onsets, envelope, and mel spectrum—for a complex naturalistic soundscape. Additionally, it highlights the diversity of meta-information that can describe these soundscapes. The upper row features continuous classification from Yamnet, providing labels for individual sound segments. The lower row presents manual annotations, categorizing sounds into different conditions (narrow and wide) and distinct types (beep, irrelevant, alarm, and speech sounds). These labels, informed by task-specific knowledge, offer deeper contextual and descriptive insights into the soundscape, illustrating the multi-layered nature of acoustic analysis in naturalistic settings.
- Figure 2.
This figure depicts the process of forward modeling. From the audio several features are extracted. Using the EEG data and the features the temporal response function is derived for each EEG channel. This temporal response function is then used to predict unseen neural data based on the corresponding feature information. The resulting prediction is correlated with the actual signal. If discrete features are used, then not every sample can be predicted. This is visualized in the pie charts, showing the proportion of samples that can be predicted. AC = acoustic features, SI = sound identity.
- Figure 3.
Exemplary model building of combining acoustic onset information with alarm tones and the respective CPs. A, Statistical comparison of prediction distributions based on AC = Acoustic, combined with either SI = sound identity, CP = cognitive prior information. *p < 0.05, **p < 0.01, ***p < 0.000, N.S. = non-significant B, Example of model weights for acoustic onsets alone, extended by SI, or CP information. A comparison of the condition effect to the results of Rosenkranz et al. (2023) can be found in the Extended Data Figure 3-1. The x-axis displays the time lags for the different weights. In the last plot, the nw = narrow and we = wide condition weights are displayed for the alarm tone.
- Figure 4.
A, A comparison of the prediction distributions of the acoustic features and their combinations. *p < 0.05, **p < 0.01, ***p < 0.000, N.S. = non-significant. A detailed comparison between the envelope and mTRF envelope can be found in Extended Data Figure 4-1. B, The result of the variance partitioning showing the unique contribution and shared explained data. The letters correspond to Table 1, which displays the values of the variance partitioning analysis.
- Figure 5.
A, Displayed are the distributions of prediction accuracies for a subset of different features and feature combinations. Each dot represents the average condition and channel correlation score of one participant. These distributions are the collapsed results over proportion of explainable data as presented in B, The results of the Wilcoxon sign rank test are presented exemplary at *p < 0.05, **p < 0.01***p < 0.000, N.S. = non-significant. B, prediction distributions share the same y-axis as that of part A. Additionally, we computed the proportion of data that is explainable using the different features. For the continuous features such as the envelope or mel-spectrogram, the variance was added for visual purposes.
- Figure 6.
The left panel shows the results of the simulation study. The x-axis depicts the different simulated SNR levels in dB. The y-axis shows the correlational values. The two graphs show the mean correlational values as a function of SNR levels for either the whole segment (blue) or only where data was predicted (red). The vertical lines show the estimated SNR level for the onset, beep, irrelevant, and alarm tone respectively. The panel on the right shows the prediction scores of the actual data.
Tables
- Table 1.
Shows the variance explained for the acoustic (AC) and sound identity (SI) features
r2 A ∩B A ∩C B ∩C A ∩B ∩C Beep (A) 0.0009 0.0007 0.0003 0.0002 0.0001 0.0003 SI Irr (B) 0.0028 0.0027 Alarm (C) 0.0041 0.0041 Mel (A) 0.0072 0.0031 0.0023 0.0039 0.0020 0.0020 AC Ons (B) 0.0023 0 mEnv (C) 0.0052 0.0012 Specifically, the first two columns display the total and unique variance explained by each feature. The last four columns highlight the shared variance ( ∩), where A refers to the first, B to the second, and C to the third feature, for SI and AC respectively (Fig. 4).
base AC SI onset mTRF envelope alarm irrelevant beep onset 1 0.76 0.72 0.81 0.49 mTRF envelope 0.79 1 0.70 0.69 0.38 Cross alarm 0.69 0.28 1 0.85 0.51 irrelevant 0.72 0.44 0.81 1 0.63 beep 0.48 0.18 0.63 0.72 1 The table should be read from left to top. The feature labels on the left are the models that were trained on these features. The labels at the top are feature information used to predict segments using the model on the left. A visualization of the results can be found in Extended Data Table 2-1 and 2-2. All correlations are significant at p < 0.01.
Extended Data 1
Download Extended Data 1, ZIP file.
Figure 3-1
Upper panel shows the TRF weights of the alarm tone for the two conditions. The grey shaded area marks the window of interest as reported by Rosenkranz et al. (2023). The black lines are the clusters detected by the permutation testing with the corresponding topographies. The lower panel shows the same but for the beep tone. Download Figure 3-1, TIF file.
Figure 4-1
The left panel shows the distribution of the prediction accuracy for the envelope and mTRF envelope model. Significance is tested at * p < 0.05, **p < 0.01, ***p < 0.000. The panel in the middle shows the model weights averaged over participants, conditions, and channels. The panel on the right highlights the frequency decomposition of the difference curve of the mTRF envelope and the envelope. Download Figure 4-1, TIF file.
Table 2-1
This figure shows the results of the cross-prediction analysis. On the x-axis are the correlational score of the testing data segment with the prediction based on feature information that the model was initially trained on. On the y-axis are the correlational scores for the same segment and feature information as on the x-axis, but using model weights derived from the depicted feature. Download Table 2-1, TIF file.
Table 2-2
This figure shows the results of the cross-prediction analysis for the sound identity marker. On the x-axis are the correlational scores of the testing data segment with the prediction based on feature information that the model was initially trained on. On the y-axis are the correlational scores for the same segment and feature information as on the x-axis, but using model weights derived from the depicted feature. Download Table 2-2, TIF file.
In this issue
