Heard or Understood? Neural Tracking of Language Features in a Comprehensible Story, an Incomprehensible Story and a Word List

Impulse response for the concatenation of the filters. The impulse is visualized in blue, and its filter response is visualized in orange.

Methodological approach to determine the added value of language features. The added value of language features was determined as the difference between two models, respectively, the complete and baseline model. We considered (pre)lexical features, denoting the onset of phonemes and words as control features, and thus, they are included in the baseline model. To determine the added value of language features, we considered all language features combined (i.e., phoneme surprisal, cohort entropy, word surprisal, and word frequency).

Behavioral results. A, The subjectively rated speech understanding as a function of the speech material, respectively, the Dutch story (orange), Frisian story (blue), and word list (yellow). B, The results of the word identification task for the Frisian story and the word list (*p < 0.05, **p < 0.01, ***p < 0.001).

Neural tracking. A, The prediction accuracies obtained with the acoustic model are shown for the three speech materials, respectively, the Dutch story (orange), Frisian story (blue), and word list (yellow). More specifically, the spatial distribution of the prediction accuracies obtained with only the acoustic features is visualized on the left side. The same prediction accuracies are visualized on the right side but averaged across a frontal channel selection (see head plot). B, The increase in prediction accuracies when including the (pre)lexical features on top of the acoustic features are shown for the three speech materials. This increase in prediction accuracies, averaged across a frontal channel selection (see head plot), are visualized on the right side. C, The increase in prediction accuracy when including the language features on top of a model with (pre)lexical and acoustic features for all three speech materials. The same prediction accuracies are visualized on the right side but averaged across a central channel selection (see head plot). The black annotated channels indicate the cluster, which drives the spatial distribution to be significantly higher than zero as obtained by the cluster-based permutation test.

Neural response to acoustic speech features. For each acoustic speech feature, the TRFs in a frontal channel selection (shown as a right inset on the top panel) are depicted (left). The significance of the TRF is indicated by the thick lines in the corresponding color and the horizontal lines below the TRF. The vertical gray blocks mark the time regions used for peak determination. For each of these peaks, the associated peak topographies are visualized (right) for, respectively, the Dutch story (orange), Frisian story (blue), and word list (yellow).

Neural response to (pre)lexical speech features. For each (pre)lexical speech feature, the TRFs in a frontal channel selection (shown as a right inset on the top panel) are depicted (left). The significance of the TRF is indicated by the thick lines in the corresponding color and the horizontal lines below the TRF. The vertical gray blocks mark the time regions used for peak determination. For each of these peaks, the associated peak topographies are visualized (right) for, respectively, the Dutch story (orange), Frisian story (blue), and word list (yellow).