Article Figures & Data
Figures
- Figure 1.
A, Stimulus material: Participants observed videos of lip movements played either forward or backward. Lip movements (opening) and the corresponding unheard speech envelope were extracted as a continuous signal. B, Analysis of the hearing group: Results of coherence calculation between the speech envelope and lip movements using the selected ICA eye component. C, Coherence calculation at the strongest voxel in the primary auditory and visual cortex. Significant frequency clusters are marked in gray (N = 49; p < 0.01). See Extended Data Figures 1-2 and 1-3 for the same figure for the deaf groups and Extended Data Figure 1-1 for a sanity check of the source reconstruction. Extended Data Figures 1-4 and 1-5 provide supplemental information about the ICA eye component.
- Figure 2.
Role of eye movements in the cortical effects in the hearing group. A, Significant clusters of the forward versus backward comparison of the speech–brain coherence. B, Significant clusters of the forward versus backward comparison of the speech–brain coherence remain the same after controlling for eye movements. C, Coherence <1 Hz in the A1 and the V1 for coherence and partial coherence. The cluster permutation threshold was a p-value of 0.01. Error bands reflect the standard error (N = 49). The asterisks indicate levels of significance: **p < 0.01. See Extended Data Figure 2-1 for topographies on sensor and source level.
- Figure 3.
A, Ocular tracking: <1 Hz forward–backward tracking of the unheard speech envelope for the three hearing groups. The hearing and the DHH group (with audiovisual listening experience) show the ocular unheard speech tracking effect, which is not the case for the congenitally deaf group (see stars above the jitter plot). Furthermore, the DHH group shows increased tracking in the forward compared with the backward condition in general (see lines in black) and increased lip tracking in the forward condition (see lines in green) compared with the other groups. B, i, The main effect across all subjects showed higher forward versus backward differences in speech than in lip tracking. ii, A1: <1 Hz tracking in the auditory cortex. The hearing and the DHH groups show the unheard speech tracking effect. iii, V1: <1 Hz tracking in the visual cortex. The congenitally deaf group has higher forward versus backward tracking than the hearing group. The same voxels were selected as in Figures 1 and 2. For the whole-brain ANOVA, the cluster permutation threshold was a p-value of 0.01. Error bands reflect the standard error. The asterisks indicate levels of significance: *p < 0.05, **p < 0.01.
Extended Data
Table 1-1
Group information. Download Table 1-1, DOCX file.
Figure 1-1
Sanity check for the source reconstruction: Voxel with the strongest lip coherence. Download Figure 1-1, TIF file.
Figure 1-2
Effects for the acquired DHH group A) results of Coherence calculation of speech envelope and the lip movements with the selected ICA eye component. B) Coherence calculation at the strongest voxel in iii) in the primary auditory and visual cortex. Significant clusters are marked in gray (N = 19). Download Figure 1-2, TIF file.
Figure 1-3
Effects for the acquired DHH group A) results of Coherence calculation of speech envelope and the lip movements with the selected ICA eye component. B) Coherence calculation at the strongest voxel in iii) in the primary auditory and visual cortex. Significant clusters are marked in gray (N = 7). Download Figure 1-3, TIF file.
Figure 1-4
Additional information about the Eye-movements between conditions. A) The Blink frequency does not differ between the forward and the backward condition. (T(144) = -0.23, p = 0.82). The blinking frequency was 0.19 Hz in the Forward condition and 0.20 Hz in the Backward condition. B) The z-scored envelope value is lower in the forward condition compared to the backward condition (T(144) = -1.89, p < 0.05.) and overall, the envelope values are below zero (average) of the z-scored envelope (T(145)= -3.43, p < 0.001). C) The power of the ICA-Eye component does not differ significantly between the conditions, but the power in the backward condition is slightly increased in low frequencies. Download Figure 1-4, TIF file.
Figure 1-5
Exemplary 10 s time course of the participant with the highest ocular tracking. In the forward condition, the blinks appear to happen mostly, when the envelope is rather low. This is in line with Supp.6 B) showing lower speech envelope values while blinking in the forward condition compared to the backward condition. This participant has a blink frequency of 0.23 Hz in the forward condition and 0.25 Hz in the backward condition. Download Figure 1-5, TIF file.
Figure 2-1
A) MEG sensor plot topographies of the Sensor- Speech and Sensor-Lip coherence in the forward condition, the backward condition. The difference is the backward condition subtracted from the forward condition. B) Forward solution of the topography in A). 20% of the highest voxels in coherence are presented. Download Figure 2-1, TIF file.
In this issue
