Sensorimotor Representation of Speech Perception. Cross-Decoding of Place of Articulation Features during Selective Attention to Syllables in 7T fMRI

Participants

Fourteen native Dutch speakers (mean ± SD age, 27.2 ± 5.1 years; nine females; two left-handed) underwent blood oxygenation level-dependent (BOLD) signal fMRI scanning while performing two delayed-match-to-sample tasks. Participants had no history of neurologic or systemic diseases and reported normal hearing abilities. All participants received monetary compensation for their participation and signed written informed consent. The Ethical Committee of the Faculty of Psychology and Neuroscience at the University of Maastricht, Maastricht, The Netherlands approved the study.

Stimuli

Stimuli were 18 CV syllables recorded by three female native Dutch speakers, generating 54 unique tokens. The syllables were the result of the CV combinations of six consonants (p, t, f, s, m, n) and three vowels (a, i, u). This subset of consonants was selected because it allowed us to build a balanced stimuli matrix that covers two features of place of articulation (i.e., bilabial/labiodental and alveolar) and three features of manner of articulation (i.e., stop, fricative, and nasal). This arrangement was needed to perform MVPA-based cross-decoding analysis and to keep the necessary number of trials for single-trial classification (see below, Multivariate analysis). Syllable recordings were selected from a subset of recordings previously used in our laboratory (Correia et al., 2015). Briefly, the consonants composing the syllables for every articulatory feature were bilabial/labiodental (p, f, m) and alveolar (t, s, n) for place of articulation; and stop (p, t), fricative (f, s), and nasal (m, n) for manner of articulation (Fig. 1A). The three different vowels and three different speakers introduced acoustic variability useful for classification. All stimuli were recorded in a soundproof chamber at a sampling rate of 44.1 kHz and digital-to-analog converted with 16-bit resolution. Stimuli were presented in the MRI scanner via MR-compatible earphones with a linear frequency transfer of up to 8 kHz (model S14, Sensimetrics Corporation). Before starting the experiment, the volume was adapted to each subject’s audible and comfortable perceptual level.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Stimuli and experimental paradigm. A, Spoken stimuli matrix and articulatory properties. The 18 syllables were selected according to the place of articulation (i.e., bilabial/labiodental and alveolar) and manner of articulation (i.e., stop, fricative, and nasal). B, Experimental procedure and task description. Example of a typical match-to-sample trial during attend to vowels task. Subjects received instructions per block, in which they attended to consonants or vowels, respectively, and conducted a match-to-sample decision within slow-event related trials. Each block started with the visual presentation of a task cue (i.e., attention target vowel or consonant), indicating which task to perform in the next 27 trials. Each trial started with a fixation period in which a fixation cross was presented at the center of screen, together with a syllable sound (340 ms). After a jittered ISI (jitter: 6–8 s) a visual cue (i.e., a written letter, vowel, or consonant) was presented for 2 s, followed by the immediate subject’s response by pressing a button either with the right index finger (for match trials) or middle index finger (for mismatch trials). The response was followed by a jittered ISI (4–10 s) to complete the jittered ITI period (12–16 s) before the next trial started. C, Schematic representation of the fMRI acquisition sequence and its relationship to the syllable sounds presented to the participants. Vow, attentd to vowels task; Con, attend to consonants task; Syll, syllable; TA, time of acquisition; TR, time for repetition; Phase, opposite phase encoding volumes acquired for distortion correction.

Experimental design

Participants performed two delayed-match-to-sample tasks in four runs of 54 trials each (i.e., each syllable token was presented once per run) during fMRI acquisition. In cognitive terms, this task allowed paying attention to a certain aspect of the acoustic stimuli, from now on referred to as “attend to vowels” and “attend to consonants.” The task also involved remembering and matching the attended vowel or consonant later on with a visually presented token (i.e., a written vowel or consonant) of the same category (with 50% match/mismatch response proportions). Each run was divided into two blocks of 27 trials, one for each task, and the tasks where counterbalanced across runs and participants. During the attend to vowels task, participants heard a syllable and received, after 6–8 s, a written vowel as “match cue.” Participants were instructed to match the vowel of an auditorily presented CV syllable with a written vowel cue as fast and accurately as possible by pressing a button with the right index (“match”) or middle (“mismatch”) finger. During the attend to consonants task, participants heard the same syllables (of the attend to vowels task) and received a written consonant as cue that should be matched with the auditory syllable (Fig. 1B). Importantly, the features of interest (i.e., place of articulation) were never part of an explicit task to avoid confounds in attentional demands. The delay-match-to-sample task is widely used to precisely control the subject’s attention. Thus, data were analyzed only for the first part of each trial before the cue onset to avoid that relevant signal was contaminated with matching and response related processing period signals.

The task was clearly explained to each subject outside the scanner, and they received at the beginning of each block an introduction display indicating the syllable component they should attend to (i.e., vowels or consonants) at the beginning of each task. A trial consisted of (1) a speech stimulus sound presentation (i.e., a CV syllable token of 340-ms duration) followed, after an interstimulus interval (ISI) of 6-8 s; by (2) a written cue (i.e., a consonant or vowel letter matching or not the sound token depending on the specific block and trial of 2 s duration, written in Times New Roman, font size 30, black color); and then followed by (3) the subject’s match/mismatch immediate response. The sound syllables and the visual cue (i.e., a written vowel or consonant) matched in 50% of the trials, and match/mismatch trials were balanced across attention conditions and randomized per subject. Mismatch cues were always of the same category (i.e., vowels in the attend to vowels task and consonants in the attend to consonants task). In total, each run lasted 15 min and the behavioral responses were collected through an MR-compatible button box (Current Designs, eight-button response device, HHSC-2x4-C).

We used fast sparse image acquisition to have a 500-ms silent gap to present each syllable sound (Di Salle et al., 2003). The fMRI acquisition was set up as a slow event-related design. The intertrial interval (ITI) between consecutive auditory stimuli was relatively long (i.e., 14 s on average; range: 12–16 s) to allow independent BOLD signal estimation per trial (Fig. 1C). The interval used to estimate the fMRI response to the spoken syllable perception per trial was restricted to the first 6 s following the sound presentation to avoid contamination from the processing of the visual cue, matching, or button press. Moreover, as the acoustic, phonetic, and phonological features of the presented syllables were identical across trials and attention conditions, the effects on the cortical representations pertain to differences in attention. All stimuli, event identities, and timings were presented and logged using Presentation from Neurobehavioral Systems (www.neurobs.com; RRID: SCR_002521).

Functional MRI acquisition

Functional and anatomic volumes were acquired on a whole-body Siemens Magnetom 7 Tesla scanner (Siemens) and a 32-channel head-coil (Nova Medical Inc.) at the Maastricht University Medical Center Imaging Center. For all functional runs, we acquired whole brain high-resolution accelerated multiband gradient echo (T2*-weighted) echo-planar imaging (EPI; Moeller et al., 2010; Setsompop et al., 2012) data [echo time (TE) = 21 ms; repetition time (TR) = 2000 ms; time of acquisition (TA) = 1500; delay in TR (silent gap) = 500 ms; multi-band factor = 3; generalized auto-calibrating partially parallel acquisitions (GRAPPAs) g-factor = 2; flip angle = 72°; field of view, FOV = 198 mm; voxel size = 1.5 × 1.5 × 1.5 mm³; number of slices = 72, without gap between slices] for each participant. To correct for EPI geometric distortions, 10 volumes with opposite phase encoding directions (i.e., posterior to anterior and anterior to posterior) were additionally acquired using the same acquisition parameters as in the functional runs. After the first 2 functional runs, we acquired a tridimensional T1-weighted magnetization prepared rapid acquisition gradient echo (3D-MP2RAGE; Marques et al., 2010) volume (240 sagittal slices; voxel size = 0.65 × 0.65 × 0.65 mm³; first inversion time TI1 = 900 ms; second inversion time TI2 = 2750 ms; TE = 2.51 ms; TR = 5000 ms; first nominal flip angle = 5°; second nominal flip angle = 3°) per participant.

Functional MRI preprocessing

Anatomic and functional data were analyzed using BrainVoyager QX (version 2.8.4; Brain Innovation; RRID: SCR_013057), and custom code written in MATLAB (R2014a version 8.3.0.532; The Mathworks Inc.; RRID: SCR_001622). Anatomic images were interpolated to a nominal voxel size of 1.5 × 1.5 × 1.5 mm³ matching the functional images’ resolution. The functional images were corrected for motion artifacts using the 3D rigid body motion correction algorithm implemented in BrainVoyager QX and all functional runs were aligned to the first volume of the second functional run. We corrected the EPI distortions using the topup tool implemented in FSL (RRID: SCR_002823; Smith et al., 2004). The reversed phase encoding images, acquired after the anatomic images, were used to estimate the susceptibility-induced off-resonance field and, then, to correct the distortions in the remaining functional runs. After this correction, functional data were high-pass filtered using a general linear model (GLM) Fourier basis set of eleven cycles sine/cosine, including a linear trend removal. Functional volumes per run were co-registered and aligned to the anatomic scan using rigid body transformations (i.e., six parameters: three translations and three rotations). Finally, functional images were normalized by transformation into Talairach space (Jean and Tournoux, 1988).

Univariate analysis

Single subject GLM analysis was performed on fMRI signal time courses normalized with percentage transform in volume space. Next, they were mapped onto surface space by sampling the values located between 1 mm below the gray/white matter boundary and up to 3 mm into the gray matter toward the pial surface using trilinear interpolation and averaging. This sampling resulted in a single value per vertex in the subject’s cortex mesh, and then the values were aligned to the cortical group surface mesh using CBA. Random-effects GLM analysis was performed on these individual time course data. The conditions were collapsed across the speaker and vowel dimensions, thus yielding 12 predictors, six predictors for each consonant × two tasks. Each predictor was convolved with a canonical double gamma hemodynamic response function (HRF). Functional maps (i.e., average β values) were calculated to assess sound-evoked fMRI responses during attend to vowels and attend to consonants tasks (i.e., all sounds attend to vowels task > baseline; all sounds attend to consonants task > baseline). Task differences were analyzed for attend to consonants and attend to vowels task activity (i.e., attend to consonants task > attend to vowels task). Univariate stimulus effects were analyzed for each place of articulation feature independently (e.g., p_Con + f_Con + m_Con + p_Vow + f_Vow + m_Vow > t_Con + s_Con + n_Con + t_Vow + s_Vow + n_Vow). All functional contrast maps were corrected for multiple comparisons by applying a permutation-based cluster-size threshold with an initial cluster forming threshold of p = 0.05. The cluster-size threshold was based on the distribution of maximum cluster sizes obtained in the 2000 permutations, only considering clusters whose size was larger than the 95% quantile.

Multivariate analysis

To investigate the cortical representations of the different articulatory features, we used MVPA in combination with a moving volumetric searchlight approach (Kriegeskorte et al., 2006). The purpose of the multivariate analysis was to decode articulatory features for each syllable independent from their specific phonetic signatures or acoustical properties. Therefore, we used a classification approach based on cross-decoding, generalizing place of articulation features across different dimensions of manner of articulation. Specifically, we trained a classifier to discriminate bilabial/labiodental versus alveolar places of articulation using syllables exhibiting two manners of articulation dimensions (e.g., stop and fricative) and tested whether this training was transferable to syllables exhibiting a third (unseen) manner dimension (e.g., nasal). MVPA generalization analysis was performed within each subject using all three combinations of generalization (i.e., leaving one manner of articulation out for testing at each split). The obtained averaged accuracies were submitted to a group analysis using random permutations for significance and nonparametric permutation-based cluster-size thresholding for multiple comparisons correction (see below, Multivariate statistical analysis, group statistics).

Within-subject decoding

Before classification, BOLD time courses were selected by extracting the responses in each trial and each cortical voxel. The BOLD responses were estimated by fitting a standard HRF using multiple linear regression, taking the first three samples per trial after sound onset (i.e., corresponding to 6000 ms). The regression coefficients resulting from the fitting of single trial data in the cortical mask were used to build an fMRI feature space (i.e., number of trials by number of cortical voxels), which was then used in the multivariate decoding. During the multivariate decoding, we kept trials belonging to each task separate.

We limited our analysis to gray matter voxels using a subject-specific cortical ribbon mask based on intensity values in the T1-weighted images (i.e., cortical ribbon segmentation). We constrained our analysis to the cortical ribbon, as we were mainly interested in cortical processing. Ribbons allowed us to exclude volumes containing white matter and subcortical voxels from the analysis, which increased the number of dimensions. For the searchlight analysis, a sphere with radius of 5.5 voxels (i.e., 8.25 mm) was moved through the cortical ribbon, which defined a feature space of 65,000 voxels on average. In each searchlight, we performed generalization of place of articulation across manner of articulation using linear discriminant analysis (LDA) with diagonal covariance matrix. Given the large number of searchlights, we employed a fast Matlab implementation described in Ontivero-Ortega et al. (2017).

We also investigated the cortical representation of place of articulation independent of the task effect. We did so by performing classification and generalization across manner of articulation in the pooled data from the two tasks (i.e., attend to vowels and attend to consonants). By doing this, we use twice the amount of data as in the previous analyses and can expect more robust and reliable findings due to the power increase in single subject analyses. To avoid any task bias, we controlled for the number of trials belonging to each task in each cross-validation split maintaining a balanced contribution of each task during the training and testing procedures. The decoding was performed as described above.

Multivariate statistical analysis, group statistics

In each subject, classification accuracies were obtained, separately for each task (i.e., attention to consonant and attention to vowel), for each searchlight centered on voxels belonging to the subject’s specific cortical ribbon. To perform a group analysis, these accuracies were projected onto each subject’s reconstructed cortical surface and then mapped back onto the group average cortical sheet using CBA. After the realignment, we considered in the group analysis only the vertices common to all subjects. To assess uncorrected significance of the classifications at the group level, we considered a non-parametric permutation test at each searchlight location independently. For each vertex, we took all the subject accuracies and the associated mean, and, using a resampling approach, we estimated the probability that such mean or a higher value could occur if the data were obtained under the null hypothesis (H0) that the decoding population mean is at chance level (i.e., 50%). This probability is the p value associated with the observed group mean. The resampling strategy was based on the fact that, under H0, the likelihood of the observed accuracies is symmetric around chance (i.e., 55% and 65% are equally likely if the population mean is 50%), and we can, therefore, build many datasets by switching some of the subjects’ accuracies around chance (Good, 2005, their section 3.2.1). We ran 2000 random permutations (Monte Carlo permutations) and for each searchlight we determined the p value as the ratio between how many times the mean of a population resample equaled or exceeded the observed mean and the total number of permutations (adding 1 to both numerator and denominator to avoid 0 p values, hence the lowest p value was 1/2001).

Finally, to correct for multiple comparisons, we employed a permutation-based cluster-size thresholding. We considered an initial Cluster Forming Threshold equal to 0.05 and, then, for each permutation, we tagged as significant those vertices whose observed mean was higher than the 1- quantile (obtained with the resampling procedure described above). For each of these “false positive” maps, we determined the maximum extent of clusters (on the surface) and built a distribution of cluster sizes across the 2000 permutations. Clusters of significant vertices, or equal to 0.05, in the observed data that were larger than the 95% quantile of the obtained distribution were deemed significant and corrected for multiple comparisons. It is worth mentioning that this procedure is based on a permutation strategy that does not suffer from inflation of false positive rate, as recently shown in Eklund et al. (2016).