The time-course and spatial distribution of brain activity associated with sentence processing
Highlights
► We contrasted sentences with word lists in a reading task using MEG. ► Increased activity observed in a range of temporal and frontal language regions. ► Time-locked aTL activity supports a link with compositional processing. ► Increases observed within 250–500 ms, supporting incrementalist comprehension models
Introduction
A broad neuroimaging literature has sought to identify neural activity associated with the cognitive operations engaged during the processing of sentences. A primary experimental design that has been used in this literature compares brain activity elicited when subjects read or listen to sentences to that observed for lists of words (Friederici et al., 2000, Humphries et al., 2005, Humphries et al., 2006, Jobard et al., 2007, Mazoyer et al., 1993, Rogalsky and Hickok, 2009, Snijders et al., 2009, Stowe et al., 1998, Vandenberghe et al., 2002, Xu et al., 2005). This contrast has the desirable property that both conditions are likely to require an equivalent degree of word-level processing, but only the sentences are assumed to require processing associated with building and comprehending sentence structure. In this comparison, sentence-processing commonly leads to increased activity in the anterior temporal lobe (aTL; see e.g. Rogalsky and Hickok, 2009, Stowe et al., 2005 for discussion), a finding consistent across both the auditory (Friederici et al., 2000, Humphries et al., 2005, Humphries et al., 2006, Jobard et al., 2007, Mazoyer et al., 1993, Rogalsky and Hickok, 2009) and visual (Jobard et al., 2007, Stowe et al., 1998, Vandenberghe et al., 2002) modalities. The majority of these studies report bilateral activation, with a stronger effect in the left hemisphere, while a subset have reported just a left-lateralized effect (Humphries et al., 2006, Vandenberghe et al., 2002).
Sentence processing is, of course, a complicated and multi-faceted task that is comprised of a host of separate computations, including the construction of sentence structure (syntactic parsing), the composition of complex meaning (semantic composition), the establishment of long distance dependencies, the determination of reference, and the drawing of pragmatic inferences. Thus, the cognitive operations involved in sentence processing differ substantially from those engaged by lists of words. While the functional role of the aTL is, accordingly, under-determined by the results from the sentence vs. word list comparison, the neuroimaging results together with deficit/lesion studies (Dronkers et al., 1994, Dronkers et al., 2004) have lead to hypotheses linking the aTL with aspects of linguistic composition (Pallier et al., 2011, Snijders et al., 2009), perhaps syntactic structure building (Grodzinsky and Friederici, 2006, Humphries et al., 2006), or incremental semantic composition (Stowe et al., 2005, Vandenberghe et al., 2002). A connection between this region and basic combinatory operations is also supported by recent work showing that the amount of sentence structure constructed word-by-word correlates with hemodynamic activity in the aTL (Brennan et al., in press). Additional evidence comes from comparisons of native language sentences to sentences in a foreign language (Mazoyer et al., 1993, Schlosser et al., 1998), non-linguistic sounds (Humphries et al., 2001), and rest (Bottini et al., 1994). Furthermore, evidence from eye-tracking and electrophysiology strongly suggests that sentence-level computations, such as syntactic and semantic composition, are engaged automatically and very rapidly word-by-word during language comprehension (Altmann and Kamide, 1999, Altmann and Steedman, 1988, Chambers et al., 2002, Friederici, 2002, Kamide et al., 2003, Kutas and Hillyard, 1980, Kutas and Hillyard, 1984, Neville et al., 1991, Tanenhaus et al., 1995). Taken together, the basic composition hypothesis of the aTL and the immediacy of sentence structure-building predict that activation in this region should be time-locked to word presentation. To test this, we examined sentence-processing, in comparison to lists of words, using magnetoencephalography (MEG), which has a temporal resolution at the millisecond level, but which also provides spatial resolution on the order of centimeters (Hämäläinen et al., 1993), permitting comparison with the existing neuroimaging literature.
The use of MEG also offers an opportunity to investigate the spatio-temporal characteristics of sentence-level processes more broadly. In addition to addressing more general questions about the temporal profile of sentence comprehension, MEG may offer some insight into a somewhat puzzling aspect of the prior findings on sentences vs. lists; namely, although in some studies the aTL effect has been accompanied by effects in other regions, such as the posterior temporal lobe (Friederici et al., 2000, Jobard et al., 2007, Pallier et al., 2011, Snijders et al., 2009, Vandenberghe et al., 2002, Xu et al., 2005) and the inferior frontal gyrus (Pallier et al., 2011, Snijders et al., 2009, Xu et al., 2005), several studies have found a rather focal effect in the aTL (Humphries et al., 2006, Rogalsky and Hickok, 2009, Stowe et al., 1998). In light of the many computations affected by this manipulation, an effect limited to a single region is rather surprising. One possible reason for hemodynamic methods to potentially yield this type of focal finding has to do with the extreme speed of linguistic computation, which stands in contrast to the slow temporal resolution of the imaging methods. As noted above, there is a great deal of evidence that during comprehension, sentences are interpreted incrementally, word-by-word (e.g. Kutas and Hillyard, 1980, Marslen-Wilson, 1975, Tanenhaus et al., 1995). Thus perhaps hemodynamic methods, which integrate over brain activity across several seconds, may have reduced sensitivity to transient effects elicited word-by-word (see also Lau et al., 2008:926). If this is correct, then a faster technique such as MEG should yield a broad network of language-related regions for sentences over lists, including, but not limited to the aTL.
Contrary to much of the prior literature (but cf. Xu et al., 2005), we also aimed to characterize sentence processing in a relatively natural setting. Many previous sentence vs. list studies have used explicit meta-linguistic tasks, having subjects judge meaningfulness (Humphries et al., 2006), identity (Mazoyer et al., 1993), or syntactic and semantic errors (Friederici et al., 2000), while others have used a passive listening task with isolated sentences devoid of any context (Humphries et al., 2005, Rogalsky and Hickok, 2009). Such stimuli and tasks, however, may lead to brain activity that is not necessarily related to core language comprehension computations. To address these concerns, our stimuli were embedded within a narrative. While this design may lead to processing associated with both sentence-level and discourse/narrative-level computations, it is not clear whether operations associated with constructing higher-level narrative representations are time-locked to individual words, which is the only type of effect we analyzed here. Psychologically plausible parsing models clearly predict that syntactic and semantic operations are engaged incrementally, word-by-word (e.g. Hale, 2011, Stabler, 1991, Steedman, 2000, Vosse and Kempen, 2000) and thus should be reflected in brain activity that is time-locked to word presentation. However, to our knowledge, there are no models of discourse-level processes in which these processes are computed word-by-word. To improve the sensitivity of our analysis, we also sought to maximize the similarity of the low-level visual properties of our stimuli by restricting our analysis to just open-class words that were three to eight characters long (cf. Pulvermüller, 2001).
What brain regions are candidates for sentence level processing? If the aTL is indeed associated with basic combinatory operations, as discussed above, then clearly sentences should elicit more aTL activity than lists, as reported in the previous imaging studies. A second candidate is the ventromedial prefrontal cortex (vmPFC), which has been implicated for semantic composition by a growing body of MEG studies (Brennan and Pylkkänen, 2008, Brennan and Pylkkänen, 2010, Pylkkänen and McElree, 2007, Pylkkänen et al., 2008, Pylkkanen et al., 2009). The vmPFC has also been observed to become more active as comprehension of a story increases (Maguire et al., 1999) and when subjects are asked to complete sentences with a word which fits with a given sentential context as opposed to one that does not (Nathaniel-James and Frith, 2002). Further, it forms part of a network of strongly connected regions observed during reading (Kujala et al., 2007).
Finally, and most famously, the left inferior frontal gyrus (LIFG; Broca's area) has long been considered a sentence processing related region, based on deficit/lesion research (Caramazza and Zurif, 1976, Zurif, 1995) and a variety of neuroimaging studies, including research on the processing of long-distance linguistic dependencies (Ben-Shachar et al., 2003, Ben-Shachar et al., 2004, Caplan et al., 2008, Grodzinsky, 2001, Just et al., 1996, Santi and Grodzinsky, 2007a, Santi and Grodzinsky, 2007b, Santi and Grodzinsky, 2010, Stromswold et al., 1996), non-standard word order (Bornkessel et al., 2005, Bornkessel-Schlesewsky et al., 2009, Grewe et al., 2005, Grewe et al., 2006), and selective attention to syntactic aspects of a stimulus (Dapretto and Bookheimer, 1999, Embick et al., 2000, Hashimoto and Sakai, 2002). Specific functional hypotheses have linked this region with syntactic movement (Grodzinsky, 2001, Grodzinsky and Friederici, 2006), working memory demands (Fiebach et al., 2005), and linearization computations (Bornkessel et al., 2005). The LIFG has also been associated with a range of lexical-level processes, including semantic retrieval (Bookheimer, 2002), cognitive control mechanisms engaged by ambiguous words or structures (Bedny et al., 2007, Thompson-Schill et al., 1997), and the evaluation of word meaning in the context of world knowledge (Hagoort et al., 2004, Lau et al., 2008). Although LIFG effects are observed in a myriad of manipulations, the sentence vs. word list contrast employed here has not systematically shown LIFG effects in fMRI or PET (but cf. Pallier et al., 2011, Snijders et al., 2009, Xu et al., 2005; see also Lerner et al., 2011), leading to a controversy about the centrality of the LIFG in processing sentence structure (Rogalsky and Hickok, 2010, Stowe et al., 2005). However, given the grossness of the sentence vs. word list contrast, a LIFG effect for this manipulation would be compatible with many of the above hypotheses, as long as they relate to combinatory processing in some way, even if indirectly (e.g., modulation of lexical access via sentential context).
To summarize, this experiment compared the processing of open-class words which were either presented in randomized word lists or embedded in a narrative using MEG. Analysis was done time-locked to word onsets to identify spatio-temporal patterns of brain activity associated with rapid, word-by-word sentence-level computations.
Section snippets
Participants
9 subjects from the New York University community participated in the experiment. Participants ranged in age from 18 to 30 (median 21) and 6 of the participants were women. All subjects were right-handed (assessed using the Edinburgh Handedness Inventory; Oldfield, 1971) and were native speakers of English with normal vision.
Stimuli
Participants were presented with the story “Sleeping Beauty” (Marshall, 1917; text available at http://www.gutenberg.org/etext/20748). The text contains 82 sentences
Results
Sensor waveforms of the sentence and list conditions for a representative subject are shown in Fig. 2. There is a visually salient increase of activity in a component spanning from approximately 250 to 450 ms. Fig. 3 shows snapshots of estimated source activity in the left hemisphere across time, grand-averaged across subjects. Visual inspection shows increased inferior frontal and anterior temporal activation between 300 and 500 ms.
Discussion
In this work we investigated the neural correlates of sentence-level processing using MEG to address two goals. First, we evaluated whether anterior temporal activity, observed in previous neuroimaging studies, is time-locked to word presentation, as expected under the hypothesis that activity in this region is associated with basic combinatoric processing (Brennan et al., in press, Dronkers et al., 2004, Grodzinsky and Friederici, 2006, Hickok and Poeppel, 2007, Humphries et al., 2006,
Conclusion
Brain activity associated with processing sentences was examined using MEG to test whether anterior temporal activation for sentence processing is time-locked to word presentation and to better understand the puzzling neuroimaging finding that sentences may lead to a focal increase of activation in the anterior temporal lobes, compared to word lists. Comparing the processing of sentences embedded in a story with the same words in random lists, more activation was found for sentences between 250
References (111)
- et al.
Incremental interpretation at verbs: restricting the domain of subsequent reference
Cognition
(1999) - et al.
Interaction with context during human sentence processing
Cognition
(1988) - et al.
Mixed-effects modeling with crossed random effects for subjects and items
J. Mem. Lang.
(2008) - et al.
Understanding words in context: the role of Broca's area in word comprehension
Brain Res.
(2007) - et al.
Neural correlates of syntactic movement: converging evidence from two fMRI experiments
Neuroimage
(2004) - et al.
Who did what to whom? The neural basis of argument hierarchies during language comprehension
Neuroimage
(2005) - et al.
Word order and Broca's region: evidence for a supra-syntactic perspective
Brain Lang.
(2009) - et al.
Processing events: behavioral and neuromagnetic correlates of aspectual coercion
Brain Lang.
(2008) - et al.
Task-dependent and task-independent neurovascular responses to syntactic processing
Cortex
(2008) - et al.
Dissociation of algorithmic and heuristic processes in language comprehension: evidence from aphasia
Brain Lang.
(1976)
Circumscribing referential domains during real-time language comprehension
J. Mem. Lang.
Dynamic statistical parametric mapping: combining fMRI and MEG for high-resolution imaging of cortical activity
Neuron
Form and content: dissociating syntax and semantics in sentence comprehension
Neuron
Hemodynamic nonlinearities affect BOLD fMRI response timing and amplitude
Neuroimage
An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest
Neuroimage
Before the N400: effects of lexical-semantic violations in visual cortex
Brain Lang.
Sensitivity to syntax in visual cortex
Cognition
Lesion analysis of the brain areas involved in language comprehension: towards a new functional anatomy of language
Cognition
The independence of syntactic processing
J. Mem. Lang.
Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain
Neuron
The sausage machine: a new two-stage parsing model
Cognition
Towards a neural basis of auditory sentence processing
Trends Cognit. Sci.
Auditory language comprehension: an event-related fMRI study on the processing of syntactic and lexical information
Brain Lang.
Thresholding of statistical maps in functional neuroimaging using the false discovery rate
Neuroimage
Linguistic prominence and Broca's area: the influence of animacy as a linearization principle
Neuroimage
Neuroimaging of syntax and syntactic processing
Curr. Opin. Neurobiol.
Specialization in the left prefrontal cortex for sentence comprehension
Neuron
The time course of visual word recognition as revealed by linear regression analysis of ERP data
Neuroimage
Localization of the syntactic mismatch negativity in the temporal cortex: an MEG study
Neuroimage
Impact of modality and linguistic complexity during reading and listening tasks
Neuroimage
The brain circuitry of syntactic comprehension
Trends Cogn. Sci.
The time-course of prediction in incremental sentence processing: evidence from anticipatory eye movements
J. Mem. Lang.
Linear and nonlinear relationships between visual stimuli, EEG and BOLD fMRI signals
Neuroimage
Nonparametric statistical testing of EEG- and MEG-data
J. Neurosci. Methods
The temporal structure of spoken language understanding
Cognition
Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension
J. Mem. Lang.
The role of the dorsolateral prefrontal cortex: evidence from the effects of contextual constraint in a sentence completion task
Neuroimage
The assessment and analysis of handedness: the Edinburgh inventory
Neuropsychologia
Brain reflections of words and their meaning
Trends Cogn. Sci.
Working memory and syntax interact in Broca's area
Neuroimage
fMRI adaptation dissociates syntactic complexity dimensions
Neuroimage
Reduction of non-periodic environmental magnetic noise in MEG measurement by continuously adjusted least squares method
IEEE Trans. Appl. Supercond.
lme4: linear mixed-effects models using S4 classes
R package, Version 0.999375-31
The control of the false discovery rate in multiple testing under dependency
Ann. Stat.
The neural reality of syntactic transformations: evidence from fMRI
Psychol. Sci.
Functional MRI of language: new approaches to understanding the cortical organization of semantic processing
Annu. Rev. Neurosci.
The role of the right hemisphere in the interpretation of figurative aspects of language: a positron emission tomography activation study
Brain
Processing psych verbs: behavioral and MEG measures of two different types of semantic complexity
Lang. Cognit. Process.
PsyScope: a new graphic interactive environment for designing psychology experiments
Behav. Res. Methods Instrum. Comput.
Cited by (92)
Predictions about prosody facilitate lexical access: Evidence from P50/N100 and MMN components
2023, International Journal of PsychophysiologyElectrophysiological correlates of basic semantic composition in people with aphasia
2023, NeuroImage: Clinical