The time-course and spatial distribution of brain activity associated with sentence processing

Abstract

Sentence comprehension involves a host of highly interrelated processes, including syntactic parsing, semantic composition, and pragmatic inferencing. In neuroimaging, a primary paradigm for examining the brain bases of sentence processing has been to compare brain activity elicited by sentences versus unstructured lists of words. These studies commonly find an effect of increased activity for sentences in the anterior temporal lobes (aTL). Together with neuropsychological data, these findings have motivated the hypothesis that the aTL is engaged in sentence level combinatorics. Combinatoric processing during language comprehension, however, occurs within tens and hundreds of milliseconds, i.e., at a time-scale much faster than the temporal resolution of hemodynamic measures. Here, we examined the time-course of sentence-level processing using magnetoencephalography (MEG) to better understand the temporal profile of activation in this common paradigm and to test a key prediction of the combinatoric hypothesis: because sentences are interpreted incrementally, word-by-word, activity associated with basic linguistic combinatorics should be time-locked to word-presentation. Our results reveal increased anterior temporal activity for sentences compared to word lists beginning approximately 250 ms after word onset. We also observed increased activation in a network of other brain areas, extending across posterior temporal, inferior frontal, and ventral medial areas. These findings confirm a key prediction of the combinatoric hypothesis for the aTL and further elucidate the spatio-temporal characteristics of sentence-level computations in the brain.

Introduction

A broad neuroimaging literature has sought to identify neural activity associated with the cognitive operations engaged during the processing of sentences. A primary experimental design that has been used in this literature compares brain activity elicited when subjects read or listen to sentences to that observed for lists of words (Friederici et al., 2000, Humphries et al., 2005, Humphries et al., 2006, Jobard et al., 2007, Mazoyer et al., 1993, Rogalsky and Hickok, 2009, Snijders et al., 2009, Stowe et al., 1998, Vandenberghe et al., 2002, Xu et al., 2005). This contrast has the desirable property that both conditions are likely to require an equivalent degree of word-level processing, but only the sentences are assumed to require processing associated with building and comprehending sentence structure. In this comparison, sentence-processing commonly leads to increased activity in the anterior temporal lobe (aTL; see e.g. Rogalsky and Hickok, 2009, Stowe et al., 2005 for discussion), a finding consistent across both the auditory (Friederici et al., 2000, Humphries et al., 2005, Humphries et al., 2006, Jobard et al., 2007, Mazoyer et al., 1993, Rogalsky and Hickok, 2009) and visual (Jobard et al., 2007, Stowe et al., 1998, Vandenberghe et al., 2002) modalities. The majority of these studies report bilateral activation, with a stronger effect in the left hemisphere, while a subset have reported just a left-lateralized effect (Humphries et al., 2006, Vandenberghe et al., 2002).

Sentence processing is, of course, a complicated and multi-faceted task that is comprised of a host of separate computations, including the construction of sentence structure (syntactic parsing), the composition of complex meaning (semantic composition), the establishment of long distance dependencies, the determination of reference, and the drawing of pragmatic inferences. Thus, the cognitive operations involved in sentence processing differ substantially from those engaged by lists of words. While the functional role of the aTL is, accordingly, under-determined by the results from the sentence vs. word list comparison, the neuroimaging results together with deficit/lesion studies (Dronkers et al., 1994, Dronkers et al., 2004) have lead to hypotheses linking the aTL with aspects of linguistic composition (Pallier et al., 2011, Snijders et al., 2009), perhaps syntactic structure building (Grodzinsky and Friederici, 2006, Humphries et al., 2006), or incremental semantic composition (Stowe et al., 2005, Vandenberghe et al., 2002). A connection between this region and basic combinatory operations is also supported by recent work showing that the amount of sentence structure constructed word-by-word correlates with hemodynamic activity in the aTL (Brennan et al., in press). Additional evidence comes from comparisons of native language sentences to sentences in a foreign language (Mazoyer et al., 1993, Schlosser et al., 1998), non-linguistic sounds (Humphries et al., 2001), and rest (Bottini et al., 1994). Furthermore, evidence from eye-tracking and electrophysiology strongly suggests that sentence-level computations, such as syntactic and semantic composition, are engaged automatically and very rapidly word-by-word during language comprehension (Altmann and Kamide, 1999, Altmann and Steedman, 1988, Chambers et al., 2002, Friederici, 2002, Kamide et al., 2003, Kutas and Hillyard, 1980, Kutas and Hillyard, 1984, Neville et al., 1991, Tanenhaus et al., 1995). Taken together, the basic composition hypothesis of the aTL and the immediacy of sentence structure-building predict that activation in this region should be time-locked to word presentation. To test this, we examined sentence-processing, in comparison to lists of words, using magnetoencephalography (MEG), which has a temporal resolution at the millisecond level, but which also provides spatial resolution on the order of centimeters (Hämäläinen et al., 1993), permitting comparison with the existing neuroimaging literature.

The use of MEG also offers an opportunity to investigate the spatio-temporal characteristics of sentence-level processes more broadly. In addition to addressing more general questions about the temporal profile of sentence comprehension, MEG may offer some insight into a somewhat puzzling aspect of the prior findings on sentences vs. lists; namely, although in some studies the aTL effect has been accompanied by effects in other regions, such as the posterior temporal lobe (Friederici et al., 2000, Jobard et al., 2007, Pallier et al., 2011, Snijders et al., 2009, Vandenberghe et al., 2002, Xu et al., 2005) and the inferior frontal gyrus (Pallier et al., 2011, Snijders et al., 2009, Xu et al., 2005), several studies have found a rather focal effect in the aTL (Humphries et al., 2006, Rogalsky and Hickok, 2009, Stowe et al., 1998). In light of the many computations affected by this manipulation, an effect limited to a single region is rather surprising. One possible reason for hemodynamic methods to potentially yield this type of focal finding has to do with the extreme speed of linguistic computation, which stands in contrast to the slow temporal resolution of the imaging methods. As noted above, there is a great deal of evidence that during comprehension, sentences are interpreted incrementally, word-by-word (e.g. Kutas and Hillyard, 1980, Marslen-Wilson, 1975, Tanenhaus et al., 1995). Thus perhaps hemodynamic methods, which integrate over brain activity across several seconds, may have reduced sensitivity to transient effects elicited word-by-word (see also Lau et al., 2008:926). If this is correct, then a faster technique such as MEG should yield a broad network of language-related regions for sentences over lists, including, but not limited to the aTL.

Contrary to much of the prior literature (but cf. Xu et al., 2005), we also aimed to characterize sentence processing in a relatively natural setting. Many previous sentence vs. list studies have used explicit meta-linguistic tasks, having subjects judge meaningfulness (Humphries et al., 2006), identity (Mazoyer et al., 1993), or syntactic and semantic errors (Friederici et al., 2000), while others have used a passive listening task with isolated sentences devoid of any context (Humphries et al., 2005, Rogalsky and Hickok, 2009). Such stimuli and tasks, however, may lead to brain activity that is not necessarily related to core language comprehension computations. To address these concerns, our stimuli were embedded within a narrative. While this design may lead to processing associated with both sentence-level and discourse/narrative-level computations, it is not clear whether operations associated with constructing higher-level narrative representations are time-locked to individual words, which is the only type of effect we analyzed here. Psychologically plausible parsing models clearly predict that syntactic and semantic operations are engaged incrementally, word-by-word (e.g. Hale, 2011, Stabler, 1991, Steedman, 2000, Vosse and Kempen, 2000) and thus should be reflected in brain activity that is time-locked to word presentation. However, to our knowledge, there are no models of discourse-level processes in which these processes are computed word-by-word. To improve the sensitivity of our analysis, we also sought to maximize the similarity of the low-level visual properties of our stimuli by restricting our analysis to just open-class words that were three to eight characters long (cf. Pulvermüller, 2001).

What brain regions are candidates for sentence level processing? If the aTL is indeed associated with basic combinatory operations, as discussed above, then clearly sentences should elicit more aTL activity than lists, as reported in the previous imaging studies. A second candidate is the ventromedial prefrontal cortex (vmPFC), which has been implicated for semantic composition by a growing body of MEG studies (Brennan and Pylkkänen, 2008, Brennan and Pylkkänen, 2010, Pylkkänen and McElree, 2007, Pylkkänen et al., 2008, Pylkkanen et al., 2009). The vmPFC has also been observed to become more active as comprehension of a story increases (Maguire et al., 1999) and when subjects are asked to complete sentences with a word which fits with a given sentential context as opposed to one that does not (Nathaniel-James and Frith, 2002). Further, it forms part of a network of strongly connected regions observed during reading (Kujala et al., 2007).

Finally, and most famously, the left inferior frontal gyrus (LIFG; Broca's area) has long been considered a sentence processing related region, based on deficit/lesion research (Caramazza and Zurif, 1976, Zurif, 1995) and a variety of neuroimaging studies, including research on the processing of long-distance linguistic dependencies (Ben-Shachar et al., 2003, Ben-Shachar et al., 2004, Caplan et al., 2008, Grodzinsky, 2001, Just et al., 1996, Santi and Grodzinsky, 2007a, Santi and Grodzinsky, 2007b, Santi and Grodzinsky, 2010, Stromswold et al., 1996), non-standard word order (Bornkessel et al., 2005, Bornkessel-Schlesewsky et al., 2009, Grewe et al., 2005, Grewe et al., 2006), and selective attention to syntactic aspects of a stimulus (Dapretto and Bookheimer, 1999, Embick et al., 2000, Hashimoto and Sakai, 2002). Specific functional hypotheses have linked this region with syntactic movement (Grodzinsky, 2001, Grodzinsky and Friederici, 2006), working memory demands (Fiebach et al., 2005), and linearization computations (Bornkessel et al., 2005). The LIFG has also been associated with a range of lexical-level processes, including semantic retrieval (Bookheimer, 2002), cognitive control mechanisms engaged by ambiguous words or structures (Bedny et al., 2007, Thompson-Schill et al., 1997), and the evaluation of word meaning in the context of world knowledge (Hagoort et al., 2004, Lau et al., 2008). Although LIFG effects are observed in a myriad of manipulations, the sentence vs. word list contrast employed here has not systematically shown LIFG effects in fMRI or PET (but cf. Pallier et al., 2011, Snijders et al., 2009, Xu et al., 2005; see also Lerner et al., 2011), leading to a controversy about the centrality of the LIFG in processing sentence structure (Rogalsky and Hickok, 2010, Stowe et al., 2005). However, given the grossness of the sentence vs. word list contrast, a LIFG effect for this manipulation would be compatible with many of the above hypotheses, as long as they relate to combinatory processing in some way, even if indirectly (e.g., modulation of lexical access via sentential context).

To summarize, this experiment compared the processing of open-class words which were either presented in randomized word lists or embedded in a narrative using MEG. Analysis was done time-locked to word onsets to identify spatio-temporal patterns of brain activity associated with rapid, word-by-word sentence-level computations.