Elsevier

NeuroImage

Volume 37, Issue 3, 1 September 2007, Pages 993-1004
NeuroImage

On sense and reference: Examining the functional neuroanatomy of referential processing

https://doi.org/10.1016/j.neuroimage.2007.05.048Get rights and content

Abstract

In an event-related fMRI study, we examined the cortical networks involved in establishing reference during language comprehension. We compared BOLD responses to sentences containing referentially ambiguous pronouns (e.g., “Ronald told Frank that he…”), referentially failing pronouns (e.g., “Rose told Emily that he…”) or coherent pronouns. Referential ambiguity selectively recruited medial prefrontal regions, suggesting that readers engaged in problem-solving to select a unique referent from the discourse model. Referential failure elicited activation increases in brain regions associated with morpho-syntactic processing, and, for those readers who took failing pronouns to refer to unmentioned entities, additional regions associated with elaborative inferencing were observed. The networks activated by these two referential problems did not overlap with the network activated by a standard semantic anomaly. Instead, we observed a double dissociation, in that the systems activated by semantic anomaly are deactivated by referential ambiguity, and vice versa. This inverse coupling may reflect the dynamic recruitment of semantic and episodic processing to resolve semantically or referentially problematic situations. More generally, our findings suggest that neurocognitive accounts of language comprehension need to address not just how we parse a sentence and combine individual word meanings, but also how we determine who's who and what's what during language comprehension.

Introduction

At least since Frege (1892), a distinction has been made between a word's general, dictionary-like meaning (sense) and the particular entity it denotes (reference). This distinction becomes clear when one compares, for example, the meaning carried by the word ‘doctor’ in (1) Hannibal Lecter is a doctor, and in (2) Hannibal Lecter is eating a doctor. The lexical item ‘doctor’ has an identical sense in 1 and 2 (i.e., someone who has completed medical school), but it provides information about Hannibal Lecter in 1, while it refers to another, rather unfortunate doctor in 2. Thus, to appreciate the meaning of a word in a given context, one must establish both the general, lexical meaning and the specific, referential meaning it conveys.

In psycholinguistics, sense and reference are often associated with semantic and referential analysis, respectively. Through semantic analysis, we retrieve the meaning of individual words from long-term memory and combine them into larger units of conceptual structure. Through referential analysis, we determine who's who and what's what by establishing the relationships between words in the discourse and entities in our mental representation of the world (e.g., persons, events, concepts, places), real or fictional. Alongside the phonological and syntactic aspects of language comprehension, semantic and referential analysis has been subjected to experimental investigation throughout the past decades (e.g., see Garnham, 2001, Carreiras and Clifton, 2004). Convergent evidence from behavioral measures and event-related potentials (ERPs) has suggested that listeners and readers immediately relate the semantic and referential meaning of incoming words to the global context (e.g., Kutas and Hillyard, 1980, Marslen-Wilson and Tyler, 1980, Nieuwland and Van Berkum, 2006b, Tanenhaus and Trueswell, 1995, Van Berkum et al., 1999).

Importantly, results from ERP experiments show that the brain responds in different ways to problems with reference than to problems with either syntax or semantics. While semantic and syntactic processing are predominantly associated with the N400 and P600/LAN respectively (e.g., Friederici, 2002, Hagoort et al., 1999), referentially ambiguous nouns (e.g., “the girl” in a two-girl context) elicit a frontally dominant and sustained negative shift (Van Berkum et al., 1999, Van Berkum et al., 2003). This referentially induced ERP effect stems from having two suitable referential candidates for a single anaphor (Nieuwland et al., 2007; see Van Berkum et al., 2007, for review), and enables us to selectively track referential processing at the level that is most relevant for comprehension, the mental representation of the situation as described in the discourse (the situation model, Zwaan and Radvansky, 1998).

To examine the causal antecedents of this referentially induced frontal negativity, we have recently investigated how people establish reference for pronouns like “he” or “she”. Pronouns have a rather shallow sense of their own and constitute our linguistic ‘pointers’ for maintaining reference to topics that are in the focus of our conversation (e.g., Gordon et al., 1993, Sanford and Garrod, 1989). Because pronouns are among the most frequently used words in many languages, pronoun resolution is often regarded as the textbook example of referential processing. Consistent with our findings for nouns, referentially ambiguous pronouns (e.g., “Ronald told Frank that he…”) also elicit a frontal negative shift compared to referentially coherent pronouns (Nieuwland and Van Berkum, 2006a, Van Berkum et al., 2004). Furthermore, referentially failing pronouns (when no suitable referent is directly available, e.g., “Rose told Emily that he…”) elicit a P600 effect (Nieuwland and Van Berkum, 2006a, Van Berkum et al., 2004), suggesting that readers initially try to find an appropriate antecedent within the given set of discourse entities (e.g., Garnham, 2001) and therefore initially ascribe the referential failure to a problem with the syntactic gender of the pronoun (e.g., Osterhout and Mobley, 1995; see Van Berkum et al., 2007, for discussion).

Taken together, the ERP findings suggest that the brain responds in qualitatively different ways to distinct problems with referential analysis (failure/ambiguity), and that both responses, in turn, differ from the brain's response to semantically unexpected words (semantic anomaly). However, these findings do not tell the whole story. In other words, although the different ERP effects are indicative of the engagement of at least partially non-overlapping neural systems, they do not tell us which exact cortical networks are involved in the establishing of reference, or what their relationship is to the networks known to be involved in the processing of meaning. In the last decade, language comprehension researchers have begun to use functional neuroimaging techniques (e.g., fMRI) to unravel the neuronal infrastructure of semantic, syntactic and phonological processing (e.g., Bookheimer, 2002, Friederici, 2002, Kaan and Swaab, 2002, Stowe et al., 2005, for reviews). However, with the exception of a few studies focusing on cohesion/coherence processing in discourse comprehension (Ferstl and von Cramon, 2001, Robertson et al., 2000), there has been no functional magnetic resonance imaging (fMRI) work on the neural substrate for referential processing. As a result, we know virtually nothing about how the brain subserves a core aspect of language comprehension, how people work out who's who and what's what in text or conversation.

In the present study, we used fMRI to examine the functional neuroanatomical correlates of referential processing, and to relate the cortical systems involved in establishing reference to the cortical systems recruited by semantic processing. Following up on earlier ERP work, we focused on the processing of referentially ambiguous and referentially failing pronouns. Note that although referential ambiguity and referential failure both seem to involve disruptions of ‘normal’ language comprehension, such disruptions are in fact not uncommon in everyday language use (e.g., Auer, 1984), and understanding how readers deal with them is thus of a priori relevance to the study of reference resolution (e.g., Gordon et al., 1993, Sanford and Garrod, 1989). Furthermore, we know from other language comprehension research that the systems recruited by semantic and syntactic problems are also implicated in the processing of non-problematic language input. For these two reasons, we take our study to provide insights into the cortical networks involved in natural referential processing.

In our experiment, participants read sentences that were presented on a screen one word at a time, and that contained two narrative characters and one singular pronoun (cf., Table 1; see also Appendix A). This pronoun was referentially ambiguous, failing, or coherent depending on whether its gender matched both, none, or one of the characters mentioned in the sentence, respectively. In addition, referentially coherent sentences were either semantically anomalous if it contained a lexical-semantically anomalous word (downstream from the pronoun), or semantically coherent if it contained a non-anomalous control word.

The available literature suggests at least two possible scenarios that link ambiguous pronoun resolution to activity in the left inferior frontal gyrus (LIFG). First, referential ambiguity may require the language system to, from the ambiguous pronoun and onwards, actively maintain two candidate fillers for an unresolved single referential slot in working memory (e.g., Gibson, 1998). In such a scenario, the brain signature of referential ambiguity may resemble that of increased working memory load, eliciting increased activity in the LIFG (BA 44/45/47; Fletcher and Henson, 2001, Kaan and Swaab, 2002, Smith and Jonides, 1999). Second, in a recent review, Novick et al. (2005) posit a similar prediction for LIFG involvement in resolving referential ambiguity, albeit for different reasons. According to Novick et al. (2005), LIFG activations may signal resolution of the representational conflict that arises when semantic representations point towards competing interpretations, an account that makes intuitive sense with regard to referential ambiguity.

Thus, the abovementioned accounts all predict that referential ambiguity (compared to referential coherence) would lead to enhanced activation in the LIFG. This said, both accounts also predict substantial overlap for cortical networks involved in dealing with referential ambiguity and semantic anomaly, as semantic anomalies are known to elicit increased responses in the left anterior inferior frontal gyrus (LIFG, BA 45/47; e.g., Baumgaertner et al., 2002, Hagoort et al., 2004, Kiehl et al., 2002, Kuperberg et al., 2003).

In a third account, the processing of referential ambiguity is not necessarily linked to the LIFG at all. This account entails that ambiguous pronoun resolution requires readers to engage in problem-solving and inference procedures (e.g., Graesser et al., 1994, Greene et al., 1992). For example, readers may launch an additional search through their discourse memory (e.g., Myers and O'Brien, 1998) and infer that one of the referents is the correct one (e.g., through reasoning or decision-making). In this respect, ambiguous pronoun resolution can be considered to be a multi-faceted enterprise that relies on interrelated, domain-independent executive cognitive processes, recruiting a network of prefrontal and parietal regions (e.g., Cabeza and Nyberg, 2000, Fletcher and Henson, 2001, Wagner et al., 2005). Following such an account, increased responses may be observed in brain regions that subserve knowledge-based, non-automatic inferences that require the use of general world knowledge and/or discourse information (e.g., Ferstl, in press, Ferstl and Siebörger, 2007, for reviews). These regions include medial frontal (BA 8/9/10) and parietal (BA 7/23/31) cortices (e.g., Ferstl and von Cramon, 2001, Ferstl and von Cramon, 2002, Ferstl et al., 2005, Kuperberg et al., 2006, Maguire et al., 1999, Volz et al., 2006a, Volz et al., 2006b, Zysset et al., 2002, Zysset et al., 2003), but also right lateral prefrontal (BA 8/9/10/46) and bilateral parietal regions (BA 39/40) involved in monitoring and evaluating of memory retrieval outcomes (e.g., Buckner and Wheeler, 2001, Wagner et al., 2005).1

For referential failure (compared to referential coherence), we predicted that the brain treats referentially failing pronouns as morpho-syntactic violations in the first instance (e.g., Osterhout and Mobley, 1995, Van Berkum et al., 2007), eliciting activity increases in left middle/superior and inferior frontal (BA 6/8/44, Hammer et al., 2007, Newman et al., 2001, Ni et al., 2000), and medial and bilateral parietal regions (BA 7/23/31/40; Kuperberg et al., 2003, Ni et al., 2000). However, because the pronouns as used here are ‘free’ in a Chomskian sense (Chomsky, 1981), there are actually alternative resolutions of the referential failure at hand. For example, readers may stick with ‘blaming’ the syntactic gender of the pronoun, but they may also invoke a third, unmentioned person (e.g., ‘he’ in “Rose told Emily that he had a positive attitude towards life” can very well be taken to refer to Rose‘s new boyfriend). Because these two different interpretations differ in whether or not the referents were only looked for ‘within the sentence’, we will refer to them as sentence-internal and sentence-external referential interpretations. In an ERP study examining referential failure, Osterhout and Mobley (1995) reported that these different interpretations may indeed elicit qualitatively different brain responses. In one of the analyses in the present study, we used the results from a post-experiment debriefing to examine whether the reliance on different resolution strategies was also reflected in differential BOLD response patterns.

In summary, the aim of the present study was to investigate the cortical networks involved in different aspects of referential processing, a crucial facet of language comprehension that has hardly been touched upon by cognitive neuroscientists. We contrasted different functional accounts of referential ambiguity that are associated with different functional neuroanatomical predictions (particularly with respect to involvement of the LIFG, a brain area typically associated with semantic processing). For referential failure, we predicted, based on earlier ERP findings, that failing pronouns would elicit increased activity in brain regions involved in morpho-syntactic processing. Furthermore, we used results from a post-experiment briefing to examine potential individual differences in reference resolution strategies. Finally, the inclusion of a semantic anomaly manipulation in our design allowed us to address whether referential and semantic aspects of language comprehension draw upon common or different brain regions.

Section snippets

Participants

Twenty-two right-handed college students (4 males, mean age 21, S.D. = 2.3) participated in this study for course credits or a small monetary reimbursement. All participants were native speakers of Dutch. None of them used medication or had a history of drug abuse, head trauma, neurological or psychiatric illness. The experiment was approved by the medical ethical committee of the Amsterdam Medical Center, and informed consent was obtained from all subjects.

Stimulus materials

180 sentences (a subset of 270

Results

We first investigated the contrasts involving referential ambiguity, referential failure and semantic anomaly compared to their coherent counterparts. The corresponding statistical results are presented in Table 2. As visible from Fig. 1A, referential ambiguity (ambiguity > coherence; Fig. 1A, Table 2A) was associated with greater activation in a set of regions that included medial frontal gyrus (BA 10), right superior frontal cortex (BA 8/9), medial parietal cortices (precuneus, BA 7/31), and

Discussion

The objective of the present study was to investigate the functional neuroanatomical correlates of referential processing. We compared event-related BOLD-fMRI responses to sentences containing referentially ambiguous (e.g., “Ronald told Frank that he…”), referentially failing pronouns (e.g., “Rose told Emily that he…”) or referentially coherent pronouns (e.g., “Frank told Emily that he…”). To provide a benchmark for these results, we also compared responses to sentences containing semantically

Acknowledgments

We gratefully thank Steven Scholte for his technical advice, Tessa Dekker and Sietske Kleibeuker for help in data collection, and Gina Kuperberg, and Evelyn Ferstl for valuable comments on an earlier draft of this manuscript. This research was supported by an NWO Innovation Impulse Vidi grant to JVB. MSN is currently funded by a Rubicon grant from the Netherlands Organisation for Scientific Research (NWO).

References (79)

  • V. Goel et al.

    Explaining modulation of reasoning by belief

    Cognition

    (2003)
  • P.C. Gordon et al.

    Pronouns, names, and the centering of attention in discourse

    Cogn. Sci.

    (1993)
  • P. Hagoort

    On broca, brain, and binding: a new framework

    Trends Cogn. Sci.

    (2005)
  • A. Hammer et al.

    When sex meets syntactic gender on a neural basis during pronoun processing

    Brain Res.

    (2007)
  • E. Kaan et al.

    The brain circuitry of syntactic comprehension

    Trends Cogn. Sci.

    (2002)
  • K.A. Kiehl et al.

    Reading anomalous sentences: an event-related fMRI study of semantic processing

    NeuroImage

    (2002)
  • G.R. Kuperberg et al.

    Making sense of discourse: an fMRI study of causal inferenceing across sentences

    NeuroImage

    (2006)
  • W. Marslen-Wilson et al.

    The temporal structure of spoken language understanding

    Cognition

    (1980)
  • M.S. Nieuwland et al.

    Individual differences and contextual bias in pronoun resolution: evidence from ERPs

    Brain Res.

    (2006)
  • L. Osterhout et al.

    Event-related brain potentials elicited by failure to agree

    J. Mem. Lang.

    (1995)
  • K.M. Petersson et al.

    Instruction-specific brain activations during episodic encoding. A generalized level of processing effect

    Neuroimage

    (2003)
  • K.M. Petersson et al.

    Interaction between a verbal working memory network and the medial temporal lobe

    NeuroImage

    (2006)
  • L.A. Stowe et al.

    Rethinking the neurological basis of language

    Lingua

    (2005)
  • C. Tallon-Baudry et al.

    Oscillatory gamma activity in humans and its role in object representation

    Trends Cogn. Sci.

    (1999)
  • M.K. Tanenhaus et al.

    Sentence comprehension

  • J.J.A. Van Berkum et al.

    Early referential context effects in sentence processing: evidence from event-related brain potentials

    J. Mem. Lang.

    (1999)
  • J.J.A. Van Berkum et al.

    Establishing reference in language comprehension: an electrophysiological perspective

    Brain Res.

    (2007)
  • A.D. Wagner et al.

    Parietal lobe contributions to episodic memory retrieval

    Trends Cogn. Sci.

    (2005)
  • S. Zysset et al.

    The anterior frontomedian cortex and evaluative judgment: an fMRI study

    NeuroImage

    (2002)
  • S. Zysset et al.

    Functional specialization within the anterior medial prefrontal cortex: a functional magnetic resonance imaging study with human subjects

    Neurosci. Lett.

    (2003)
  • D. Badre et al.

    Semantic retrieval, mnemonic control, and prefrontal cortex

    Behav. Cogn. Neurosci. Rev.

    (2002)
  • S. Bookheimer

    Functional MRI of language: new approaches to understanding the cortical organization of semantic processing

    Annu. Rev. Neurosci.

    (2002)
  • M. Brett et al.

    The problem of functional localization in the human brain

    Nat. Rev., Neurosci.

    (2002)
  • R.L. Buckner et al.

    The cognitive neuroscience of remembering

    Nat. Rev., Neurosci.

    (2001)
  • P.W. Burgess et al.

    The gateway hypothesis of rostral prefrontal cortex (area 10) function

  • R. Cabeza et al.

    Imaging cognition II: an empirical review of 275 PET and fMRI studies

    J. Cogn. Neurosci.

    (2000)
  • M. Carreiras et al.

    The On-Line Study of Sentence Comprehension: Eyetracking, ERPs and Beyond

    (2004)
  • Chomsky, N., 1981. Lectures on Government and Binding: The Pisa Lectures. Holland: Foris Publications 7th ed, Berlin...
  • A.C. Evans et al.
  • Cited by (76)

    • Decoding the silence: Neural bases of zero pronoun resolution in Chinese

      2022, Brain and Language
      Citation Excerpt :

      In both fMRI and MEG, Li et al. found that the left middle temporal gyrus (LMTG) is consistently activated for third-person pronoun processing in both English and Chinese. Yet they also found additional medial parietal activity from the MEG data, consistent with Brodbeck and Pylkkänen (2017), Brodbeck et al. (2016), and Nieuwland et al. (2007). To sum up, referential processing has been implicated in a number of regions, including the medial parietal lobe.

    View all citing articles on Scopus
    View full text