Elsevier

NeuroImage

Volume 150, 15 April 2017, Pages 292-307
NeuroImage

Cross-modal representation of spoken and written word meaning in left pars triangularis

https://doi.org/10.1016/j.neuroimage.2017.02.032Get rights and content

Highlights

  • Voxelwise univariate and multivoxel pattern analysis of two independent fMRI datasets.

  • Cross-modal semantic similarity effect in the left anterodorsal pars triangularis.

  • Semantic similarity effect for written words in the left ventromedial temporal cortex.

  • Semantic similarity effect for spoken words in the left superior temporal gyrus.

  • Pars triangularis is a convergence zone of written and spoken words processing streams.

Abstract

The correspondence in meaning extracted from written versus spoken input remains to be fully understood neurobiologically. Here, in a total of 38 subjects, the functional anatomy of cross-modal semantic similarity for concrete words was determined based on a dual criterion: First, a voxelwise univariate analysis had to show significant activation during a semantic task (property verification) performed with written and spoken concrete words compared to the perceptually matched control condition. Second, in an independent dataset, in these clusters, the similarity in fMRI response pattern to two distinct entities, one presented as a written and the other as a spoken word, had to correlate with the similarity in meaning between these entities. The left ventral occipitotemporal transition zone and ventromedial temporal cortex, retrosplenial cortex, pars orbitalis bilaterally, and the left pars triangularis were all activated in the univariate contrast. Only the left pars triangularis showed a cross-modal semantic similarity effect. There was no effect of phonological nor orthographic similarity in this region. The cross-modal semantic similarity effect was confirmed by a secondary analysis in the cytoarchitectonically defined BA45. A semantic similarity effect was also present in the ventral occipital regions but only within the visual modality, and in the anterior superior temporal cortex only within the auditory modality. This study provides direct evidence for the coding of word meaning in BA45 and positions its contribution to semantic processing at the confluence of input-modality specific pathways that code for meaning within the respective input modalities.

Introduction

According to one of the most influential contemporary language models, the neural network underlying speech processing can be divided into a dorsal and a ventral stream (Hickok and Poeppel, 2004, Ueno et al., 2011). Both streams project into the inferior frontal gyrus (IFG) which has a crucial role in the interaction of these two streams (Saur et al., 2008, Rijntjes et al., 2012, Hamzei et al., 2016). The most natural word input modalities are spoken words (Chafe and Tannen, 1987) but in many cultures writing can convey meaning efficiently. Given the similarity in the meaningful messages conveyed, the path connecting written word input with word meaning presumably converges with that for auditory input at a given stage. The cognitive and neurobiological architecture of this confluence between written and spoken word input has been of longstanding interest to evolutionary and developmental neuroscience (Chafe and Tannen, 1987), neuropsychology (Allport and Funnell, 1981) and, more recently, functional imaging of the intact brain (Booth et al., 2002, Chee et al., 1999, Constable et al., 2004, Gold et al., 2005, Homae, 2002, Spitsyna et al., 2006, Wagner et al., 2001). The study of the differences and commonalities in processing of meaning between different input-modalities (written or spoken words, pictures, etc.) should not be confounded with the study of the effect of type of attributes of concrete entities (e.g. shape, sound, etc.) (Vandenbulcke et al., 2006, Huth et al., 2016) nor with the study of the inner format of semantic representations (Caramazza et al., 1990, Barsalou, 2016).

fMRI activations during semantic processing that are common for written and spoken words (Booth et al., 2002, Chee et al., 1999, Constable et al., 2004, Gold et al., 2005, Homae, 2002, Spitsyna et al., 2006, Wagner et al., 2001) (for review see Binder et al., 2009) can arise for various reasons. Domain-general processes (e.g. common selection, Thompson-Schill et al., 1997, or control processes, Gold et al., 2005) may operate on written and spoken word meaning that is represented at a distance (Gold et al., 2005, Hagoort, 2005). Or written words may be sounded out internally and the phonological operations associated with this process may give rise to apparent commonality with spoken word input. Thirdly, neuronal populations may code for the meaningful content of the words independent of the input modality in which the words were originally presented. A classical conjunction univariate analysis might reveal conjoint activation for these different reasons. Representational similarity analysis (RSA) (Kriegeskorte et al., 2008, Fairhall and Caramazza, 2013, Devereux et al., 2013) provides an opportunity to directly test the representational content and its dependence on input-modality. Depending on the behavioral matrix to which the fMRI matrix is compared and the extent of the stimulus set, RSA can reach a level of cognitive specificity and item-by-item granularity that cannot be attained by previous approaches to cross-modal processing (Homae, 2002, Kircher et al., 2009, Sass et al., 2009, Simanova et al., 2014). Among the cross-modal RSA studies, the majority used written words together with pictures (Devereux et al., 2013). An advantage of using words only is that it avoids the perceptual confound induced by the covariance between visual characteristics of pictures and their meaning (Fernandino et al., 2015): Compared to pictures, the relation between word form and meaning is far more arbitrary.

In the current study, we determined RSA effects of semantic similarity for cross-modal pairs, written versus spoken concrete nouns. First, a set of regions was defined based on a univariate analysis using an explicit semantic task performed with written and with spoken words compared to a lower-level control condition with consonant letter strings and rotated spectrograms, respectively. In a subsequent independent experiment, we determined within this set of regions whether semantic similarity between pairs of words is reflected in the activity pattern despite differences in word input-modality, spoken versus written. We also ascertained that the similarity in activity patterns could not be explained by phonological or orthographic similarity between words. In order to obtain a more complete picture of how crossmodal semantic similarity effects relate anatomically to input-modality specific processing pathways, we additionally searched for semantic similarity effects within the written or the spoken word processing pathway.

Section snippets

Participants

Eighteen subjects (12 women, 6 men), between 18 and 28 years, participated in a first fMRI experiment (univariate analysis). Twenty different subjects (14 women, 6 men), between 18 and 28 years, participated in a second, independent experiment, which was optimized for RSA (see below). All subjects were native Dutch speakers, right-handed, free of neurological or psychiatric history and had no hearing impairment. There was no overlap between the two subject groups. All the procedures were

Behavioral analysis

In the VOI defining experiment, the main effect of task on reaction times was significant (F(1,17)=36.147; P=0.00001) as was the main effect of input-modality (F(1,17)=223.68; P=0.00000), without interaction effect between the two factors (F(1,17)=0.00046; P=0.98315). Responses were significantly slower during the experimental than during the control condition and during the auditory than during the written conditions (Table 1). The effect of input modality on reaction times was confirmed in

Discussion

Previous fMRI studies using univariate analysis (Booth et al., 2002, Chee et al., 1999, Constable et al., 2004, Gold et al., 2005, Homae, 2002, Spitsyna et al., 2006, Vandenberghe et al., 1996, Wagner et al., 2001) have implicated BA45 in semantic processing. In the current study, using RSA we demonstrate that BA45 codes for the semantic relationships between entities and makes abstraction of the exact input modality in which these entities were presented. This novel finding provides insight in

Acknowledgments

R.V. is a Senior Clinical Investigator of the Research Foundation Flanders (FWO). R.B. is a postdoctoral fellow of the Research Foundation Flanders (FWO). Funded by Federaal Wetenschapsbeleid (Belspo 7/11), FWO (Grant no. G0925.15) and KU Leuven (OT/12/097).

References (95)

  • S.B. Eickhoff et al.

    A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data

    Neuroimage

    (2005)
  • A.D. Friederici

    Towards a neural basis of auditory sentence processing

    Trends Cogn. Sci.

    (2002)
  • A.D. Friederici et al.

    The language network

    Curr. Opin. Neurobiol.

    (2013)
  • T. Goucha et al.

    The language skeleton after dissecting meaning, a functional segregation within Broca's Area

    NeuroImage

    (2015)
  • P. Hagoort

    On Broca, brain, and binding, a new framework

    Trends Cogn. Sci.

    (2005)
  • G. Handjaras et al.

    How concepts are encoded in the human brain, a modality independent, category-based cortical organization of semantic knowledge

    NeuroImage

    (2016)
  • S. Heim et al.

    The role of the left Brodmann's areas 44 and 45 in reading words and pseudowords

    Cogn. Brain Res.

    (2005)
  • G. Hickok et al.

    Dorsal and ventral streams, a framework for understanding aspects of the functional anatomy of language

    Cognition

    (2004)
  • F. Homae

    From perception to sentence comprehension, the convergence of auditory and visual information of language in the left inferior frontal cortex

    NeuroImage

    (2002)
  • A.G. Huth et al.

    A continuous semantic space describes the representation of thousands of object and action categories across the human brain

    Neuron

    (2012)
  • K.B. McDermott et al.

    A procedure for identifying regions preferentially activated by attention to semantic and phonological relations using functional magnetic resonance imaging

    Neuropsychologia

    (2003)
  • T. Naselaris et al.

    Encoding and decoding in fMRI

    Neuroimage

    (2011)
  • R.A. Poldrack et al.

    Functional specialization for semantic and phonological processing in the left inferior prefrontal cortex

    Neuroimage

    (1999)
  • J.B. Poline et al.

    Combining spatial extent and peak intensity to test for activations in functional imaging

    NeuroImage

    (1997)
  • K. Sass et al.

    Lion-tiger-stripes, neural correlates of indirect semantic priming across processing modalities

    Neuroimage

    (2009)
  • D. Saur et al.

    Combining functional and anatomical connectivity reveals brain networks for auditory language comprehension

    NeuroImage

    (2010)
  • M.G. Stokes et al.

    Dynamic coding for cognitive control in prefrontal cortex

    Neuron

    (2013)
  • T. Ueno et al.

    Lichtheim 2, synthesizing aphasia and the neural basis of language in a neurocomputational model of the dual dorsal-ventral language pathways

    Neuron

    (2011)
  • L. van Doren et al.

    The amodal system for conscious word and picture identification in the absence of a semantic task

    NeuroImage

    (2010)
  • R. Vandenberghe et al.

    The associative-semantic network for words and pictures, effective connectivity and graph analysis

    Brain Lang.

    (2013)
  • A.D. Wagner et al.

    Recovering meaning

    Neuron

    (2001)
  • C. Weiller et al.

    How the ventral pathway got lost-and what its recovery might mean

    Brain Lang.

    (2011)
  • Y. Xu et al.

    Intrinsic functional network architecture of human semantic processing, modules and hubs

    NeuroImage

    (2016)
  • D.A. Allport et al.

    Components of the mental lexicon

    Philos. Trans. R. Soc. B: Biol. Sci.

    (1981)
  • K. Amunts et al.

    Broca's region, novel organizational principles and multiple receptor mapping

    PLoS Biol.

    (2010)
  • K. Amunts et al.

    Broca's region revisited, cytoarchitecture and intersubject variability

    J. Comp. Neurol.

    (1999)
  • B.J. Baars

    A Cognitive Theory of Consciousness

    (1993)
  • Baayen, R.H., Piepenbrock, R. Gulikers, L., 1995. The CELEX Lexical Database (Release 2) [CD-ROM]. Linguistic Data...
  • R. Baayen et al.

    The CELEX Lexical Database

    (1993)
  • L.W. Barsalou

    On staying grounded and avoiding quixotic dead ends

    Psychon. Bull. Rev.

    (2016)
  • J.R. Binder et al.

    Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies

    Cereb. Cortex

    (2009)
  • M.F. Bonner et al.

    Where is the anterior temporal lobe and what does it do?

    J. Neurosci.

    (2013)
  • J.R. Booth et al.

    Modality independence of word comprehension

    Hum. Brain Mapp.

    (2002)
  • R. Bruffaerts et al.

    Similarity of fMRI activity patterns in left perirhinal cortex reflects semantic similarity between words

    J. Neurosci.

    (2013)
  • C. Bundesen et al.

    A neural theory of visual attention, bridging cognition and neurophysiology

    Psychol. Rev.

    (2005)
  • A. Caramazza et al.

    The multiple semantics hypothesis, multiple confusions?

    Cogn. Neuropsychol.

    (1990)
  • W. Chafe et al.

    The relation between written and spoken language

    Annu. Rev. Anthropol.

    (1987)
  • Cited by (36)

    • Similar activation patterns in the bilateral dorsal inferior frontal gyrus for monolingual and bilingual contexts in second language production

      2021, Neuropsychologia
      Citation Excerpt :

      Our interpretation of cross-context PS in the bilateral pars opercularis as a reflection of similar phonological information was supported by two additional analyses. First, the bilateral ventral IFGs (i.e., pars triangularis) as key regions for semantic processing (Binder et al., 2009; Fridriksson et al., 2015; Liuzzi et al., 2017; Poldrack et al., 1999) did not show significant cross-context PS in L2 naming. Second, the bilateral pars opercularis did not show significant cross-language PS for the same items relative to different items.

    • Representation of associative and affective semantic similarity of abstract words in the lateral temporal perisylvian language regions

      2020, NeuroImage
      Citation Excerpt :

      Liuzzi et al. (2019) demonstrated that the associative model is well-suited for the representation of concrete words, but here, concrete words alone did not elicit significant effects. Notably, the searchlight comprises regions in the ventral occipitotemporal pathway, implicated in concrete word processing (Bruffaerts et al., 2013; Liuzzi et al., 2015, 2017, 2019), but the RSA did not confirm the embedding of concrete word meaning. Traditionally, concrete concepts are considered to be organised in a hierarchical and categorical fashion (Crutch and Warrington, 2005, 2010).

    View all citing articles on Scopus
    View full text