Cross-modal representation of spoken and written word meaning in left pars triangularis
Introduction
According to one of the most influential contemporary language models, the neural network underlying speech processing can be divided into a dorsal and a ventral stream (Hickok and Poeppel, 2004, Ueno et al., 2011). Both streams project into the inferior frontal gyrus (IFG) which has a crucial role in the interaction of these two streams (Saur et al., 2008, Rijntjes et al., 2012, Hamzei et al., 2016). The most natural word input modalities are spoken words (Chafe and Tannen, 1987) but in many cultures writing can convey meaning efficiently. Given the similarity in the meaningful messages conveyed, the path connecting written word input with word meaning presumably converges with that for auditory input at a given stage. The cognitive and neurobiological architecture of this confluence between written and spoken word input has been of longstanding interest to evolutionary and developmental neuroscience (Chafe and Tannen, 1987), neuropsychology (Allport and Funnell, 1981) and, more recently, functional imaging of the intact brain (Booth et al., 2002, Chee et al., 1999, Constable et al., 2004, Gold et al., 2005, Homae, 2002, Spitsyna et al., 2006, Wagner et al., 2001). The study of the differences and commonalities in processing of meaning between different input-modalities (written or spoken words, pictures, etc.) should not be confounded with the study of the effect of type of attributes of concrete entities (e.g. shape, sound, etc.) (Vandenbulcke et al., 2006, Huth et al., 2016) nor with the study of the inner format of semantic representations (Caramazza et al., 1990, Barsalou, 2016).
fMRI activations during semantic processing that are common for written and spoken words (Booth et al., 2002, Chee et al., 1999, Constable et al., 2004, Gold et al., 2005, Homae, 2002, Spitsyna et al., 2006, Wagner et al., 2001) (for review see Binder et al., 2009) can arise for various reasons. Domain-general processes (e.g. common selection, Thompson-Schill et al., 1997, or control processes, Gold et al., 2005) may operate on written and spoken word meaning that is represented at a distance (Gold et al., 2005, Hagoort, 2005). Or written words may be sounded out internally and the phonological operations associated with this process may give rise to apparent commonality with spoken word input. Thirdly, neuronal populations may code for the meaningful content of the words independent of the input modality in which the words were originally presented. A classical conjunction univariate analysis might reveal conjoint activation for these different reasons. Representational similarity analysis (RSA) (Kriegeskorte et al., 2008, Fairhall and Caramazza, 2013, Devereux et al., 2013) provides an opportunity to directly test the representational content and its dependence on input-modality. Depending on the behavioral matrix to which the fMRI matrix is compared and the extent of the stimulus set, RSA can reach a level of cognitive specificity and item-by-item granularity that cannot be attained by previous approaches to cross-modal processing (Homae, 2002, Kircher et al., 2009, Sass et al., 2009, Simanova et al., 2014). Among the cross-modal RSA studies, the majority used written words together with pictures (Devereux et al., 2013). An advantage of using words only is that it avoids the perceptual confound induced by the covariance between visual characteristics of pictures and their meaning (Fernandino et al., 2015): Compared to pictures, the relation between word form and meaning is far more arbitrary.
In the current study, we determined RSA effects of semantic similarity for cross-modal pairs, written versus spoken concrete nouns. First, a set of regions was defined based on a univariate analysis using an explicit semantic task performed with written and with spoken words compared to a lower-level control condition with consonant letter strings and rotated spectrograms, respectively. In a subsequent independent experiment, we determined within this set of regions whether semantic similarity between pairs of words is reflected in the activity pattern despite differences in word input-modality, spoken versus written. We also ascertained that the similarity in activity patterns could not be explained by phonological or orthographic similarity between words. In order to obtain a more complete picture of how crossmodal semantic similarity effects relate anatomically to input-modality specific processing pathways, we additionally searched for semantic similarity effects within the written or the spoken word processing pathway.
Section snippets
Participants
Eighteen subjects (12 women, 6 men), between 18 and 28 years, participated in a first fMRI experiment (univariate analysis). Twenty different subjects (14 women, 6 men), between 18 and 28 years, participated in a second, independent experiment, which was optimized for RSA (see below). All subjects were native Dutch speakers, right-handed, free of neurological or psychiatric history and had no hearing impairment. There was no overlap between the two subject groups. All the procedures were
Behavioral analysis
In the VOI defining experiment, the main effect of task on reaction times was significant (F(1,17)=36.147; P=0.00001) as was the main effect of input-modality (F(1,17)=223.68; P=0.00000), without interaction effect between the two factors (F(1,17)=0.00046; P=0.98315). Responses were significantly slower during the experimental than during the control condition and during the auditory than during the written conditions (Table 1). The effect of input modality on reaction times was confirmed in
Discussion
Previous fMRI studies using univariate analysis (Booth et al., 2002, Chee et al., 1999, Constable et al., 2004, Gold et al., 2005, Homae, 2002, Spitsyna et al., 2006, Vandenberghe et al., 1996, Wagner et al., 2001) have implicated BA45 in semantic processing. In the current study, using RSA we demonstrate that BA45 codes for the semantic relationships between entities and makes abstraction of the exact input modality in which these entities were presented. This novel finding provides insight in
Acknowledgments
R.V. is a Senior Clinical Investigator of the Research Foundation Flanders (FWO). R.B. is a postdoctoral fellow of the Research Foundation Flanders (FWO). Funded by Federaal Wetenschapsbeleid (Belspo 7/11), FWO (Grant no. G0925.15) and KU Leuven (OT/12/097).
References (95)
- et al.
Analysis of neural mechanisms underlying verbal fluency in cytoarchitectonically defined stereotaxic space—the roles of Brodmann areas 44 and 45
Neuroimage
(2004) The conscious access hypothesis: origins and recent evidence
Trends Cogn. Sci.
(2002)- et al.
The neurobiology of semantic memory
Trends Cogn. Sci.
(2011) - et al.
Heteromodal conceptual processing in the angular gyrus
NeuroImage
(2013) - et al.
Right fusiform response patterns reflect visual object identity rather than semantic similarity
NeuroImage
(2013) - et al.
Processing of visually presented sentences in Mandarin and English studied with fMRI
Neuron
(1999) - et al.
Sentence complexity and input modality effects in sentence comprehension, an fMRI study
NeuroImage
(2004) - et al.
Common regions of the human frontal lobe recruited by diverse cognitive demands
Trends Neurosci.
(2000) - et al.
Testing anatomically specified hypotheses in functional imaging using cytoarchitectonic maps
Neuroimage
(2006) - et al.
Assignment of functional activations to probabilistic cytoarchitectonic areas revisited
Neuroimage
(2007)
A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data
Neuroimage
Towards a neural basis of auditory sentence processing
Trends Cogn. Sci.
The language network
Curr. Opin. Neurobiol.
The language skeleton after dissecting meaning, a functional segregation within Broca's Area
NeuroImage
On Broca, brain, and binding, a new framework
Trends Cogn. Sci.
How concepts are encoded in the human brain, a modality independent, category-based cortical organization of semantic knowledge
NeuroImage
The role of the left Brodmann's areas 44 and 45 in reading words and pseudowords
Cogn. Brain Res.
Dorsal and ventral streams, a framework for understanding aspects of the functional anatomy of language
Cognition
From perception to sentence comprehension, the convergence of auditory and visual information of language in the left inferior frontal cortex
NeuroImage
A continuous semantic space describes the representation of thousands of object and action categories across the human brain
Neuron
A procedure for identifying regions preferentially activated by attention to semantic and phonological relations using functional magnetic resonance imaging
Neuropsychologia
Encoding and decoding in fMRI
Neuroimage
Functional specialization for semantic and phonological processing in the left inferior prefrontal cortex
Neuroimage
Combining spatial extent and peak intensity to test for activations in functional imaging
NeuroImage
Lion-tiger-stripes, neural correlates of indirect semantic priming across processing modalities
Neuroimage
Combining functional and anatomical connectivity reveals brain networks for auditory language comprehension
NeuroImage
Dynamic coding for cognitive control in prefrontal cortex
Neuron
Lichtheim 2, synthesizing aphasia and the neural basis of language in a neurocomputational model of the dual dorsal-ventral language pathways
Neuron
The amodal system for conscious word and picture identification in the absence of a semantic task
NeuroImage
The associative-semantic network for words and pictures, effective connectivity and graph analysis
Brain Lang.
Recovering meaning
Neuron
How the ventral pathway got lost-and what its recovery might mean
Brain Lang.
Intrinsic functional network architecture of human semantic processing, modules and hubs
NeuroImage
Components of the mental lexicon
Philos. Trans. R. Soc. B: Biol. Sci.
Broca's region, novel organizational principles and multiple receptor mapping
PLoS Biol.
Broca's region revisited, cytoarchitecture and intersubject variability
J. Comp. Neurol.
A Cognitive Theory of Consciousness
The CELEX Lexical Database
On staying grounded and avoiding quixotic dead ends
Psychon. Bull. Rev.
Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies
Cereb. Cortex
Where is the anterior temporal lobe and what does it do?
J. Neurosci.
Modality independence of word comprehension
Hum. Brain Mapp.
Similarity of fMRI activity patterns in left perirhinal cortex reflects semantic similarity between words
J. Neurosci.
A neural theory of visual attention, bridging cognition and neurophysiology
Psychol. Rev.
The multiple semantics hypothesis, multiple confusions?
Cogn. Neuropsychol.
The relation between written and spoken language
Annu. Rev. Anthropol.
Cited by (36)
Representations of conceptual information during automatic and active semantic access
2021, NeuropsychologiaSimilar activation patterns in the bilateral dorsal inferior frontal gyrus for monolingual and bilingual contexts in second language production
2021, NeuropsychologiaCitation Excerpt :Our interpretation of cross-context PS in the bilateral pars opercularis as a reflection of similar phonological information was supported by two additional analyses. First, the bilateral ventral IFGs (i.e., pars triangularis) as key regions for semantic processing (Binder et al., 2009; Fridriksson et al., 2015; Liuzzi et al., 2017; Poldrack et al., 1999) did not show significant cross-context PS in L2 naming. Second, the bilateral pars opercularis did not show significant cross-language PS for the same items relative to different items.
Representation of associative and affective semantic similarity of abstract words in the lateral temporal perisylvian language regions
2020, NeuroImageCitation Excerpt :Liuzzi et al. (2019) demonstrated that the associative model is well-suited for the representation of concrete words, but here, concrete words alone did not elicit significant effects. Notably, the searchlight comprises regions in the ventral occipitotemporal pathway, implicated in concrete word processing (Bruffaerts et al., 2013; Liuzzi et al., 2015, 2017, 2019), but the RSA did not confirm the embedding of concrete word meaning. Traditionally, concrete concepts are considered to be organised in a hierarchical and categorical fashion (Crutch and Warrington, 2005, 2010).