Multisensory contributions to low-level, ‘unisensory’ processing
Introduction
Recent studies in both monkey and human subjects have provided evidence for multisensory convergence at low-level, putatively unisensory, stages of the sensory cortical pathways [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11•]. For example, somatosensory responses can be observed in auditory belt cortical regions (i.e. at the second level of auditory processing [4]), and eye position input modulates auditory responses even at the primary cortical (A1) level [12, 13]. Parallel findings have emerged in carnivores [12, 13, 14]. Most dramatically, two laboratories [15, 16] have shown anatomical interconnections between low-level visual and low-level auditory areas, which include the primary cortices (V1 and A1), and two others have shown that eye position can affect the gain of auditory responses in A1 [17, 18]. Recent reviews [9, 19, 20] have highlighted the fact that low-level (early) multisensory convergence is paradoxical from a hierarchical sensory perspective, and its functions are not yet clear.
This review focuses on low-level multisensory convergence in the primate auditory system. First, we review the neurophysiological evidence in this area. Second, we discuss the potential anatomical sources of non-auditory input, and the types of projections used (i.e. feedforward, feedback, lateral). Finally, we consider the functional implications of early multisensory integration in the context of the hierarchical model of auditory processing. To avoid confusion, we will use the term ‘low-level’ to refer to the anatomical stage at which a multisensory process is observed, reserving the term ‘early’ for reference to the time domain.
Section snippets
Visual and somatosensory responses in auditory cortex
Studies using event related potentials (ERPs) in humans have demonstrated short latency audio–visual [3, 7] and audio–somatosensory [21, 22] interactions, and have raised the possibility that these interactions occur in auditory cortices of the superior temporal plane. Localization of multisensory interactions within the superior temporal plane is independently supported by findings from other brain imaging techniques that have better anatomical resolution, including magnetoencephalography
Anatomical mechanisms of multisensory convergence in auditory cortex
In considering possible sources of somatosensory input to posterior auditory cortex, it is important to recognize that the anatomical inputs mediating electrically-evoked, median nerve responses [4, 32] and those mediating cutaneous, proprioceptive and vibratory inputs [8] might not be of uniform type. In fact, we think that the electrically-evoked, median nerve responses might be mediated by subcortical inputs from nonspecific, extralemniscal systems, whereas the cutaneous, proprioceptive and
Functional implications of multisensory convergence in low-level cortical processing
Assuming that activity in auditory cortex generally corresponds to a perceptual experience of something heard, a probable function of a converging visual or somatosensory input would be to enhance auditory analysis of that stimulus. How does auditory processing gain from non-auditory input? We consider two leading possibilities.
The first possibility is that somatosensory and visual inputs, due to their greater spatial precision, might support auditory spatial localization. This is in line with
Conclusions: low-level multisensory convergence and ‘the sensory hierarchy’
Suborning of activity in unisensory cortex by input from another sensory modality is an intriguing phenomenon that presents a paradox for hierarchical models of sensory system organization [33, 51, 52]. On the one hand, it appears that in multisensory processing, as in sensory contextual processing [53], and in attentional modulation of processing [54, 55, 56, 57], that recruitment of low level sensory areas into the cognitive–perceptual process is partially attributable to feedback-dependent
References and recommended reading
Papers of particular interest, published within the annual period of review, have been highlighted as:
• of special interest
•• of outstanding interest
Acknowledgements
Sincere thanks to P Lakatos, J Smiley, S Molholm and T Hackett, for material assistance and crucial critique and discussion. This work has been supported in part by grants from the National Institute of Mental Health MH61989 and MH65350.
References (58)
- et al.
Vibration-induced auditory cortex activation in a congenitally deaf adult
Curr Biol
(1998) - et al.
The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex
Brain Res Cogn Brain Res
(2002) - et al.
Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study
Brain Res Cogn Brain Res
(2002) Cortico-cortical connectivity and the architecture of cross-modal circuits
- et al.
Timing and laminar profile of eye position effects on auditory responses in primate auditory cortex
J Neurophysiol
(2004) - et al.
Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment
Cereb Cortex
(2004) - et al.
Activation of the human posterior parietal and temperopariatal cortices during audiotactile interaction
Neuroimage
(2003) - et al.
Activation of auditory cortex during silent lipreading
Science
(1997) - et al.
Functional properties of neurons in the temperopariatal association area of the awake monkey
Exp Brain Res
(1980) - et al.
Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex
Nat Neurosci
(1999)
Viewpoint: The core and matrix of thalamic organization
Neuroscience
The thalamic matrix and thalamocortical synchrony
Trends Neurosci
Convergent and complementary projections of the caudal paralaminar thalamic nuclei to rat temporal and insular cortex
Cereb Cortex
Intermodal selective attention in monkeys II: Physiologic mechanisms of modulation
Cereb Cortex
Response amplification in sensory-specific cortices during crossmodal binding
Neuroreport
Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study
J Cogn Neurosci
Somatosensory input to auditory association cortex in the macaque monkey
J Neurophysiol
Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study
J Neurophysiol
Auditory cortical neurons respond to somatosensory input
J Neurosci
Multisensory convergence in early cortical processing
Human-simian correspondence in the early cortical processing of multisensory cues
Cognitive Processing
Primary auditory cortex driven by visual speech: an fMRI study at 3T
Neuroreport
On the neuronal basis for multisensory convergence: a brief overview
Brain Res Cogn Brain Res
Anatomical evidence of multimodal integration in primate striate cortex
J Neurosci
Eye position affects activity in primary auditory cortex of primates
Curr Biol
Defining the neural bases of visual selective attention: conceptual and empirical issues
Int J Neurosci
The case for a feedforward component in multisensory integration mechanisms
Neuroreport
Cited by (393)
Multimodal fusion for sensorimotor control in steering angle prediction
2023, Engineering Applications of Artificial Intelligence