Current perspectives and methods in studying neural mechanisms of multisensory interactions

doi:10.1016/j.neubiorev.2011.04.015

Neuroscience & Biobehavioral Reviews

Volume 36, Issue 1, January 2012, Pages 111-133

https://doi.org/10.1016/j.neubiorev.2011.04.015 Get rights and content

Abstract

In the past decade neuroscience has witnessed major advances in the field of multisensory interactions. A large body of research has revealed several new types of cross-sensory interactions. In addition, multisensory interactions have been reported at temporal and spatial system levels previously thought of as strictly unimodal. We review the findings that have led to the current broad consensus that most, if not all, higher, as well as lower level neural processes are in some form multisensory. We continue by outlining the progress that has been made in identifying the functional significance of different types of interactions, for example, in subserving stimulus binding and enhancement of perceptual certainty. Finally, we provide a critical introduction to cutting edge methods from bayes optimal integration to multivoxel pattern analysis as applied to multisensory research at different system levels.

Highlights

► We Introduce concepts central to multisensory research. ► 5 classes of multisensory interactions are described. ► We portrait the spatio-temporal ubiquity of multisensory interaction in the brain. ► Likely functional architecture underlying multisensory interactions is discussed. ► Methods with great potential in advancing multisensory research are presented.

Section snippets

Multimodal or supramodal—conceptual clarification

One controversial concept in multisensory research is the precise definition of supramodal processing or supramodal brain activation. At a general level, supramodal brain activation appears to be caused by the stimulation of one sensory modality, or by the simultaneous stimulation of several modalities. Over the past 20 years, an extensive literature has accumulated in support of theories that propose supramodal brain areas (e.g., Eimer et al., 2003, Farah et al., 1989, Macaluso, 2010, Macaluso

Classes of multisensory interactions

As noted in Section 1.1, since the detailed characterization of multisensory responses of SC neurons in the cat by Meredith and Stein, 1986, Meredith and Stein, 1996 a number of different types of multisensory interactions have been identified. Here we summarise five broad classes of multisensory interactions that have been shown to date and discuss the role of the measurement technique in the observation of the different types of interactions.

Where and when do multisensory interactions take place in the brain?

In Section 1 we outlined five broad classes of multisensory interactions observed throughout the brain. The purpose of this section is to give a selective overview of where in the brain multisensory interactions take place and when these interactions arise. While it is our aim to show that possibly all regions of the brain are involved in multisensory processing, the list of brain sites is not intended to be exhaustive. Similarly, we focus on early temporal interactions between sensory

What is the functional architecture that supports crossmodal interactions?

In this section, we will discuss different hypotheses about hierarchical architectures underlying multisensory integration. Hierarchical systems are a central feature of modern theories about the brain as an inference machine (i.e., theories that state that the brain tries to predict its multisensory inputs). These ideas date back to the days of von Helmholtz (1867) and have emerged in several guises over the past century (e.g., as Analysis by Synthesis (Neisser, 1967) and Predictive Coding (

Research methods for multisensory research

In this section we highlight experimental methods as well as statistical analyses that are either new or have thus far not received adequate attention in the field and that are advancing our knowledge of multisensory processing. We will also outline approaches that, to our knowledge, have thus far not been applied in multisensory research, but which promise to provide new insights.

General conclusion

In this review we have focused on the neural basis of multisensory processing in the human and animal brain, foregrounding studies of multisensory attention against the background of more general multisensory interactions. Functional interplay between the senses occurs at all anatomical levels of the processing hierarchy and across a varied time course, indicating that the neural networks underlying multisensory interactions are heavily interconnected. With this interconnected view now

Acknowledgements

Preparation of this manuscript was supported by the Biotechnology and Biological Sciences Research Council (BBSRC; research grant BB/E020291/1; David Phillips Fellowship, CDC). We thank Sven Bestmann and Cyril Charron for helpful discussions regarding Bayesian inference, Suresh Muthukumaraswamy for his help in shaping the section on effective connectivity for M/EEG, and Lorraine Woods for her assistance with the art-work.

References (325)

D.E. Angelaki et al.
Multisensory integration: psychophysics, neurophysiology, and computation
Curr. Opin. Neurobiol.
(2009)
M.S. Beauchamp et al.
Integration of auditory and visual information about objects in superior temporal sulcus
Neuron
(2004)
M.S. Beauchamp et al.
Touch, sound and vision in human superior temporal sulcus
Neuroimage
(2008)
R.A. Berman et al.
Auditory and visual attention modulate motion processing in area MT+
Brain Res. Cogn. Brain Res.
(2002)
N. Bolognini et al.
Proprioceptive alignment of visual and somatosensory maps in the posterior parietal cortex
Curr. Biol.
(2007)
B. Bonath et al.
Neural basis of the ventriloquist illusion
Curr. Biol.
(2007)
F. Bremmer et al.
Polymodal motion processing in posterior parietal and premotor cortex: a human fMRI study strongly implies equivalencies between humans and monkeys
Neuron
(2001)
E. Budinger et al.
Multisensory processing via early cortical stages: Connections of the primary auditory cortical field with other sensory systems
Neuroscience
(2006)
D. Buelte et al.
The role of the anterior intraparietal sulcus in crossmodal processing of object features in humans: An rTMS study
Brain Res.
(2008)
L.R. Burnett et al.
Superior colliculus lesions preferentially disrupt multisensory orientation
Neuroscience
(2004)

G.A. Calvert et al.

Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex

Curr. Biol.

(2000)

C.D. Chambers et al.

Neurodisruption of selective attention: insights and implications

Trends Cogn. Sci.

(2005)

C.D. Chambers et al.

Parietal disruption impairs reflexive spatial attention within and between sensory modalities

Neuropsychologia

(2007)

C.D. Chambers et al.

Modality-specific control of strategic spatial attention in parietal cortex

Neuron

(2004)

S. Chikkerur et al.

What and where: a Bayesian inference theory of attention

Vision Res.

(2010)

E.H. Chudler et al.

Multisensory convergence and integration in the neostriatum and globus pallidus of the rat

Brain Res.

(1995)

Y.E. Cohen

Multimodal activity in the parietal cortex

Hearing Res.

(2009)

C.L. Colby

Action-oriented spatial reference frames in cortex

Neuron

(1998)

F. Collette et al.

Involvement of both prefrontal and inferior parietal cortex in dual-task performance

Brain Res. Cogn. Brain Res.

(2005)

M. Corbetta et al.

The reorienting system of the human brain: from environment to theory of mind

Neuron

(2008)

D.D. Cox et al.

Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex

Neuroimage

(2003)

O. David et al.

Dynamic causal modeling of evoked responses in EEG and MEG

Neuroimage

(2006)

J. Driver et al.

Concurrent brain-stimulation and neuroimaging for studies of cognition

Trends Cogn. Sci.

(2009)

J. Driver et al.

Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments

Neuron

(2008)

R. Dubner et al.

Response properties and receptive fields of cells in an anatomically defined region of the superior temporal sulcus in the monkey

Brain Res.

(1971)

M. Eimer et al.

Crossmodal links in endogenous and exogenous spatial attention: evidence from event-related brain potential studies

Neurosci. Biobehav. Rev.

(2001)

M. Eimer et al.

Shifts of attention in light and in darkness: an ERP study of supramodal attentional control and crossmodal links in spatial attention

Cogn. Brain Res.

(2003)

A.K. Engel et al.

Temporal binding and the neural correlates of sensory awareness

Trends Cogn. Sci.

(2001)

M.O. Ernst et al.

Merging the senses into a robust percept

Trends Cogn. Sci.

(2004)

M.J. Farah et al.

Parietal lobe mechanisms of spatial attention: modality-specific or supramodal?

Neuropsychologia

(1989)

J.J. Foxe et al.

Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping

Brain Res. Cogn. Brain Res.

(2000)

M. Franca et al.

Effects of theta burst stimulation protocols on phosphene threshold

Clin. Neurophysiol.

(2006)

S.H. Frey et al.

Cortical topography of human anterior intraparietal cortex active during visually guided grasping

Brain Res. Cogn. Brain Res.

(2005)

P. Fries

A mechanism for cognitive dynamics: neuronal communication through neuronal coherence

Trends Cogn. Sci.

(2005)

K. Friston

Functional integration and inference in the brain

Prog. Neurobiol.

(2002)

K. Friston

Learning and inference in the brain

Neural Netw.

(2003)

A.A. Ghazanfar et al.

Is neocortex essentially multisensory?

Trends Cogn. Sci.

(2006)

R.A. Adcock et al.

Functional neuroanatomy of executive processes involved in dual-task performance

Proc. Natl. Acad. Sci. U.S.A.

(2000)

B.L. Allman et al.

Do cross-modal projections always result in multisensory integration?

Cereb. Cortex

(2008)

B.L. Allman et al.

Not just for bimodal neurons anymore: the contribution of unimodal neurons to cortical multisensory processing

Brain Topogr.

(2009)

B.L. Allman et al.

Multisensory processing in “Unimodal” neurons: cross-modal subthreshold auditory effects in cat extrastriate visual cortex

J. Neurophysiol.

(2007)

J.C. Alvarado et al.

Cortex mediates multisensory but not unisensory integration in superior colliculus

J. Neurosci.

(2007)

A. Amedi et al.

Visuo-haptic object-related activation in the ventral visual pathway

Nat. Neurosci.

(2001)

A. Amedi et al.

Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex

Nat. Neurosci.

(2007)

R.A. Andersen et al.

Corticocortical connections of anatomically defined subdivisions within the inferior parietal lobe

J. Compar. Neurol.

(1990)

P. Anninos et al.

MEG evaluation of Parkinson's diseased patients after external magnetic stimulation

Acta Neurol. Belg.

(2007)

M. Avillac et al.

Multisensory integration in the ventral intraparietal area of the macaque monkey

J. Neurosci.

(2007)

B. Baier et al.

Cross-modal processing in early visual and auditory cortices depends on expected statistical relationship of multisensory information

J. Neurosci.

(2006)

J.S. Baizer et al.

Comparison of subcortical connections of inferior temporal and posterior parietal cortex in monkeys

Visual Neurosci.

(1993)

N.E. Barraclough et al.

Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions

J. Cogn. Neurosci.

(2005)

Cited by (92)

Multi-scale spatiotemporal attention network for neuron based motor imagery EEG classification
2024, Journal of Neuroscience Methods
In recent times, the expeditious expansion of Brain-Computer Interface (BCI) technology in neuroscience, which relies on electroencephalogram (EEG) signals associated with motor imagery, has yielded outcomes that rival conventional approaches, notably due to the triumph of deep learning. Nevertheless, the task of developing and training a comprehensive network to extract the underlying characteristics of motor imagining EEG data continues to pose challenges.
This paper presents a multi-scale spatiotemporal self-attention (SA) network model that relies on an attention mechanism. This model aims to classify motor imagination EEG signals into four classes (left hand, right hand, foot, tongue/rest) by considering the temporal and spatial properties of EEG. It is employed to autonomously allocate greater weights to channels linked to motor activity and lesser weights to channels not related to movement, thus choosing the most suitable channels. Neuron utilises parallel multi-scale Temporal Convolutional Network (TCN) layers to extract feature information in the temporal domain at various scales, effectively eliminating temporal domain noise.
The suggested model achieves accuracies of 79.26%, 85.90%, and 96.96% on the BCI competition datasets IV-2a, IV-2b, and HGD, respectively.
In terms of single-subject classification accuracy, this strategy demonstrates superior performance compared to existing methods.
The results indicate that the proposed strategy exhibits favourable performance, resilience, and transfer learning capabilities.
Beyond sense-specific processing: decoding texture in the brain from touch and sonified movement
2023, iScience
Texture, a fundamental object attribute, is perceived through multisensory information including touch and auditory cues. Coherent perceptions may rely on shared texture representations across different senses in the brain. To test this hypothesis, we delivered haptic textures coupled with a sound synthesizer to generate real-time textural sounds. Participants completed roughness estimation tasks with haptic, auditory, or bimodal cues in an MRI scanner. Somatosensory, auditory, and visual cortices were all activated during haptic and auditory exploration, challenging the traditional view that primary sensory cortices are sense-specific. Furthermore, audio-tactile integration was found in secondary somatosensory (S2) and primary auditory cortices. Multivariate analyses revealed shared spatial activity patterns in primary motor and somatosensory cortices, for discriminating texture across both modalities. This study indicates that primary areas and S2 have a versatile representation of multisensory textures, which has significant implications for how the brain processes multisensory cues to interact more efficiently with our environment.
Noise affects mate choice based on visual information via cross-sensory interference
2022, Environmental Pollution
Animal communication is often hampered by noise interference. Noise masking has primarily been studied in terms of its unimodal effect on sound information provision and use, while little is known about its cross-modal effect and how animals weigh unimodal and multimodal courtship cues in noisy environments. Here, we examined the cross-modal effects of background noise on female visual perception of mate choice and female preference for multimodal displays (sound + vocal sac) in a species of treefrog. We tested female mate choices using audio/video playbacks in the presence and absence of noise (white noise band-filtered to match or mismatch female sensitive hearing range, heterospecific chorus). Surprisingly, multimodal displays do not improve receiver performance in noise. The heterospecific chorus and white noise band-filtered to match female sensitive hearing ranges, significantly reduced female responses to the attractive visual stimuli in addition to directly impairing auditory information use. Meanwhile, the cross-modal impacts of background noise are influenced to some extent by whether the noise band matches female sensitive hearing range and the difficulty of distinguishing tasks. Our results add to the evidence for cross-modal effects of noise and are the first to demonstrate that background noise can disrupt female responses to visual information related to mate choice, which may reduce the communication efficiency of audiovisual signals in noisy environments and impose fitness consequences. This study has key ecological and evolutionary implications because it illustrates how noise influences mate choice in wildlife via cross-sensory interference, which is crucial in revealing the function and evolution of multimodal signals in noisy environments as well as informing evidence-based conservation strategies for forecasting and mitigating the multimodal impacts of noise interference on wildlife.
Age-related changes in brain functional networks under multisensory-guided hand movements assessed by the functional near – Infrared spectroscopy
2022, Neuroscience Letters
This study aims to explore the age-related effects of hand rehabilitation training under multisensory stimulation interaction on brain functional networks.
A multisensory stimulation training glove (MSTG) was designed to realize 3 sensory guidance modes, namely audio-visual guidance (AVG), visual guidance (VG) and no guidance (NG). This study recruited 20 older subjects as the experimental group and 22 young people as the control group. Functional near-infrared spectroscopy (fNIRS) was used to monitor haemoglobin concentration in the motor cortex (MC), prefrontal cortex (PFC), temporary lo be (TL) and occipital lobe (OL) under three different guidance stages, and further analysed the cortical activation and functional connectivity (FC).
Multisensory guidance stage showed more activation and higher FC in all subjects. The activated brain regions of the older subjects showed bilateral activation, which is consistent with the Hemispheric Asymmetry Reduction in Older Adults (HAROLD) model. In terms of brain region coordination, older people have a more balanced and denser functional network in the left and right hemispheres compared to younger people. Meanwhile, multisensory stimulation produced a positive training effect on the number of training and reaction time.
Audio-visual combined stimulation had a significant gain effect on hand training at different ages. However, older adults induce a wider range of cortical activations. At the same time, young and older people have different intercortical coordination networks. All these results provide theoretically and applied references for multisensory stimulation in the prevention and rehabilitation of ageing and brain neurological disorders.
Contribution of muscle proprioception to limb movement perception and proprioceptive decline with ageing
2021, Current Opinion in Physiology
Perception of self-body movements relies on multiple senses among which muscle proprioception plays a crucial role. However, estimating its specific contribution is challenging given the difficulty to stimulate the deep muscle spindles independently from the skin and other tissues. Recent advances in psychophysical and neuroimaging studies that allow targeting muscle proprioception reveal (i) its specific contribution with respect to touch and vision, (ii) the interplay between bimanual proprioceptive signals and (iii) the sensorimotor cerebral network associated with movement illusion. Moreover, they highlight in older adults a larger decline in muscle proprioception than touch, which goes with structural alteration of cortical gray matter and corpus callosum, and with increased inter-network connectivity. These findings underline the importance to address disruption of sense of movement for both diagnosis and rehabilitation.
Impact of multisensory learning on perceptual and lexical processing of unisensory Morse code
2021, Brain Research
Multisensory learning profits from stimulus congruency at different levels of processing. In the current study, we sought to investigate whether multisensory learning can potentially be based on high-level feature congruency (same meaning) without perceptual congruency (same time) and how this relates to changes in brain function and behaviour. 50 subjects learned to decode Morse code (MC) either in unisensory or different multisensory manners. During unisensory learning, the MC was trained as sequences of auditory trains. For low-level congruent (perceptual) multisensory learning, MC was applied as tactile stimulation to the left hand simultaneously to the auditory stimulation. In contrast, high-level congruent multisensory learning involved auditory training, followed by the production of MC sequences requiring motor actions and thereby excludes perceptual congruency. After learning, group differences were observed within three distinct brain regions while processing unisensory (auditory) MC. Both types of multisensory learning were associated with increased activation in the right inferior frontal gyrus. Multisensory low-level learning elicited additional activation in the somatosensory cortex, while multisensory high-level learners showed a reduced activation in the inferior parietal lobule, which is relevant for decoding MC. Furthermore, differences in brain function associated with multisensory learning was related to behavioural reaction times for both multisensory learning groups. Overall, our data support the idea that multisensory learning is potentially based on high-level features without perceptual congruency. Furthermore, learning of multisensory associations involves neural representations of stimulus features involved in learning, but also share common brain activation (i.e. the right IFG), which seems to serve as a site of multisensory integration.

View all citing articles on Scopus

View full text

ReviewCurrent perspectives and methods in studying neural mechanisms of multisensory interactions

Abstract

Highlights

Section snippets

Multimodal or supramodal—conceptual clarification

Classes of multisensory interactions

Where and when do multisensory interactions take place in the brain?

What is the functional architecture that supports crossmodal interactions?

Research methods for multisensory research

General conclusion

Acknowledgements

Curr. Opin. Neurobiol.

Neuron

Neuroimage

Brain Res. Cogn. Brain Res.

Curr. Biol.

Curr. Biol.

Neuron

Neuroscience

Brain Res.

Neuroscience

Curr. Biol.

Trends Cogn. Sci.

Neuropsychologia

Neuron

Vision Res.

Brain Res.

Hearing Res.

Neuron

Brain Res. Cogn. Brain Res.

Neuron

Neuroimage

Neuroimage

Trends Cogn. Sci.

Neuron

Brain Res.

Neurosci. Biobehav. Rev.

Cogn. Brain Res.

Trends Cogn. Sci.

Trends Cogn. Sci.

Neuropsychologia

Brain Res. Cogn. Brain Res.

Clin. Neurophysiol.

Brain Res. Cogn. Brain Res.

Trends Cogn. Sci.

Prog. Neurobiol.

Neural Netw.

Trends Cogn. Sci.

Functional neuroanatomy of executive processes involved in dual-task performance

Proc. Natl. Acad. Sci. U.S.A.

Do cross-modal projections always result in multisensory integration?

Cereb. Cortex

Not just for bimodal neurons anymore: the contribution of unimodal neurons to cortical multisensory processing

Brain Topogr.

Multisensory processing in “Unimodal” neurons: cross-modal subthreshold auditory effects in cat extrastriate visual cortex

J. Neurophysiol.

Cortex mediates multisensory but not unisensory integration in superior colliculus

J. Neurosci.

Visuo-haptic object-related activation in the ventral visual pathway

Nat. Neurosci.

Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex

Nat. Neurosci.

Corticocortical connections of anatomically defined subdivisions within the inferior parietal lobe

J. Compar. Neurol.

MEG evaluation of Parkinson's diseased patients after external magnetic stimulation

Acta Neurol. Belg.

Multisensory integration in the ventral intraparietal area of the macaque monkey

J. Neurosci.

Cross-modal processing in early visual and auditory cortices depends on expected statistical relationship of multisensory information

J. Neurosci.

Comparison of subcortical connections of inferior temporal and posterior parietal cortex in monkeys

Visual Neurosci.

Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions

J. Cogn. Neurosci.

Review
Current perspectives and methods in studying neural mechanisms of multisensory interactions