Models and processes of multisensory cue combination
Introduction
To make sense of a world that is noisy and ambiguous, neural systems combine information across sensory modalities to create unified and stable percepts. Numerous examples highlight the vital role of this process. When driving, we decide whether it is safe to change lanes based on a combination of sights and sounds, our perceived speed, and the force applied to the gas pedal. To better comprehend what someone is saying, we often look at their lips while listening to them speak. If you tilt your head to the side, the scene does not appear rotated because information from the inner ear is used to stabilize your visual perception of the world.
Because the brain often integrates the senses seamlessly, it is easy to overlook the complexities of multisensory cue combination. When presented with two sensory signals (say, light and sound), the brain must determine if they have a common source, reconcile differences in the reference frames in which they are encoded, and integrate information across time to form a coherent percept (Figure 1a). In this review, we discuss how information is combined across senses and examine how theoretical and computational neuroscience has informed our understanding of the neural underpinnings of multisensory cue combination.
Section snippets
Bayesian cue integration
Because sensory information is noisy and subject to ambiguity, we must infer the state of the world [1]. To improve this inference, information from different senses is combined through multisensory integration. Behavioral studies suggest that sensory signals are often combined in a Bayes-optimal (or nearly optimal) fashion [2, 3, 4••, 5•, 6•] to create a probability distribution over the range of possible stimuli that could have given rise to the signals. This process is probabilistic in the
A theory of how neurons implement multisensory integration
The behavioral observation that cue integration is probabilistic suggests that the brain may directly encode the reliability of sensory information. This led to the investigation of how the brain can simultaneously represent multiple pieces of sensory information along with their reliabilities, and combine them optimally to implement Bayesian cue integration [14].
An intriguing possibility is that this is achieved by populations of neurons whose combined activity describes the likelihood of a
Reference frame transformations
In primates, the posterior parietal cortex is an important locus of multisensory cue combination. Individual parietal neurons often encode information from multiple senses; for example, neurons in the ventral intraparietal area (VIP) can respond to visual, vestibular, tactile, and auditory stimuli [27, 28, 29, 30]. Considering that different sensory systems encode information relative to different egocentric reference frames (e.g., the eyes, head, or body), an important question to ask is: how
Decision making and speed-accuracy trade-off
In many studies, the dynamics of the decision process are hidden because subjects only report a final percept. A common approach to studying how a decision is formed is to use a reaction-time paradigm, in which the subjects control when the decision is reported. Previous work using this paradigm showed that observers make trade-offs between speed and accuracy [38] and that more reliable evidence leads to faster decisions [39], suggesting that perceptual evidence is accumulated over time until a
Development and calibration of multisensory integration
Although Bayesian multisensory integration appears normative in adults, children are far from optimal. Instead, one sense dominates childrens’ judgments, suggesting that the brain may forgo multisensory integration while it is learning to calibrate sensory systems relative to each other [44, 45•, 46]. Consider, for example, the use of vision and touch to perceive an object. Recent studies have shown that children with congenital visual deficits have an impaired ability to determine the object's
Conclusions
In this review we discussed several key components of multisensory cue combination, explored our understanding of each at the behavioral and neural levels, and examined a theoretical framework describing how single neurons might combine sensory information. However, we are far from fully understanding the complexities of how information from different senses is combined. For example, while several studies have considered the influence of naturally occurring priors on perception [7, 56, 57, 58],
References and recommended reading
Papers of particular interest, published within the period of review, have been highlighted as:
• of special interest
•• of outstanding interest
Acknowledgements
We thank Greg DeAngelis, Eliana Klier, Wei Ji Ma, and Adhira Sunkara for their comments on the manuscript. This work was supported by NIH grants T32EY007001 (R.L.S) as well as EY019087 and EY022538 (D.E.A.).
References (62)
- et al.
The ventriloquist effect results from near-optimal bimodal integration
Curr Biol
(2004) - et al.
Probabilistic interpretation of population codes
Neural Comput
(1998) - et al.
A fast and simple population code for orientation in primate V1
J Neurosci
(2012) - et al.
Normalization is a general neural mechanism for context-dependent decision making
Proc Natl Acad Sci U S A
(2013) - et al.
Eye-centered representation of optic flow tuning in the ventral intraparietal area
J Neurosci
(2013) - et al.
Probabilistic population codes for Bayesian decision making
Neuron
(2008) - et al.
Multisensory integration of natural odors and sounds in the auditory cortex
Neuron
(2011) - et al.
Early experience determines how the senses will interact
J Neurophysiol
(2007) - et al.
Bayesian cue integration as a developmental outcome of reward mediated learning
PLoS ONE
(2011) - et al.
Noise characteristics and prior expectations in human visual speed perception
Nat Neurosci
(2006)
Music can elicit a visual motion aftereffect
Atten Percept Psychophys
Touch influences visual perception with a tight orientation-tuning
PLOS One
Perception as Bayesian inference
Humans integrate visual and haptic information in a statistically optimal fashion
Nature
Neural correlates of reliability-based cue weighting during multisensory integration
Nat Neurosci
Multisensory decision-making in rats and humans
J Neurosci
Dynamic weighting of multisensory stimuli shapes decision-making in rats and humans
J Vis
The functional significance of velocity storage and its dependence on gravity
Exp Brain Res
Computation of linear acceleration through an internal model in the macaque cerebellum
Nat Neurosci
Neural representation of gravity in the macaque vestibulocerebellum
Neuron
Organizing probabilistic models of perception
Trends Cogn Sci
Neural correlates of multisensory cue integration in macaque MSTd
Nat Neurosci
Causal inference in multisensory perception
PLoS ONE
Causal inference in perception
Trends Cogn Sci
Bayesian inference with probabilistic population codes
Nat Neurosci
Correlated neuronal discharge rate and its implications for psychophysical performance
Nature
Perceptual learning reduces interneuronal correlations in macaque visual cortex
Neuron
High response reliability of neurons in primary visual cortex (V1) of alert, trained monkeys
Cereb Cortex
Decoding the activity of neuronal populations in macaque primary visual cortex
Nat Neurosci
A normalization model of multisensory integration
Nat Neurosci
Normalization as a canonical neural computation
Nat Rev Neurosci
Cited by (85)
Hierarchical computation of 3D motion across macaque areas MT and FST
2023, Cell ReportsChildren's stories and multisensory engagement: Insights from a cultural probes study
2022, International Journal of Educational ResearchFrom Multisensory Integration to Multisensory Decision-Making
2024, Advances in Experimental Medicine and Biology
- 1
Equal author contribution.