Models and processes of multisensory cue combination

https://doi.org/10.1016/j.conb.2013.11.008Get rights and content

Highlights

  • The brain can optimally integrate information across sensory systems.

  • Probabilistic population codes solve many problems in multisensory cue combination.

  • Multisensory neurons can represent different senses in different reference frames.

  • Sensory evidence is temporally weighted based on its moment-by-moment reliability.

  • Multisensory integration requires cross-sensory calibration throughout life.

Fundamental to our perception of a unified and stable environment is the capacity to combine information across the senses. Although this process appears seamless as an adult, the brain's ability to successfully perform multisensory cue combination takes years to develop and relies on a number of complex processes including cue integration, cue calibration, causal inference, and reference frame transformations. Further complexities exist because multisensory cue combination is implemented across time by populations of noisy neurons. In this review, we discuss recent behavioral studies exploring how the brain combines information from different sensory systems, neurophysiological studies relating behavior to neuronal activity, and a theory of neural sensory encoding that can account for many of these experimental findings.

Introduction

To make sense of a world that is noisy and ambiguous, neural systems combine information across sensory modalities to create unified and stable percepts. Numerous examples highlight the vital role of this process. When driving, we decide whether it is safe to change lanes based on a combination of sights and sounds, our perceived speed, and the force applied to the gas pedal. To better comprehend what someone is saying, we often look at their lips while listening to them speak. If you tilt your head to the side, the scene does not appear rotated because information from the inner ear is used to stabilize your visual perception of the world.

Because the brain often integrates the senses seamlessly, it is easy to overlook the complexities of multisensory cue combination. When presented with two sensory signals (say, light and sound), the brain must determine if they have a common source, reconcile differences in the reference frames in which they are encoded, and integrate information across time to form a coherent percept (Figure 1a). In this review, we discuss how information is combined across senses and examine how theoretical and computational neuroscience has informed our understanding of the neural underpinnings of multisensory cue combination.

Section snippets

Bayesian cue integration

Because sensory information is noisy and subject to ambiguity, we must infer the state of the world [1]. To improve this inference, information from different senses is combined through multisensory integration. Behavioral studies suggest that sensory signals are often combined in a Bayes-optimal (or nearly optimal) fashion [2, 3, 4••, 5•, 6•] to create a probability distribution over the range of possible stimuli that could have given rise to the signals. This process is probabilistic in the

A theory of how neurons implement multisensory integration

The behavioral observation that cue integration is probabilistic suggests that the brain may directly encode the reliability of sensory information. This led to the investigation of how the brain can simultaneously represent multiple pieces of sensory information along with their reliabilities, and combine them optimally to implement Bayesian cue integration [14].

An intriguing possibility is that this is achieved by populations of neurons whose combined activity describes the likelihood of a

Reference frame transformations

In primates, the posterior parietal cortex is an important locus of multisensory cue combination. Individual parietal neurons often encode information from multiple senses; for example, neurons in the ventral intraparietal area (VIP) can respond to visual, vestibular, tactile, and auditory stimuli [27, 28, 29, 30]. Considering that different sensory systems encode information relative to different egocentric reference frames (e.g., the eyes, head, or body), an important question to ask is: how

Decision making and speed-accuracy trade-off

In many studies, the dynamics of the decision process are hidden because subjects only report a final percept. A common approach to studying how a decision is formed is to use a reaction-time paradigm, in which the subjects control when the decision is reported. Previous work using this paradigm showed that observers make trade-offs between speed and accuracy [38] and that more reliable evidence leads to faster decisions [39], suggesting that perceptual evidence is accumulated over time until a

Development and calibration of multisensory integration

Although Bayesian multisensory integration appears normative in adults, children are far from optimal. Instead, one sense dominates childrens’ judgments, suggesting that the brain may forgo multisensory integration while it is learning to calibrate sensory systems relative to each other [44, 45•, 46]. Consider, for example, the use of vision and touch to perceive an object. Recent studies have shown that children with congenital visual deficits have an impaired ability to determine the object's

Conclusions

In this review we discussed several key components of multisensory cue combination, explored our understanding of each at the behavioral and neural levels, and examined a theoretical framework describing how single neurons might combine sensory information. However, we are far from fully understanding the complexities of how information from different senses is combined. For example, while several studies have considered the influence of naturally occurring priors on perception [7, 56, 57, 58],

References and recommended reading

Papers of particular interest, published within the period of review, have been highlighted as:

  • • of special interest

  • •• of outstanding interest

Acknowledgements

We thank Greg DeAngelis, Eliana Klier, Wei Ji Ma, and Adhira Sunkara for their comments on the manuscript. This work was supported by NIH grants T32EY007001 (R.L.S) as well as EY019087 and EY022538 (D.E.A.).

References (62)

  • S.C. Hedger et al.

    Music can elicit a visual motion aftereffect

    Atten Percept Psychophys

    (2013)
  • O. van der Groen et al.

    Touch influences visual perception with a tight orientation-tuning

    PLOS One

    (2013)
  • D.C. Knill et al.

    Perception as Bayesian inference

    (1996)
  • M.O. Ernst et al.

    Humans integrate visual and haptic information in a statistically optimal fashion

    Nature

    (2002)
  • C.R. Fetsch et al.

    Neural correlates of reliability-based cue weighting during multisensory integration

    Nat Neurosci

    (2011)
  • D. Raposo et al.

    Multisensory decision-making in rats and humans

    J Neurosci

    (2012)
  • J.P. Sheppard et al.

    Dynamic weighting of multisensory stimuli shapes decision-making in rats and humans

    J Vis

    (2013)
  • J. Laurens et al.

    The functional significance of velocity storage and its dependence on gravity

    Exp Brain Res

    (2011)
  • J. Laurens et al.

    Computation of linear acceleration through an internal model in the macaque cerebellum

    Nat Neurosci

    (2013)
  • J. Laurens et al.

    Neural representation of gravity in the macaque vestibulocerebellum

    Neuron

    (2013)
  • W.J. Ma

    Organizing probabilistic models of perception

    Trends Cogn Sci

    (2012)
  • Y. Gu et al.

    Neural correlates of multisensory cue integration in macaque MSTd

    Nat Neurosci

    (2008)
  • K.P. Kording et al.

    Causal inference in multisensory perception

    PLoS ONE

    (2007)
  • L. Shams et al.

    Causal inference in perception

    Trends Cogn Sci

    (2010)
  • W.J. Ma et al.

    Bayesian inference with probabilistic population codes

    Nat Neurosci

    (2006)
  • E. Zohary et al.

    Correlated neuronal discharge rate and its implications for psychophysical performance

    Nature

    (1994)
  • Y. Gu et al.

    Perceptual learning reduces interneuronal correlations in macaque visual cortex

    Neuron

    (2011)
  • M. Gur et al.

    High response reliability of neurons in primary visual cortex (V1) of alert, trained monkeys

    Cereb Cortex

    (2006)
  • A.B. Graf et al.

    Decoding the activity of neuronal populations in macaque primary visual cortex

    Nat Neurosci

    (2011)
  • T. Ohshiro et al.

    A normalization model of multisensory integration

    Nat Neurosci

    (2011)
  • M. Carandini et al.

    Normalization as a canonical neural computation

    Nat Rev Neurosci

    (2011)
  • Cited by (85)

    • From Multisensory Integration to Multisensory Decision-Making

      2024, Advances in Experimental Medicine and Biology
    View all citing articles on Scopus
    1

    Equal author contribution.

    View full text