Elsevier

Brain Research

Volume 1242, 25 November 2008, Pages 4-12
Brain Research

Review
Linking neurons to behavior in multisensory perception: A computational review

https://doi.org/10.1016/j.brainres.2008.04.082Get rights and content

Abstract

A large body of psychophysical and physiological findings has characterized how information is integrated across multiple senses. This work has focused on two major issues: how do we integrate information, and when do we integrate, i.e., how do we decide if two signals come from the same source or different sources. Recent studies suggest that humans and animals use Bayesian strategies to solve both problems. With regard to how to integrate, computational studies have also started to shed light on the neural basis of this Bayes-optimal computation, suggesting that, if neuronal variability is Poisson-like, a simple linear combination of population activity is all that is required for optimality. We review both sets of developments, which together lay out a path towards a complete neural theory of multisensory perception.

Section snippets

Optimal cue integration

When a common source is assumed, a systematic strategy to quantify cue combination is to introduce a small discrepancy (also called conflict, disparity, or incongruency) between the cues. The conflict must be small in order to not violate the common-source assumption. In such a paradigm, the percept (estimate of the stimulus) inferred from both cues presented together will lie somewhere in between the percepts inferred from each cue individually. The intuition is that higher weight will be

Optimal cue integration with neural populations

When studying how neuronal circuits implement near-optimal cue integration, an important fact to take into account is that the responses of cortical neurons are typically very variable (Compte et al., 2003, Dean, 1981, Holt et al., 1996, Tolhurst et al., 1982). Presenting the same stimulus repeatedly will give rise to many different population responses. A first sight, such variability is a nuisance that could compromise optimality. Recent work, however, has argued that the presence of

Multisensory integration in relation to other probabilistic computations

The appeal of probabilistic population codes, in which a population pattern of activity encodes the certainty about a stimulus, is that they are not limited to multisensory perception. Ecologically important tasks often require combining pieces of uncertain sensory information with each other or with prior information. In multisensory perception, cues from different modalities get combined. Examples in other domains include perceptual decision-making (combining information over time and

Comparison with physiology

Using this theoretical framework, it is now possible to link optimal behavior to neural population activity. Imagine recording with a multi-electrode array from a population of multisensory neurons in an awake, behaving animal engaged in optimal cue combination (as tested behaviorally). Then the theory predicts that the response of multisensory neurons when two cues are presented is equal to the sum of their responses when each cue is presented separately (this follows from the fact that the

Cue combination without forced integration

The second question concerns the number of sources, or multiplicity, of multisensory cues. When an auditory and a visual stimulus are observed, they could have either the same source or different sources. In cue conflict experiments, the disparity between the cues is usually kept small, so that the observer has no difficulty imagining that they originate from the same source (forced integration). However, in natural circumstances, large disparities in space, time, or feature space occur

Towards a complete theory of multisensory integration

Three decades ago, the question was posed whether there is a unified explanation for multisensory localization judgments under conflict (Warren, 1979). Behavioral theories of Bayes-optimal cue combination have brought us closer to this goal. Not only do they explain a wide range of existing data, they are also firmly rooted in a principled, probabilistic description of the purpose of multisensory perception, which is to increase precision if two cues have a common origin, but to keep cues with

References (89)

  • SteinB.E. et al.

    Neurons and behavior: the same rules of multisensory integration

    Brain Res

    (1988)
  • AnastasioT.J. et al.

    Using Bayes' rule to model multisensory enhancement in the superior colliculus

    Neural Computation

    (2000)
  • AverbeckB.B. et al.

    Neural correlations, population coding, and computation

    Nat. Rev. Neurosci

    (2006)
  • BanksM.S.

    What you see and hear is what you get.

    Curr. Biology

    (2004)
  • BattagliaP.W. et al.

    Bayesian integration of visual and auditory signals for spatial localization

    J. Opt. Soc. Am. A. Opt. Image Sci. Vis.

    (2003)
  • BeauchampM.S.

    Statistical criteria in fMRI studies of multisensory integration

    Neuroinformatics

    (2005)
  • Beierholm, U., Kording, K., Shams, L., Ma, W.J., 2007. Comparing Bayesian models for multisensory cue combination...
  • BellA.H. et al.

    Crossmodal integration in the primate superior colliculus underlying the preparation and initiation of saccadic eye movements

    J. Neurophysiol

    (2005)
  • BertelsonP. et al.

    Automatic visual bias of perceived auditory location

    Psychon. B. Rev

    (1998)
  • BertelsonP.

    Ventriloquism: a case of crossmodal perceptual grouping

  • BregmanA.S.

    Auditory Scene Analysis: The Perceptual Organization of Sound, Vol.

    (1990)
  • BrescianiJ.P. et al.

    Feeling what you hear: auditory signals can modulate tactile tap perception.

    Exp Brain Res.

    (2005)
  • BrescianiJ.P. et al.

    Vision and touch are automatically integrated for the perception of sequences of events

    J Vis

    (2006)
  • ChoeC.S. et al.

    The “ventriloquist effect”: visual dominance or response bias?

    Perception and Psychophysics

    (1975)
  • ClarkJ.J. et al.

    Data Fusion for Sensory Information Processing Systems, Vol.

    (1990)
  • ColoniusH. et al.

    A two-stage model for visual–auditory interaction in saccadic latencies

    Percept Psychophys

    (2001)
  • ColoniusH. et al.

    Multisensory interaction in saccadic reaction time: a time-window-of-integration model

    J Cogn Neurosci

    (2004)
  • CompteA. et al.

    Temporally irregular mnemonic persistent activity in prefrontal neurons of monkeys during a delayed response task

    J Neurophys

    (2003)
  • CorneilB.D. et al.

    Auditory–visual interactions subserving goal-directed saccades in a complex scene

    J Neurophysiol

    (2002)
  • De GelderB. et al.

    The perception of emotion by ear and by eye

    Cogn. Emot

    (2000)
  • DeanA.F.

    The variability of discharge of simple cells in the cat striate cortex

    Exp Brain Res

    (1981)
  • DeneveS. et al.

    Reading population codes: a neural implementation of ideal observers

    Nature Neuroscience

    (1999)
  • DeneveS. et al.

    Efficient computation and cue integration with noisy population codes

    Nature Neuroscience

    (2001)
  • DiederichA. et al.

    Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time

    Percept Psychophys

    (2004)
  • DiederichA. et al.

    Modeling spatial effects in visual-tactile saccadic reaction time

    Percept Psychophys

    (2007)
  • ElderJ.H. et al.

    Ecological statistics of Gestalt laws for the perceptual organization of contours

    J. Vision

    (2002)
  • ErnstM.O. et al.

    Humans integrate visual and haptic information in a statistically optimal fashion

    Nature

    (2002)
  • ErnstM.O.

    Learning to integrate arbitrary signals from vision and touch

    Journal of Vision

    (2007)
  • FeldmanJ.

    Bayesian contour integration

    Percept. Psychophys

    (2001)
  • FengA.S. et al.

    Neural basis of hearing in real-world situations

    Annu. Rev. Psychol

    (2000)
  • FoldiakP.

    The ‘ideal homunculus’: statistical inference from neural population responses.

  • GurM. et al.

    Response variability of neurons in primary visual cortex (V1) of alert monkeys

    J. Neurosci

    (1997)
  • HatfieldG.C.
  • HoltG.R. et al.

    Comparison of discharge variability in vitro and in vivo in cat visual cortex neurons.

    J. Neurophys.

    (1996)
  • Cited by (66)

    • A simple vector-like law for perceptual information combination is also followed by a class of cortical multisensory bimodal neurons

      2021, iScience
      Citation Excerpt :

      Because we have much data on neural variability and no data on neural Bayesian priors, the variant of Bayesian theory which is best suited to modeling spike count data is maximum likelihood estimation (MLE), which has widely been applied to sensory data (see Ernst, 2012 and Ma and Pouget, 2008 for reviews) and which is equivalent to a full Bayesian model for relatively flat priors. In the past, it has been suggested that bimodal neurons – especially superadditive bimodal neurons (which make up a substantial minority of our data) may be poor candidates for MLE (Ma and Pouget, 2008; Beck et al., 2008) but prior sensible results with MLE for visual-vestibular bimodal neurons (Morgan et al., 2008; Gu et al., 2008; Fetsch et al., 2010; Angelaki et al., 2012) suggest that it might be interesting to try. MLE makes two strong predictions about auditory-visual responses and their variability.

    • 6.30 - Multisensory Integration for Self-Motion Perception

      2020, The Senses: A Comprehensive Reference: Volume 1-7, Second Edition
    View all citing articles on Scopus
    View full text