Skip to main content
Log in

Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Single-neuron studies provide a foundation for understanding many facets of multisensory integration. These studies have used a variety of criteria for identifying and quantifying multisensory integration. While a number of techniques have been used, an explicit discussion of the assumptions, criteria, and analytical methods traditionally used to define the principles of multisensory integration is lacking. This was not problematic when the field was small, but with rapid growth a number of alternative techniques and models have been introduced, each with its own criteria and sets of implicit assumptions to define and characterize what is thought to be the same phenomenon. The potential for misconception prompted this reexamination of traditional approaches in order to clarify their underlying assumptions and analytic techniques. The objective here is to review and discuss traditional quantitative methods advanced in the study of single-neuron physiology in order to appreciate the process of multisensory integration and its impact.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14:257–262

    PubMed  CAS  Google Scholar 

  • Alvarado JC, Stanford TR, Vaughan JW, Stein BE (2007a) Cortex mediates multisensory but not unisensory integration in superior colliculus. J Neurosci 27:12775–12786

    Article  PubMed  CAS  Google Scholar 

  • Alvarado JC, Vaughan JW, Stanford TR, Stein BE (2007b) Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J Neurophysiol 97:3193–3205

    Article  PubMed  Google Scholar 

  • Anastasio TJ, Patton PE (2003) A two-stage unsupervised learning algorithm reproduces multisensory enhancement in a neural network model of the corticotectal system. J Neurosci 23:6713–6727

    PubMed  CAS  Google Scholar 

  • Anastasio TJ, Patton PE, Belkacem-Boussaid K (2000) Using Bayes’ rule to model multisensory enhancement in the superior colliculus. Neural Comput 12:1165–1187

    Article  PubMed  CAS  Google Scholar 

  • Arieh Y, Marks LE (2008) Cross-modal interaction between vision and hearing: a speed-accuracy analysis. Percept Psychophys 70:412–421

    Article  PubMed  Google Scholar 

  • Beauchamp MS (2005) Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3:93–113

    Article  PubMed  Google Scholar 

  • Bolognini N, Leo F, Passamonti C, Stein BE, Ladavas E (2007) Multisensory-mediated auditory localization. Perception 36:1477–1485

    Article  PubMed  Google Scholar 

  • Bonath B, Noesselt T, Martinez A, Mishra J, Schwiecker K, Heinze HJ, Hillyard SA (2007) Neural basis of the ventriloquist illusion. Curr Biol 17:1697–1703

    Article  PubMed  CAS  Google Scholar 

  • Burnett LR, Stein BE, Chaponis D, Wallace MT (2004) Superior colliculus lesions preferentially disrupt multisensory orientation. Neuroscience 124:535–547

    Article  PubMed  CAS  Google Scholar 

  • Burnett LR, Stein BE, Perrault TJ Jr, Wallace MT (2007) Excitotoxic lesions of the superior colliculus preferentially impact multisensory neurons and multisensory integration. Exp Brain Res 179:325–338

    Article  PubMed  Google Scholar 

  • Calvert GA (2001) Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex 11:1110–1123

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427–438

    Article  PubMed  CAS  Google Scholar 

  • Carriere BN, Royal DW, Perrault TJ, Morrison SP, Vaughan JW, Stein BE, Wallace MT (2007) Visual deprivation alters the development of cortical multisensory integration. J Neurophysiol 98:2858–2867

    Article  PubMed  Google Scholar 

  • Deneve S, Pouget A (2004) Bayesian multisensory integration and cross-modal spatial links. J Physiol Paris 98:249–258

    Article  PubMed  Google Scholar 

  • Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgements. Neuron 57:11–23

    Article  PubMed  CAS  Google Scholar 

  • Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415:429–433

    Article  PubMed  CAS  Google Scholar 

  • Foxe JJ, Schroeder CE (2005) The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16:419–423

    Article  PubMed  Google Scholar 

  • Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK (2005) Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci 25:5004–5012

    Article  PubMed  CAS  Google Scholar 

  • Gillmeister H, Eimer M (2007) Tactile enhancement of auditory detection and perceived loudness. Brain Res 1160:58–68

    Article  PubMed  CAS  Google Scholar 

  • Gingras G, Rowland BA, Stein BE (2006) Unisensory versus multisensory integration: computational distinctions in behavior. Soc Neurosci Abstr (36th Annual Meeting) (Abstract)

  • Hecht D, Reiner M, Karni A (2008a) Enhancement of response times to bi- and tri-modal sensory stimuli during active movements. Exp Brain Res 185:655–665

    Article  PubMed  Google Scholar 

  • Hecht D, Reiner M, Karni A (2008b) Multisensory enhancement: gains in choice and in simple response times. Exp Brain Res 189:133–143

    Article  PubMed  Google Scholar 

  • Holmes NP (2007) The law of inverse effectiveness in neurons and behaviour: multisensory integration versus normal variability. Neuropsychologia 45:3340–3345

    Article  PubMed  Google Scholar 

  • Holmes NP (2008) The seemingly inviolable principle of inverse effectiveness: in search of a null hypothesis. International Multisensory Research Forum, Hamburg, Germany, 2008-07-17

  • Holt GR, Softky WR, Koch C, Douglas RJ (1996) Comparison of discharge variability in vitro and in vivo in cat visual cortex neurons. J Neurophysiol 75:1806–1814

    PubMed  CAS  Google Scholar 

  • Jiang W, Stein BE (2003) Cortex controls multisensory depression in superior colliculus. J Neurophysiol 90:2123–2135

    Article  PubMed  Google Scholar 

  • Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE (2001) Two cortical areas mediate multisensory integration in superior colliculus neurons. J Neurophysiol 85:506–522

    PubMed  CAS  Google Scholar 

  • Jiang W, Jiang H, Stein BE (2002) Two corticotectal areas facilitate multisensory orientation behavior. J Cogn Neurosci 14:1240–1255

    Article  PubMed  Google Scholar 

  • Kadunce DC, Vaughan JW, Wallace MT, Benedek G, Stein BE (1997) Mechanisms of within- and cross-modality suppression in the superior colliculus. J Neurophysiol 78:2834–2847

    PubMed  CAS  Google Scholar 

  • Khanna SM, Teich MC (1989) Spectral characteristics of the responses of primary auditory-nerve fibers to amplitude-modulated signals. Hear Res 39:143–157

    Article  PubMed  CAS  Google Scholar 

  • King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60:492–500

    Article  PubMed  CAS  Google Scholar 

  • Lakatos P, Chen CM, O’Connell MN, Mills A, Schroeder CE (2007) Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53:279–292

    Article  PubMed  CAS  Google Scholar 

  • Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE (2005) On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp Brain Res 166:289–297

    Article  PubMed  Google Scholar 

  • Lippert M, Logothetis NK, Kayser C (2007) Improvement of visual contrast detection by a simultaneous sound. Brain Res 1173:102–109

    Article  PubMed  CAS  Google Scholar 

  • Logothetis NK, Pfeuffer J (2004) On the nature of the BOLD fMRI contrast mechanism. Magn Reson Imaging 22:1517–1531

    Article  PubMed  Google Scholar 

  • Lovelace CT, Stein BE, Wallace MT (2003) An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Brain Res Cogn Brain Res 17:447–453

    Article  PubMed  Google Scholar 

  • Ma WJ, Pouget A (2008) Linking neurons to behavior in multisensory perception: a computational review. Brain Res 1242:4–12

    Article  PubMed  CAS  Google Scholar 

  • Magosso E, Cuppini C, Serino A, di Pellegrino G, Ursino M (2008) A theoretical study of multisensory integration in the superior colliculus by a neural network model. Neural Netw 21:817–829

    Article  PubMed  Google Scholar 

  • Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221:389–391

    Article  PubMed  CAS  Google Scholar 

  • Meredith MA, Stein BE (1986) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56:640–662

    PubMed  CAS  Google Scholar 

  • Meredith MA, Stein BE (1996) Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol 75:1843–1857

    PubMed  CAS  Google Scholar 

  • Miller LM, D’Esposito M (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25:5884–5893

    Article  PubMed  CAS  Google Scholar 

  • Odgaard EC, Arieh Y, Marks LE (2003) Cross-modal enhancement of perceived brightness: sensory interaction versus response bias. Percept Psychophys 65:123–132

    PubMed  Google Scholar 

  • Patton P, Belkacem-Boussaid K, Anastasio TJ (2002) Multimodality in the superior colliculus: an information theoretic analysis. Brain Res Cogn Brain Res 14:10–19

    Article  PubMed  Google Scholar 

  • Peck CK (1996) Visual-auditory integration in cat superior colliculus: implications for neuronal control of the orienting response. Prog Brain Res 112:167–177

    Article  PubMed  CAS  Google Scholar 

  • Perrault TJ Jr, Vaughan JW, Stein BE, Wallace MT (2005) Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiol 93:2575–2586

    Article  PubMed  Google Scholar 

  • Populin LC, Yin TC (2002) Bimodal interactions in the superior colliculus of the behaving cat. J Neurosci 22:2826–2834

    PubMed  CAS  Google Scholar 

  • Pouget A, Deneve S, Duhamel JR (2002) A computational perspective on the neural basis of multisensory spatial representations. Nat Rev Neurosci 3:741–747

    Article  PubMed  CAS  Google Scholar 

  • Rach S, Diederich A (2006) Visual–tactile integration: does stimulus duration influence the relative amount of response enhancement? Exp Brain Res 173:514–520

    Article  PubMed  Google Scholar 

  • Ross LA, Saint-Amour D, Leavitt VM, Javitt DC, Foxe JJ (2007) Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb Cortex 17:1147–1153

    Article  PubMed  Google Scholar 

  • Rowland BA, Stein BE (2007) Multisensory integration produces an initial response enhancement. Front Integr Neurosci 1:4

    Article  PubMed  Google Scholar 

  • Rowland B, Stanford T, Stein B (2007a) A Bayesian model unifies multisensory spatial localization with the physiological properties of the superior colliculus. Exp Brain Res 180:153–161

    Article  PubMed  Google Scholar 

  • Rowland BA, Quessy S, Stanford TR, Stein BE (2007b) Multisensory integration shortens physiological response latencies. J Neurosci 27:5879–5884

    Article  PubMed  CAS  Google Scholar 

  • Schroeder CE (2001) Somatosensory input to auditory association cortex in the macaque monkey. J Neurophysiol 85:1322–1327

    PubMed  CAS  Google Scholar 

  • Softky WR, Koch C (1993) The highly irregular firing of cortical cells is inconsistent with temporal integration of random EPSPs. J Neurosci 13:334–350

    PubMed  CAS  Google Scholar 

  • Stanford TR, Stein BE (2007) Superadditivity in multisensory integration: putting the computation in context. Neuroreport 18:787–792

    Article  PubMed  Google Scholar 

  • Stanford TR, Quessy S, Stein BE (2005) Evaluating the operations underlying multisensory integration in the cat superior colliculus. J Neurosci 25:6499–6508

    Article  PubMed  CAS  Google Scholar 

  • Stein BE (2005) The development of a dialogue between cortex and midbrain to integrate multisensory information. Cortex 168:305–315

    Google Scholar 

  • Stein BE, Meredith MA (1990) Multisensory integration. Neural and behavioral solutions for dealing with stimuli from different sensory modalities. Ann N Y Acad Sci 608:51–65

    Article  PubMed  CAS  Google Scholar 

  • Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MA

    Google Scholar 

  • Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9:255–266

    Article  PubMed  CAS  Google Scholar 

  • Stein BE, Huneycutt WS, Meredith MA (1988) Neurons and behavior: the same rules of multisensory integration apply. Brain Res 448:355–358

    Article  PubMed  CAS  Google Scholar 

  • Stein BE, Meredith MA, Huneycutt WS, McDade L (1989) Behavioral indices of multisensory integration: orientation to visual cues is affected by visual stimuli. J Cogn Neurosci 1:12–24

    Article  Google Scholar 

  • Stein BE, London N, Wilkinson LK, Price DD (1996) Enhancement of perceived visual intensity by auditory stimuli: a psychophysical analysis. J Cogn Neurosci 8:497–506

    Article  Google Scholar 

  • Stevenson RA, Geoghegan ML, James TW (2007) Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp Brain Res 179:85–95

    Article  PubMed  Google Scholar 

  • Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215

    Article  Google Scholar 

  • Teich MC, Heneghan C, Lowen SB, Ozaki T, Kaplan E (1997) Fractal character of the neural spike train in the visual system of the cat. J Opt Soc Am A Opt Image Sci Vis 14:529–546

    Article  PubMed  CAS  Google Scholar 

  • Ursino M, Cuppini C, Magosso E, Serino A, di Pellegrino G (2008) Multisensory integration in the superior colliculus: a neural network model. J Comput Neurosci 26:55–73

    Article  PubMed  Google Scholar 

  • Wallace MT, Stein BE (1994) Cross-modal synthesis in the midbrain depends on input from cortex. J Neurophysiol 71:429–432

    PubMed  CAS  Google Scholar 

  • Wallace MT, Stein BE (1997) Development of multisensory neurons and multisensory integration in cat superior colliculus. J Neurosci 17:2429–2444

    PubMed  CAS  Google Scholar 

  • Wallace MT, Meredith MA, Stein BE (1992) Integration of multiple sensory modalities in cat cortex. Exp Brain Res 91:484–488

    Article  PubMed  CAS  Google Scholar 

  • Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76:1246–1266

    PubMed  CAS  Google Scholar 

  • Wallace MT, Meredith MA, Stein BE (1998) Multisensory integration in the superior colliculus of the alert cat. J Neurophysiol 80:1006–1010

    PubMed  CAS  Google Scholar 

  • Wallace MT, Perrault TJ Jr, Hairston WD, Stein BE (2004) Visual experience is necessary for the development of multisensory integration. J Neurosci 24:9580–9584

    Article  PubMed  CAS  Google Scholar 

  • Wang Y, Celebrini S, Trotter Y, Barone P (2008) Visuo-auditory interactions in the primary visual cortex of the behaving monkey: electrophysiological evidence. BMC Neurosci 9:79

    Article  PubMed  Google Scholar 

  • Wilkinson LK, Meredith MA, Stein BE (1996) The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp Brain Res 112:1–10

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgments

We thank Nancy London for technical assistance. Part of the research discussed here was supported by NIH grants EY016716 and NS36916.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Thomas J. Perrault Jr..

Rights and permissions

Reprints and permissions

About this article

Cite this article

Stein, B.E., Stanford, T.R., Ramachandran, R. et al. Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness. Exp Brain Res 198, 113–126 (2009). https://doi.org/10.1007/s00221-009-1880-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-009-1880-8

Keywords

Navigation