Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Neural correlates of cross-modal binding

Nature Neuroscience volume 6, pages 190–195 (2003)Cite this article

Abstract

Little is known about how the brain binds together signals from multiple sensory modalities to produce unified percepts of objects and events in the external world. Using event-related functional magnetic resonance imaging (fMRI) in humans, we measured transient brain responses to auditory/visual binding, as evidenced by a sound-induced change in visual motion perception. Identical auditory and visual stimuli were presented in all trials, but in some trials they were perceived to be bound together and in others they were perceived as unbound unimodal events. Cross-modal binding was associated with higher activity in multimodal areas, but lower activity in predominantly unimodal areas. This activation pattern suggests that a reciprocal and 'competitive' interaction between multimodal and unimodal areas underlies the perceptual interpretation of simultaneous signals from multiple sensory modalities.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Behavioral task.
Figure 2: Brain regions with significantly higher activity during 'collide' trials than 'pass' trials.
Figure 3: Brain regions with significantly higher activity during pass trials than during collide trials.

Similar content being viewed by others

References

  1. Morrell, L.K. Temporal characteristics of sensory interaction in choice reaction times. J. Exp. Psychol. 77, 14–18 (1968).

    Article  CAS  Google Scholar 

  2. McGurk, H. & MacDonald, J. Hearing lips and seeing voices. Nature 264, 746–748 (1976).

    Article  CAS  Google Scholar 

  3. Driver, J. Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading. Nature 381, 66–68 (1996).

    Article  CAS  Google Scholar 

  4. Sekuler, R., Sekuler, A.B. & Lau, R. Sound alters visual motion perception [letter]. Nature 385, 308 (1997).

  5. Stein, B.E. Neural mechanisms for synthesizing sensory information and producing adaptive behaviors. Exp. Brain Res. 123, 124–135 (1998).

    Article  CAS  Google Scholar 

  6. Fuster, J.M., Bodner, M. & Kroger, J.K. Cross-modal and cross-temporal association in neurons of frontal cortex. Nature 405, 347–351 (2000).

    Article  CAS  Google Scholar 

  7. Macaluso, E., Frith, C.D. & Driver, J. Modulation of human visual cortex by cross-modal spatial attention. Science 289, 1206–1208 (2000).

    Article  CAS  Google Scholar 

  8. Calvert, G.A., Hansen, P.C., Iversen, S.D. & Brammer, M.J. Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the bold effect. Neuroimage 14, 427–438 (2001).

    Article  CAS  Google Scholar 

  9. Bushara, K.O., Grafman, J. & Hallett, M. Neural correlates of auditory-visual stimulus onset asynchrony detection. J. Neurosci. 21, 300–304 (2001).

    Article  CAS  Google Scholar 

  10. Watson, J.D. et al. Area V5 of the human brain: evidence from a combined study using positron emission tomography and magnetic resonance imaging. Cereb. Cortex 3, 79–94 (1993).

    Article  CAS  Google Scholar 

  11. Fletcher, P.C. et al. The mind's eye—precuneus activation in memory-related imagery. Neuroimage 2, 195–200 (1995).

    Article  CAS  Google Scholar 

  12. Alho, K., Woods, D.L. & Algazi, A. Processing of auditory stimuli during auditory and visual attention as revealed by event-related potentials. Psychophysiology 31, 469–479 (1994).

    Article  CAS  Google Scholar 

  13. Spence, C., Nicholls, M.E. & Driver, J. The cost of expecting events in the wrong sensory modality. Percept. Psychophys. 63, 330–336 (2001).

    Article  CAS  Google Scholar 

  14. van Gelderen, P. et al. Three-dimensional functional magnetic resonance imaging of human brain on a clinical 1.5-T scanner. Proc. Natl. Acad. Sci. USA 92, 6906–6910 (1995).

    Article  CAS  Google Scholar 

  15. Josephs, O. & Henson, R.N. Event-related functional magnetic resonance imaging: modelling, inference and optimization. Philos. Trans. R. Soc. Lond. B Biol. Sci. 354, 1215–1228 (1999).

    Article  CAS  Google Scholar 

  16. Friston, K.J. et al. Event-related fMRI: characterizing differential responses. Neuroimage 7, 30–40 (1998).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank J. Rauschecker for comments on the manuscript, D.G. Schoenberg for editing the manuscript, and N. Dang, P. van Gelderen and H. Shill for their assistance. Supported by the National Institute of Neurological Disorders and Stroke and the Department of Veterans Affairs.

Author information

Authors and Affiliations

  1. Neurology Service, Room 4B135, Minneapolis Veterans Affairs Medical Center, One Veterans Drive, Minneapolis, 55417, Minnesota, USA

    Khalafalla O. Bushara

  2. Human Motor Control Section, NINDS, NIH, Building 10, Room 5N226, 10 Center Drive, Bethesda, 20892-1428, Maryland, USA

    Khalafalla O. Bushara, Takashi Hanakawa, Ilka Immisch, Keiichiro Toma, Kenji Kansaku & Mark Hallett

Authors
  1. Khalafalla O. Bushara
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Takashi Hanakawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ilka Immisch
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Keiichiro Toma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kenji Kansaku
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mark Hallett
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Khalafalla O. Bushara.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bushara, K., Hanakawa, T., Immisch, I. et al. Neural correlates of cross-modal binding. Nat Neurosci 6, 190–195 (2003). https://doi.org/10.1038/nn993

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn993

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing