Trends in Cognitive Sciences
Volume 14, Issue 9, September 2010, Pages 400-410
Journal home page for Trends in Cognitive Sciences

Review
The multifaceted interplay between attention and multisensory integration

https://doi.org/10.1016/j.tics.2010.06.008Get rights and content

Multisensory integration has often been characterized as an automatic process. Recent findings indicate that multisensory integration can occur across various stages of stimulus processing that are linked to, and can be modulated by, attention. Stimulus-driven, bottom-up mechanisms induced by crossmodal interactions can automatically capture attention towards multisensory events, particularly when competition to focus elsewhere is relatively low. Conversely, top-down attention can facilitate the integration of multisensory inputs and lead to a spread of attention across sensory modalities. These findings point to a more intimate and multifaceted interplay between attention and multisensory integration than was previously thought. We review developments in the current understanding of the interactions between attention and multisensory processing, and propose a framework that unifies previous, apparently discordant, findings.

Section snippets

Bidirectional influences between multisensory integration and attention

Our brains are continuously inundated with stimulation arriving through our various sensory pathways. The processes involved in synthesizing and organizing this multisensory deluge of inputs are fundamental to effective perception and cognitive functioning. Although the combination of information across the senses has been investigated since psychology became an experimental discipline [1], the past decade has seen a sharp increase of interest in this question. Within this context, the issue of

A framework for interactions between attention and multisensory integration

As noted above, and as reviewed in the sections to follow, the evidence in favor of the existence of bidirectional influences between attention and multisensory integration is considerable. Under certain circumstances the co-occurrence of stimuli in different modalities can lead preattentively and automatically to multisensory integration in a bottom-up fashion, which then makes it more probable that the resulting event will capture attention and thus capitalize available processing resources

Stimulus-driven influences of multisensory integration on attention

A clear demonstration of the possible involvement of stimulus-driven multisensory integration on attentional selection was recently provided using a difficult visual search task [24]. In this study visual targets were presented among an array of similar distractor stimuli (Figure 2). Search times for the visual target increased with an increasing number of display items (distractors), as has typically been found in demanding unisensory visual search tasks 47, 48. In these displays, targets and

Influence of top-down directed attention on multisensory processing

As mentioned earlier, recent studies have provided evidence for the influence of top-down attention on multisensory integration processes. In a recent EEG study, for example, it was shown that spatial attention can strongly influence multiple stages of multisensory processing, beginning as early as 80 ms post-stimulus [40]. Complementarily, a recent fMRI study [39] showed increased activity in multiple brain areas, including the superior temporal sulcus, striate visual cortex, extrastriate

Crossmodal spreading of attention

Another way that attention can interactively influence the processing of stimulus inputs in different sensory modalities that do occur at the same time has been termed crossmodal spreading of attention, which has been found to occur even when the visual and auditory stimuli arise from different spatial locations [37]. More specifically, it has been shown that a task-irrelevant, centrally presented, auditory stimulus can elicit a different brain response when paired in time with an attended

Concluding remarks and future directions

Although it is generally acknowledged that multisensory integration processes can influence the bottom-up orienting of attention to salient stimuli [23], the role of top-down attention and the interplay between bottom-up and top-down mechanisms during multisensory integration processes remains a matter of ongoing debate. On the one hand, there is relatively little effect of top-down attention on multisensory integration under conditions of low competition between stimuli [33]. On the other

Acknowledgements

We wish to thank Erik van der Burg, Scott Fairhall and Emiliano Macaluso for providing figure material for this paper, and Sarah Donohue and Matthijs Noordzij for helpful comments on an earlier version. The effort for this work was supported by funding from the Institute for Behavioral Research at the University of Twente to D.T., by a grant from the German Research Foundation (SE 1859/1-1) to D.S., by grants from the MICINN (PSI2010-15426 - CDS2007-00012) and DIUE (SRG2009-092) to S.S.-F., and

Glossary

Attentional orienting
Attention involves mechanisms whereby processing resources are preferentially allocated toward particular locations, features or objects. Attentional orienting refers to the process responsible for moving the focus of attention from one location, feature or object, to another. Orienting can occur covertly, that is, in the absence of movements of the eyes or other sensory receptor surfaces (e.g. ears), as well as overtly, where the shift is accompanied by a reorienting of

References (104)

  • J. Navarra

    Assessing the role of attention in the audiovisual integration of speech

    Information Fusion

    (2010)
  • J.J. Stekelenburg

    Illusory sound shifts induced by the ventriloquist illusion evoke the mismatch negativity

    Neurosci. Lett.

    (2004)
  • B. Bonath

    Neural Basis of the Ventriloquist Illusion

    Curr. Biol.

    (2007)
  • A. Alsius

    Audiovisual integration of speech falters under high attention demands

    Curr. Biol.

    (2005)
  • A.M. Treisman et al.

    A feature-integration theory of attention

    Cogn. Psychol.

    (1980)
  • J.M. Wolfe

    Moving towards solutions to some enduring controversies in visual search

    Trends Cogn. Sci.

    (2003)
  • D. Senkowski

    Look who's talking: The deployment of visuo-spatial attention during multisensory speech processing under noisy environmental conditions

    NeuroImage

    (2008)
  • W.A. Teder-Sälejärvi

    Intra-modal and cross-modal spatial attention to auditory and visual stimuli. An event-related brain potential study

    Brain Res. Cogn. Brain Res.

    (1999)
  • A. Degerman

    Human brain activity associated with audiovisual perception and attention

    NeuroImage

    (2007)
  • L.A. Ross

    Impaired multisensory processing in schizophrenia: Deficits in the visual enhancement of speech comprehension under noisy environmental conditions

    Schizophr. Res.

    (2007)
  • R. Jardri

    Neural functional organization of hallucinations in schizophrenia: Multisensory dissolution of pathological emergence in consciousness

    Conscious. Cogn.

    (2009)
  • J.T. Serences et al.

    Feature-Based Attentional Modulations in the Absence of Direct Visual Stimulation

    Neuron

    (2007)
  • S. Yantis et al.

    Cortical mechanisms of space-based and object-based attentional control

    Curr. Opin. in Neurobiol.

    (2003)
  • T. Moore

    Visuomotor origins of covert spatial attention

    Neuron

    (2003)
  • Y.H. Kim

    The large-scale neural network for spatial attention displays multifunctional overlap but differential asymmetry

    NeuroImage

    (1999)
  • C.T. Wu

    The neural circuitry underlying the executive control of auditory spatial attention

    Brain Res.

    (2007)
  • M. Eimer et al.

    Crossmodal links in endogenous and exogenous spatial attention: Evidence from event-related brain potential studies

    Neurosci. Biobehav. Rev.

    (2001)
  • V. Urbantschisch

    Über den Einfluss einer Sinneserrugun auf die übrigen Sinnesempfindungen

    Arch. Gesch. Psych.

    (1880)
  • C. Spence et al.

    Cross-Modal Space and Cross-Modal attention

    (2004)
  • B. Stein et al.

    The merging of the senses

    (1993)
  • M.T. Wallace

    Multisensory integration in the superior colliculus of the alert cat

    J. Neurophys.

    (1998)
  • R.B. Welch et al.

    Intersensory Interactions

  • N.P. Holmes

    The principle of inverse effectiveness in multisensory integration: some statistical considerations

    Brain Topogr.

    (2009)
  • M.H. Giard et al.

    Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study

    J. Cogn. Neurosci.

    (1999)
  • J.J. Foxe et al.

    The case for feedforward multisensory convergence during early cortical processing

    NeuroReport

    (2005)
  • C. Kayser et al.

    Do early sensory cortices integrate cross-modal information?

    Brain Struct. Funct.

    (2007)
  • M.O. Ernst et al.

    Humans integrate visual and haptic information in a statistically optimal fashion

    Nature

    (2002)
  • G.A. Calvert

    Crossmodal processing in the human brain: Insights from functional neuroimaging studies

    Cereb. Cortex

    (2001)
  • M.S. Beauchamp

    Statistical criteria in fMRI studies of multisensory integration

    Neuroinformatics

    (2005)
  • H. Colonius et al.

    Multisensory interaction in saccadic reaction time: A time-window-of- integration model

    J. Cogn. Neurosci.

    (2004)
  • H. Pashler

    The Psychology of Attention

    (1999)
  • J. Theeuwes

    Exogenous and Endogenous Control of Attention: the Effect of Visual Onsets and Offsets

    Percep. Psychophys.

    (1991)
  • J. Driver

    Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading

    Nature

    (1996)
  • B.E. Stein

    Multisensory Integration in Single Neurons of the Midbrain

  • E. Van der Burg

    Pip and Pop: Nonspatial Auditory Signals Improve Spatial Visual Search

    J. Exp. Psychol. Hum. Percep. Perform.

    (2008)
  • A. Fort

    Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans

    Cereb. Cortex

    (2002)
  • R.B. Welch et al.

    Immediate perceptual response to intersensory discrepancy

    Psych. Bull.

    (1980)
  • A. Vatakis et al.

    Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli

    Percept. Psychophys.

    (2007)
  • P. Bertelson

    The ventriloquist effect does not depend on the direction of deliberate visual attention

    Percept. Psychophys.

    (2000)
  • J. Vroomen

    The ventriloquist effect does not depend on the direction of automatic visual attention

    Percept. Psychophys.

    (2001)
  • Cited by (0)

    View full text