Where are multisensory signals combined for perceptual decision-making?
Introduction
Multisensory integration is evident in many behaviours and occurs at multiple locations in the brain [1]. Perceptual decision-making is the process by which sensory evidence is evaluated in order to make a decision and guide behaviour. Different sensory systems can provide independent estimates of a real-world source that are often complementary; for example, visual signals typically provide more precise spatial information, whereas auditory signals offer superior temporal resolution [2]. Integrating signals across sensory modalities can therefore reduce the inherent uncertainty within any sensory estimate and so improve performance in perceptual decision-making tasks. In the mammalian brain, the neural processes underlying decision-making [3, 4] and multisensory integration [5, 6, 7] have become increasingly well understood but remain largely independent lines of investigation. In particular, it remains an open question at what point(s) in the decision-making process information is combined across modalities. Here we address the question of how and where across-sensory information is combined for perceptual decision-making, and highlight the importance of manipulating neural activity in order to determine causal roles of early and late multisensory integration in decision-making.
Section snippets
Multisensory integration in perceptual decision-making
Perceptual decision-making is traditionally viewed as a serial, hierarchical process moving from the encoding of sensory information, to the accumulation of sensory evidence, through the calculation of decision variables and application of decision rules, and finally to production of a motor response [3, 4]. Encoding of (uni)sensory evidence emerges in the sensory cortices of the mammalian brain [8, 9•, 10, 11] while evidence accumulation and decision formation are associated with pre-motor,
Causality in distributed networks
To fully understand the relationship between multisensory integration and decision-making, it is necessary to establish the perceptual relevance of integration at different levels. For example, it is possible that neurons in sensory areas integrating cross-modal stimuli make no contribution to decision-making. This issue cannot be addressed with behavioural neurophysiology or neuroimaging alone and requires neural manipulation to establish the causal effects of perturbing multisensory
Functional connectivity
Network connectivity also complicates investigations of causal links between brain areas during multisensory integration and decision-making. The standard experimental design to test functional connectivity is to manipulate the activity in one region and observe any resulting effects in another. However in distributed networks, downstream effects may be realized through direct connections between regions, or via indirect pathways involving intermediary regions (Figure 2a). These mechanisms
Towards a synthesis
Early multisensory integration in sensory cortex offers the potential for decision-making based on multisensory representations. There is also substantial evidence for integration of cross-modal signals later in the decision-making process. Whether correlates of multisensory evidence accumulation and decision-making are observed in sensory cortex as well as, or instead of, in higher areas may depend on the nature of the task and on whether multisensory signals are perceived as originating from
References and recommended reading
Papers of particular interest, published within the period of review, have been highlighted as:
• of special interest
•• of outstanding interest
Conflict of interest statement
Nothing declared.
Acknowledgements
This work was supported by a Sir Henry Dale Research Fellowship (WT098418MA) and a Human Frontiers Science Program grant (RGY0068).
References (53)
- et al.
Multisensory integration: methodological approaches and emerging principles in the human brain
J Physiol Paris
(2004) - et al.
Defining auditory–visual objects: behavioral tests and physiological mechanisms
Trends Neurosci
(2016) - et al.
Multisensory integration: flexible use of general operations
Neuron
(2014) - et al.
Auditory cortex represents both pitch judgments and the corresponding acoustic cues
Curr Biol
(2013) - et al.
Sense and the single neuron: probing the physiology of perception
Annu Rev Neurosci
(1998) - et al.
Neural encoding of auditory discrimination in ventral premotor cortex
Proc Natl Acad Sci U S A
(2009) - et al.
Dynamic faces speed up the onset of auditory cortical spiking responses during vocal detection
Proc Natl Acad Sci U S A
(2013) - et al.
Possible anatomical pathways for short-latency multisensory integration processes in primary sensory cortices
Brain Struct Funct
(2015) - et al.
Physiological and anatomical evidence for multisensory interactions in auditory cortex
Cereb Cortex
(2007) - et al.
Neural correlates of reliability-based cue weighting during multisensory integration
Nat Neurosci
(2012)
Parietal disruption alters audiovisual binding in the sound-induced flash illusion
NeuroImage
Multisensory integration in the superior colliculus requires synergy among corticocollicular inputs
J Neurosci
Distinct effects of prefrontal and parietal cortex inactivations on an accumulation of evidence task in the rat
Elife
A multilevel multimodal circuit enhances action selection in Drosophila
Nature
Immediate perceptual response to intersensory discrepancy
Psychol Bull
The neural basis of decision making
Annu Rev Neurosci
The neural systems that mediate human perceptual decision making
Nat Rev Neurosci
Dissecting neural circuits for multisensory integration and crossmodal processing
Philos Trans R Soc Lond B Biol Sci
Causal contribution of primate auditory cortex to auditory perceptual decision-making
Nat Neurosci
Sense, memory, and decision-making in the somatosensory cortical network
Curr Opin Neurobiol
Neural correlates of reaching decisions in dorsal premotor cortex: specification of multiple direction choices and final selection of action
Neuron
Distinct relationships of parietal and prefrontal cortices to evidence accumulation
Nature
Multisensory decision-making in rats and humans
J Neurosci
Do sensory cortices process more than one sensory modality during perceptual judgments?
Neuron
Role of auditory cortex in sound localization in the midsagittal plane
J Neurophysiol
Visual enhancement of the information representation in auditory cortex
Curr Biol
Cited by (66)
Audiovisual integration in children with cochlear implants revealed through EEG and fNIRS
2023, Brain Research BulletinNeurocomputational mechanisms underlying cross-modal associations and their influence on perceptual decisions
2022, NeuroImageCitation Excerpt :When exposed to complementary features of sensory information, features that refer to the same object are redundantly associated, forming cross-modal associations, enabling the brain to exploit the correlation between such informational cues when forming perceptual decisions from ambiguous, and often noisy, unisensory information (Bien et al., 2012; Glicksohn and Cohen, 2013) . Cross-modal associations have been shown to influence the consolidation of multisensory information when forming perceptual decisions (Bizley et al., 2016; Drugowitsch et al., 2014; Engel et al., 2012). This has been evidenced in studies that have used speeded classification paradigms, demonstrating behavioural effects such as increased response speed (i.e. decreased reaction times; RTs; Kayser and Kayser, 2018; Laurienti et al., 2004; Silva et al., 2017), increased choice accuracy (Franzen et al., 2020; Kayser and Kayser, 2018; Kayser et al., 2017; Kim et al., 2008), and improved stimulus detection (Adam and Noppeney, 2014; Aller et al., 2015).
Visual modulation of firing and spectrotemporal receptive fields in mouse auditory cortex
2022, Current Research in Neurobiology