Elsevier

Current Opinion in Neurobiology

Volume 40, October 2016, Pages 31-37
Current Opinion in Neurobiology

Where are multisensory signals combined for perceptual decision-making?

https://doi.org/10.1016/j.conb.2016.06.003Get rights and content

Highlights

  • Multisensory integration is observed in sensory cortices and higher cortical areas.

  • Multisensory information could be integrated early and/or late in decision-making.

  • Manipulation of neural activity allows causal relationships to be established.

  • Causal tests indicate distributed networks underlie multisensory decision-making.

Multisensory integration is observed in many subcortical and cortical locations including primary and non-primary sensory cortex, and higher cortical areas including frontal and parietal cortex. During unisensory perceptual tasks many of these same brain areas show neural signatures associated with decision-making. It is unclear whether multisensory representations in sensory cortex directly inform decision-making in a multisensory task, or if cross-modal signals are only combined after the accumulation of unisensory evidence at a final decision-making stage in higher cortical areas. Manipulations of neuronal activity are required to establish causal roles for given brain regions in multisensory perceptual decision-making, and so far indicate that distributed networks underlie multisensory decision-making. Understanding multisensory integration requires synthesis of small-scale pathway specific and large-scale network level manipulations.

Introduction

Multisensory integration is evident in many behaviours and occurs at multiple locations in the brain [1]. Perceptual decision-making is the process by which sensory evidence is evaluated in order to make a decision and guide behaviour. Different sensory systems can provide independent estimates of a real-world source that are often complementary; for example, visual signals typically provide more precise spatial information, whereas auditory signals offer superior temporal resolution [2]. Integrating signals across sensory modalities can therefore reduce the inherent uncertainty within any sensory estimate and so improve performance in perceptual decision-making tasks. In the mammalian brain, the neural processes underlying decision-making [3, 4] and multisensory integration [5, 6, 7] have become increasingly well understood but remain largely independent lines of investigation. In particular, it remains an open question at what point(s) in the decision-making process information is combined across modalities. Here we address the question of how and where across-sensory information is combined for perceptual decision-making, and highlight the importance of manipulating neural activity in order to determine causal roles of early and late multisensory integration in decision-making.

Section snippets

Multisensory integration in perceptual decision-making

Perceptual decision-making is traditionally viewed as a serial, hierarchical process moving from the encoding of sensory information, to the accumulation of sensory evidence, through the calculation of decision variables and application of decision rules, and finally to production of a motor response [3, 4]. Encoding of (uni)sensory evidence emerges in the sensory cortices of the mammalian brain [8, 9•, 10, 11] while evidence accumulation and decision formation are associated with pre-motor,

Causality in distributed networks

To fully understand the relationship between multisensory integration and decision-making, it is necessary to establish the perceptual relevance of integration at different levels. For example, it is possible that neurons in sensory areas integrating cross-modal stimuli make no contribution to decision-making. This issue cannot be addressed with behavioural neurophysiology or neuroimaging alone and requires neural manipulation to establish the causal effects of perturbing multisensory

Functional connectivity

Network connectivity also complicates investigations of causal links between brain areas during multisensory integration and decision-making. The standard experimental design to test functional connectivity is to manipulate the activity in one region and observe any resulting effects in another. However in distributed networks, downstream effects may be realized through direct connections between regions, or via indirect pathways involving intermediary regions (Figure 2a). These mechanisms

Towards a synthesis

Early multisensory integration in sensory cortex offers the potential for decision-making based on multisensory representations. There is also substantial evidence for integration of cross-modal signals later in the decision-making process. Whether correlates of multisensory evidence accumulation and decision-making are observed in sensory cortex as well as, or instead of, in higher areas may depend on the nature of the task and on whether multisensory signals are perceived as originating from

References and recommended reading

Papers of particular interest, published within the period of review, have been highlighted as:

  • • of special interest

  • •• of outstanding interest

Conflict of interest statement

Nothing declared.

Acknowledgements

This work was supported by a Sir Henry Dale Research Fellowship (WT098418MA) and a Human Frontiers Science Program grant (RGY0068).

References (53)

  • M.R. Kamke et al.

    Parietal disruption alters audiovisual binding in the sound-induced flash illusion

    NeuroImage

    (2012)
  • J.C. Alvarado et al.

    Multisensory integration in the superior colliculus requires synergy among corticocollicular inputs

    J Neurosci

    (2009)
  • J.C. Erlich et al.

    Distinct effects of prefrontal and parietal cortex inactivations on an accumulation of evidence task in the rat

    Elife

    (2015)
  • T. Ohyama et al.

    A multilevel multimodal circuit enhances action selection in Drosophila

    Nature

    (2015)
  • R.B. Welch et al.

    Immediate perceptual response to intersensory discrepancy

    Psychol Bull

    (1980)
  • J.I. Gold et al.

    The neural basis of decision making

    Annu Rev Neurosci

    (2007)
  • H.R. Heekeren et al.

    The neural systems that mediate human perceptual decision making

    Nat Rev Neurosci

    (2008)
  • J.M. Yau et al.

    Dissecting neural circuits for multisensory integration and crossmodal processing

    Philos Trans R Soc Lond B Biol Sci

    (2015)
  • J. Tsunada et al.

    Causal contribution of primate auditory cortex to auditory perceptual decision-making

    Nat Neurosci

    (2016)
  • R. Romo et al.

    Sense, memory, and decision-making in the somatosensory cortical network

    Curr Opin Neurobiol

    (2012)
  • P. Cisek et al.

    Neural correlates of reaching decisions in dorsal premotor cortex: specification of multiple direction choices and final selection of action

    Neuron

    (2005)
  • T.D. Hanks et al.

    Distinct relationships of parietal and prefrontal cortices to evidence accumulation

    Nature

    (2015)
  • D. Raposo et al.

    Multisensory decision-making in rats and humans

    J Neurosci

    (2012)
  • L. Lemus et al.

    Do sensory cortices process more than one sensory modality during perceptual judgments?

    Neuron

    (2010)
  • J.K. Bizley et al.

    Role of auditory cortex in sound localization in the midsagittal plane

    J Neurophysiol

    (2007)
  • C. Kayser et al.

    Visual enhancement of the information representation in auditory cortex

    Curr Biol

    (2010)
  • Cited by (66)

    • Neurocomputational mechanisms underlying cross-modal associations and their influence on perceptual decisions

      2022, NeuroImage
      Citation Excerpt :

      When exposed to complementary features of sensory information, features that refer to the same object are redundantly associated, forming cross-modal associations, enabling the brain to exploit the correlation between such informational cues when forming perceptual decisions from ambiguous, and often noisy, unisensory information (Bien et al., 2012; Glicksohn and Cohen, 2013) . Cross-modal associations have been shown to influence the consolidation of multisensory information when forming perceptual decisions (Bizley et al., 2016; Drugowitsch et al., 2014; Engel et al., 2012). This has been evidenced in studies that have used speeded classification paradigms, demonstrating behavioural effects such as increased response speed (i.e. decreased reaction times; RTs; Kayser and Kayser, 2018; Laurienti et al., 2004; Silva et al., 2017), increased choice accuracy (Franzen et al., 2020; Kayser and Kayser, 2018; Kayser et al., 2017; Kim et al., 2008), and improved stimulus detection (Adam and Noppeney, 2014; Aller et al., 2015).

    View all citing articles on Scopus
    View full text