Trends in Cognitive Sciences
OpinionCharacterizing the dynamics of mental representations: the temporal generalization method
Section snippets
Understanding the organization of processing stages: from behavior to neuroimaging
Understanding how mental representations unfold in time during the performance of a task is a central goal for cognitive psychology. Donders [1] first suggested that mental operations could be dissected by comparing the subjects’ response times in different experimental conditions. This ‘mental chronometry’ was later enriched with several methodological inventions, including the additive-factors method [2] and the psychological refractory period method [3]. Although these behavioral techniques
Decoding fMRI data identifies the localization and structure of mental representations.
MVPA was first introduced to brain imaging in order to refine the analysis of functional MRI (fMRI). Temporal resolution aside, fMRI is an efficient tool to isolate and localize the brain mechanisms underlying specific mental representations. Initially, fMRI was primarily used with binary contrasts that revealed major differences in regional brain activity (e.g., faces versus non-faces in the fusiform cortex [8]). MVPA, however, led to a considerable refinement of such inferences because it
Decoding time-resolved signals identifies when mental representations are activated
MVPA applied to fMRI signals does not reveal much about the dynamics with which mental representations are activated. Here we focus specifically on the less explored question of what MVPA may bring to our understanding of the dynamics of information processing in the brain.
Methodologically, MVPA readily applies to EEG, MEG, or intracranial recording data (e.g., multiunit neuronal recordings, local field potentials) where time can be considered as an additional dimension besides the spatial
Generalization across time reveals how mental representations are dynamically transformed
The decoding approach detailed above can be extended to ask whether the neural code that supports above-chance decoding is stable or is dynamically evolving (see Figure I in Box 1). The principle is simple: instead of applying a different classifier at each time point, the classifier trained at time t can be tested on its ability to generalize to time t’. Generalization implies that the neural code that was identified at time t recurred at time t’.
Systematically adopting this approach leads to
Generalization across conditions reveals how information processing is changed
We just showed that the temporal generalization matrix provides detailed information about the sequence of processing stages engaged in a particular task. If we change the experimental conditions, however, some of these processing stages may remain unaffected whereas others may be accelerated, slowed, deleted, or inserted. Can decoding also illuminate such reorganizations? If a classifier is trained in one condition and tested on its ability to generalize to another, the resulting temporal
Concluding remarks
Since Donders and Sternberg, brain algorithms have been dissected by manipulating experimental factors such as attention, expectation, or instructions that selectively accelerate, slow, remove, insert, or reorder specific processing stages. Behavioral methods of mental chronometry, however, provide only indirect information about such reorganizations. Here we summarized several ideas and empirical studies that suggest that MVPA can provide considerable information about the fine temporal
Acknowledgments
The authors thank Alexandre Gramfort, Imen El Karoui, Sébastien Marti, Florent Meyniel, Lionel Naccache, and Aaron Schurger as well as Shbana Ahmad and our anonymous reviewers for their helpful comments. This work was supported by a grant from the Direction Générale de l’Armement (DGA) to J-R.K. and by the Institut National de la Recherche Médicale (INSERM), the Commissariat à l’Energie Atomique et aux Energies Alternatives (CEA), and a European Research Council (ERC) senior grant ‘NeuroConsc’
References (83)
On the speed of mental processes
Acta Psychol. (Amst.)
(1969)The discovery of processing stages: extensions of Donders’ method
Acta Psychol. (Amst.)
(1969)- et al.
Viewing lip forms: cortical dynamics
Neuron
(2002) - et al.
Representational geometry: integrating cognition, computation, and the brain
Trends Cogn. Sci.
(2013) Less is more: expectation sharpens representations in the primary visual cortex
Neuron
(2012)Visual image reconstruction from human brain activity using a combination of multiscale local image decoders
Neuron
(2008)Inverse retinotopy: inferring the visual content of images from brain activation patterns
Neuroimage
(2006)Dynamic decoding of ongoing perception
Neuroimage
(2011)Reconstructing visual experiences from brain activity evoked by natural movies
Curr. Biol.
(2011)Disentangling visual imagery and perception of real-world objects
Neuroimage
(2012)
Shared representations for working memory and mental imagery in early visual cortex
Curr. Biol.
Near-real-time feature-selective modulations in human cortex
Curr. Biol.
Single-trial decoding of auditory novelty responses facilitates the detection of residual consciousness
Neuroimage
Decoding word and category-specific spatiotemporal representations from MEG and EEG
Neuroimage
Tracking neural coding of perceptual and semantic features of concrete nouns
Neuroimage
Name that tune: decoding music from the listening brain
Neuroimage
Theta-coupled periodic replay in working memory
Curr. Biol.
Dynamic coding for cognitive control in prefrontal cortex
Neuron
Temporally structured replay of awake hippocampal ensemble activity during rapid eye movement sleep
Neuron
Experience-dependent phase-reversal of hippocampal neuron firing during REM sleep
Brain Res.
Memory of sequential experience in the hippocampus during slow wave sleep
Neuron
Hippocampal replay of extended experience
Neuron
Reading hidden intentions in the human brain
Curr. Biol.
How does the brain solve visual object recognition?
Neuron
Unsupervised natural visual experience rapidly reshapes size-invariant object representation in inferior temporal cortex
Neuron
The Bayesian brain: the role of uncertainty in neural coding and computation
Trends Neurosci.
Topographically specific functional connectivity between visual field maps in the human brain
Neuroimage
Dual-task interference in simple tasks: data and theory
Psychol. Bull.
Serial vs parallel processing: sometimes they look like Tweedledum and Tweedledee but they can (and should) be distinguished
Psychol. Sci.
On the time-relations of mental processes: an examination of systems of processes in cascade
Psychol. Rev.
The organization of brain activations in number comparison: event-related potentials and the additive-factors methods
J. Cogn. Neurosci.
The fusiform face area: a module in human extrastriate cortex specialized for face perception
J. Neurosci.
Extracting information from neuronal populations: information theory and decoding approaches
Nat. Rev. Neurosci.
Decoding mental states from brain activity in humans
Nat. Rev. Neurosci.
Decoding the visual and subjective contents of the human brain
Nat. Neurosci.
Decoding reveals the contents of visual working memory in early visual areas
Nature
Orientation decoding depends on maps, not columns
J. Neurosci.
Decoding and reconstructing color from responses in human visual cortex
J. Neurosci.
Bayesian reconstruction of perceptual experiences from human brain activity
Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience
I can see what you see
Nat. Neurosci.
Decoding visual percepts induced by word reading with fMRI
2012 Second International Workshop on Pattern Recognition in NeuroImaging
Cited by (481)
Hearing fearful prosody impairs visual working memory maintenance
2024, International Journal of PsychophysiologyThe neural dynamics of conflict adaptation induced by conflict observation: Evidence from univariate and multivariate analysis
2024, International Journal of PsychophysiologyThe computational foundations of dynamic coding in working memory
2024, Trends in Cognitive SciencesEEG-based multivariate pattern analysis reveals the control mechanisms of emotion regulation through distancing
2024, International Journal of Clinical and Health PsychologyStrength of predicted information content in the brain biases decision behavior
2023, Current Biology