Characterizing the dynamics of mental representations: the temporal generalization method

doi:10.1016/j.tics.2014.01.002

Trends in Cognitive Sciences

Volume 18, Issue 4, April 2014, Pages 203-210

https://doi.org/10.1016/j.tics.2014.01.002 Get rights and content

Highlights

•
MVPA is often used to detect static fMRI activation patterns.
•
Adapting MVPA to time-resolved signals can characterize the dynamics of neural responses.
•
The temporal generalization matrix reveals a repertoire of canonical brain dynamics.

Parsing a cognitive task into a sequence of operations is a central problem in cognitive neuroscience. We argue that a major advance is now possible owing to the application of pattern classifiers to time-resolved recordings of brain activity [electroencephalography (EEG), magnetoencephalography (MEG), or intracranial recordings]. By testing at which moment a specific mental content becomes decodable in brain activity, we can characterize the time course of cognitive codes. Most importantly, the manner in which the trained classifiers generalize across time, and from one experimental condition to another, sheds light on the temporal organization of information-processing stages. A repertoire of canonical dynamical patterns is observed across various experiments and brain regions. This method thus provides a novel way to understand how mental representations are manipulated and transformed.

Section snippets

Understanding the organization of processing stages: from behavior to neuroimaging

Understanding how mental representations unfold in time during the performance of a task is a central goal for cognitive psychology. Donders [1] first suggested that mental operations could be dissected by comparing the subjects’ response times in different experimental conditions. This ‘mental chronometry’ was later enriched with several methodological inventions, including the additive-factors method [2] and the psychological refractory period method [3]. Although these behavioral techniques

Decoding fMRI data identifies the localization and structure of mental representations.

MVPA was first introduced to brain imaging in order to refine the analysis of functional MRI (fMRI). Temporal resolution aside, fMRI is an efficient tool to isolate and localize the brain mechanisms underlying specific mental representations. Initially, fMRI was primarily used with binary contrasts that revealed major differences in regional brain activity (e.g., faces versus non-faces in the fusiform cortex [8]). MVPA, however, led to a considerable refinement of such inferences because it

Decoding time-resolved signals identifies when mental representations are activated

MVPA applied to fMRI signals does not reveal much about the dynamics with which mental representations are activated. Here we focus specifically on the less explored question of what MVPA may bring to our understanding of the dynamics of information processing in the brain.

Methodologically, MVPA readily applies to EEG, MEG, or intracranial recording data (e.g., multiunit neuronal recordings, local field potentials) where time can be considered as an additional dimension besides the spatial

Generalization across time reveals how mental representations are dynamically transformed

The decoding approach detailed above can be extended to ask whether the neural code that supports above-chance decoding is stable or is dynamically evolving (see Figure I in Box 1). The principle is simple: instead of applying a different classifier at each time point, the classifier trained at time t can be tested on its ability to generalize to time t’. Generalization implies that the neural code that was identified at time t recurred at time t’.

Systematically adopting this approach leads to

Generalization across conditions reveals how information processing is changed

We just showed that the temporal generalization matrix provides detailed information about the sequence of processing stages engaged in a particular task. If we change the experimental conditions, however, some of these processing stages may remain unaffected whereas others may be accelerated, slowed, deleted, or inserted. Can decoding also illuminate such reorganizations? If a classifier is trained in one condition and tested on its ability to generalize to another, the resulting temporal

Concluding remarks

Since Donders and Sternberg, brain algorithms have been dissected by manipulating experimental factors such as attention, expectation, or instructions that selectively accelerate, slow, remove, insert, or reorder specific processing stages. Behavioral methods of mental chronometry, however, provide only indirect information about such reorganizations. Here we summarized several ideas and empirical studies that suggest that MVPA can provide considerable information about the fine temporal

Acknowledgments

The authors thank Alexandre Gramfort, Imen El Karoui, Sébastien Marti, Florent Meyniel, Lionel Naccache, and Aaron Schurger as well as Shbana Ahmad and our anonymous reviewers for their helpful comments. This work was supported by a grant from the Direction Générale de l’Armement (DGA) to J-R.K. and by the Institut National de la Recherche Médicale (INSERM), the Commissariat à l’Energie Atomique et aux Energies Alternatives (CEA), and a European Research Council (ERC) senior grant ‘NeuroConsc’

References (83)

F.C. Donders
On the speed of mental processes
Acta Psychol. (Amst.)
(1969)
S. Sternberg
The discovery of processing stages: extensions of Donders’ method
Acta Psychol. (Amst.)
(1969)
N. Nishitani et al.
Viewing lip forms: cortical dynamics
Neuron
(2002)
N. Kriegeskorte et al.
Representational geometry: integrating cognition, computation, and the brain
Trends Cogn. Sci.
(2013)
P. Kok
Less is more: expectation sharpens representations in the primary visual cortex
Neuron
(2012)
Y. Miyawaki
Visual image reconstruction from human brain activity using a combination of multiscale local image decoders
Neuron
(2008)
B. Thirion
Inverse retinotopy: inferring the visual content of images from brain activation patterns
Neuroimage
(2006)
M.A.J. Van Gerven
Dynamic decoding of ongoing perception
Neuroimage
(2011)
S. Nishimoto
Reconstructing visual experiences from brain activity evoked by natural movies
Curr. Biol.
(2011)
S-H. Lee
Disentangling visual imagery and perception of real-world objects
Neuroimage
(2012)

A.M. Albers

Shared representations for working memory and mental imagery in early visual cortex

Curr. Biol.

(2013)

J.O. Garcia

Near-real-time feature-selective modulations in human cortex

Curr. Biol.

(2013)

J-R. King

Single-trial decoding of auditory novelty responses facilitates the detection of residual consciousness

Neuroimage

(2013)

A.M. Chan

Decoding word and category-specific spatiotemporal representations from MEG and EEG

Neuroimage

(2011)

G. Sudre

Tracking neural coding of perceptual and semantic features of concrete nouns

Neuroimage

(2012)

R.S. Schaefer

Name that tune: decoding music from the listening brain

Neuroimage

(2011)

L. Fuentemilla

Theta-coupled periodic replay in working memory

Curr. Biol.

(2010)

M.G. Stokes

Dynamic coding for cognitive control in prefrontal cortex

Neuron

(2013)

K. Louie et al.

Temporally structured replay of awake hippocampal ensemble activity during rapid eye movement sleep

Neuron

(2001)

G.R. Poe

Experience-dependent phase-reversal of hippocampal neuron firing during REM sleep

Brain Res.

(2000)

A.K. Lee et al.

Memory of sequential experience in the hippocampus during slow wave sleep

Neuron

(2002)

T.J. Davidson

Hippocampal replay of extended experience

Neuron

(2009)

J-D. Haynes

Reading hidden intentions in the human brain

Curr. Biol.

(2007)

J.J. DiCarlo

How does the brain solve visual object recognition?

Neuron

(2012)

N. Li et al.

Unsupervised natural visual experience rapidly reshapes size-invariant object representation in inferior temporal cortex

Neuron

(2010)

D.C. Knill et al.

The Bayesian brain: the role of uncertainty in neural coding and computation

Trends Neurosci.

(2004)

J. Heinzle

Topographically specific functional connectivity between visual field maps in the human brain

Neuroimage

(2011)

H. Pashler

Dual-task interference in simple tasks: data and theory

Psychol. Bull.

(1994)

J.T. Townsend

Serial vs parallel processing: sometimes they look like Tweedledum and Tweedledee but they can (and should) be distinguished

Psychol. Sci.

(1990)

J.L. McClelland

On the time-relations of mental processes: an examination of systems of processes in cascade

Psychol. Rev.

(1979)

S. Dehaene

The organization of brain activations in number comparison: event-related potentials and the additive-factors methods

J. Cogn. Neurosci.

(1996)

N. Kanwisher

The fusiform face area: a module in human extrastriate cortex specialized for face perception

J. Neurosci.

(1997)

R. Quian Quiroga et al.

Extracting information from neuronal populations: information theory and decoding approaches

Nat. Rev. Neurosci.

(2009)

J-D.D. Haynes et al.

Decoding mental states from brain activity in humans

Nat. Rev. Neurosci.

(2006)

Y. Kamitani et al.

Decoding the visual and subjective contents of the human brain

Nat. Neurosci.

(2005)

S.A. Harrison et al.

Decoding reveals the contents of visual working memory in early visual areas

Nature

(2009)

J. Freeman

Orientation decoding depends on maps, not columns

J. Neurosci.

(2011)

G.J. Brouwer et al.

Decoding and reconstructing color from responses in human visual cortex

J. Neurosci.

(2009)

J. Gallant

Bayesian reconstruction of perceptual experiences from human brain activity

Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience

(2009)

K.N. Kay et al.

I can see what you see

Nat. Neurosci.

(2009)

A. Gramfort

Decoding visual percepts induced by word reading with fMRI

2012 Second International Workshop on Pattern Recognition in NeuroImaging

(2012)

Cited by (481)

Hearing fearful prosody impairs visual working memory maintenance
2024, International Journal of Psychophysiology
Interference by distractors has been associated multiple times with diminished visual and auditory working memory (WM) performance. Negative emotional distractors in particular lead to detrimental effects on WM. However, these associations have only been seen when distractors and items to maintain in WM are from the same sensory modality. In this study, we investigate cross-modal interference on WM. We invited 20 participants to complete a visual change-detection task, assessing visual WM (VWM), while hearing emotional (fearful) and neutral auditory distractors. Electrophysiological activity was recorded to measure contralateral delay activity (CDA) and auditory P2 event-related potentials (ERP), indexing WM maintenance and distractor salience respectively. At the behavioral level, fearful prosody didn't decrease significantly working memory accuracy, compared to neutral prosody. Regarding ERPs, fearful distractors evoked a greater P2 amplitude than neutral distractors. Correlations between the two ERP potentials indicated that P2 amplitude difference between the two types of prosody was associated with the difference in CDA amplitude for fearful and neutral trials. This association suggests that cognitive resources required to process fearful prosody detrimentally impact VWM maintenance. That result provides a piece of additional evidence that negative emotional stimuli produce greater interference than neutral stimuli and that the cognitive resources used to process stimuli from different modalities come from a common pool.
The neural dynamics of conflict adaptation induced by conflict observation: Evidence from univariate and multivariate analysis
2024, International Journal of Psychophysiology
Conflict adaptation can be expressed as greater performance (shorter response time and lower error rate) after incongruent trials when compared to congruent trials. It has been observed in designs that minimize confounding factors, i.e., feature integration, contingency learning, and temporal learning. Our current study aimed to further elucidate the temporal evolution mechanisms of conflict adaptation. To address this issue, the current study employed a combination of behavioral, univariate, and multivariate analysis (MVPA) methods in a modified color-word Stroop task, where half of the trials required button presses (DO trials), and the other half only required observation (LOOK trials). Both behavioral and the ERP results (N450 and SP) in the LOOK-DO transition trials revealed significant conflict adaptation without feature integration, contingency learning, and temporal learning, providing support for the conflict monitoring theory. Furthermore, during the LOOK trials, significant Stroop effect in the N450 and SP components were observed, indicating that conflict monitoring occurred at the stimulus level and triggered reactive control adjustments. The MVPA results decoded the congruent-incongruent and incongruent-incongruent conditions during the conflict adjustment phase but not during the conflict monitoring phase, emphasizing the unique contribution of conflict adjustment to conflict adaptation. The current research findings provided more compelling supporting evidence for the conflict monitoring theory, while also indicating that future studies should employ the present design to elucidate the specific processes of conflict adaptation.
Tracking the neural codes for words and phrases during semantic composition, working-memory storage, and retrieval
2024, Cell Reports
The ability to compose successive words into a meaningful phrase is a characteristic feature of human cognition, yet its neural mechanisms remain incompletely understood. Here, we analyze the cortical mechanisms of semantic composition using magnetoencephalography (MEG) while participants read one-word, two-word, and five-word noun phrases and compared them with a subsequent image. Decoding of MEG signals revealed three processing stages. During phrase comprehension, the representation of individual words was sustained for a variable duration depending on phrasal context. During the delay period, the word code was replaced by a working-memory code whose activation increased with semantic complexity. Finally, the speed and accuracy of retrieval depended on semantic complexity and was faster for surface than for deep semantic properties. In conclusion, we propose that the brain initially encodes phrases using factorized dimensions for successive words but later compresses them in working memory and requires a period of decompression to access them.
The computational foundations of dynamic coding in working memory
2024, Trends in Cognitive Sciences
Working memory (WM) is a fundamental aspect of cognition. WM maintenance is classically thought to rely on stable patterns of neural activities. However, recent evidence shows that neural population activities during WM maintenance undergo dynamic variations before settling into a stable pattern. Although this has been difficult to explain theoretically, neural network models optimized for WM typically also exhibit such dynamics. Here, we examine stable versus dynamic coding in neural data, classical models, and task-optimized networks. We review principled mathematical reasons for why classical models do not, while task-optimized models naturally do exhibit dynamic coding. We suggest an update to our understanding of WM maintenance, in which dynamic coding is a fundamental computational feature rather than an epiphenomenon.
EEG-based multivariate pattern analysis reveals the control mechanisms of emotion regulation through distancing
2024, International Journal of Clinical and Health Psychology
A neurocognitive model of distancing has systematically identified a set of brain regions that support the control mechanisms for emotion regulation (ER). However, the temporal dynamics of these control mechanisms during ER remains unclear.
To address this issue, we recorded behavioral and electroencephalogram (EEG) data to compare proactive and reactive ER modes in an adapted ER task (N = 30 adults). In different ER modes, participants were instructed to downregulate their negative emotional experiences by applying the reappraisal tactic of distancing.
The behavioral results showed that proactive ER, which involves preparing for the upcoming regulation, reduced the negative emotional experience more than reactive ER, which involves no preparation process, in the reappraisal-negative condition. This indicated that proactive ER was more effective than reactive ER in regulating negative emotions. Event-related potential (ERP) and multivariate pattern analysis (MVPA) results showed that ER through distancing involved two phases: First, the reappraisal cue enhanced the allocation of attention to activate the mental building blocks and constructed a new perspective in the preparation process. Second, participants who benefited from the preparation process initiated the ER earlier and adaptively re-engaged in the ER if time permitted.
Taken together, the control mechanisms underlying the preparation process influence the timing of ER, while the control mechanisms underlying the regulation process determine the regulatory effect.
Strength of predicted information content in the brain biases decision behavior
2023, Current Biology
Prediction-for-perception theories suggest that the brain predicts incoming stimuli to facilitate their categorization.¹^,²^,³^,⁴^,⁵^,⁶^,⁷^,⁸^,⁹^,¹⁰^,¹¹^,¹²^,¹³^,¹⁴^,¹⁵^,¹⁶^,¹⁷ However, it remains unknown what the information contents of these predictions are, which hinders mechanistic explanations. This is because typical approaches cast predictions as an underconstrained contrast between two categories¹⁸^,¹⁹^,²⁰^,²¹^,²²^,²³^,²⁴—e.g., faces versus cars, which could lead to predictions of features specific to faces or cars, or features from both categories. Here, to pinpoint the information contents of predictions and thus their mechanistic processing in the brain, we identified the features that enable two different categorical perceptions of the same stimuli. We then trained multivariate classifiers to discern, from dynamic MEG brain responses, the features tied to each perception. With an auditory cueing design, we reveal where, when, and how the brain reactivates visual category features (versus the typical category contrast) before the stimulus is shown. We demonstrate that the predictions of category features have a more direct influence (bias) on subsequent decision behavior in participants than the typical category contrast. Specifically, these predictions are more precisely localized in the brain (lateralized), are more specifically driven by the auditory cues, and their reactivation strength before a stimulus presentation exerts a greater bias on how the individual participant later categorizes this stimulus. By characterizing the specific information contents that the brain predicts and then processes, our findings provide new insights into the brain’s mechanisms of prediction for perception.

View all citing articles on Scopus

View full text

Trends in Cognitive Sciences

OpinionCharacterizing the dynamics of mental representations: the temporal generalization method

Highlights

Section snippets

Understanding the organization of processing stages: from behavior to neuroimaging

Decoding fMRI data identifies the localization and structure of mental representations.

Decoding time-resolved signals identifies when mental representations are activated

Generalization across time reveals how mental representations are dynamically transformed

Generalization across conditions reveals how information processing is changed

Concluding remarks

Acknowledgments

Acta Psychol. (Amst.)

Acta Psychol. (Amst.)

Neuron

Trends Cogn. Sci.

Neuron

Neuron

Neuroimage

Neuroimage

Curr. Biol.

Neuroimage

Curr. Biol.

Curr. Biol.

Neuroimage

Neuroimage

Neuroimage

Neuroimage

Curr. Biol.

Neuron

Neuron

Brain Res.

Neuron

Neuron

Curr. Biol.

Neuron

Neuron

Trends Neurosci.

Neuroimage

Dual-task interference in simple tasks: data and theory

Psychol. Bull.

Serial vs parallel processing: sometimes they look like Tweedledum and Tweedledee but they can (and should) be distinguished

Psychol. Sci.

On the time-relations of mental processes: an examination of systems of processes in cascade

Psychol. Rev.

The organization of brain activations in number comparison: event-related potentials and the additive-factors methods

J. Cogn. Neurosci.

The fusiform face area: a module in human extrastriate cortex specialized for face perception

J. Neurosci.

Extracting information from neuronal populations: information theory and decoding approaches

Nat. Rev. Neurosci.

Decoding mental states from brain activity in humans

Nat. Rev. Neurosci.

Decoding the visual and subjective contents of the human brain

Nat. Neurosci.

Decoding reveals the contents of visual working memory in early visual areas

Nature

Orientation decoding depends on maps, not columns

J. Neurosci.

Decoding and reconstructing color from responses in human visual cortex

J. Neurosci.

Bayesian reconstruction of perceptual experiences from human brain activity

Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience

I can see what you see

Nat. Neurosci.

Decoding visual percepts induced by word reading with fMRI

2012 Second International Workshop on Pattern Recognition in NeuroImaging

Opinion
Characterizing the dynamics of mental representations: the temporal generalization method