Computational advances towards linking BOLD and behavior
Highlights
► Traditional analysis approaches that focus on mean response amplitudes discard useful information. ► Multivariate techniques exploit information contained in spatially distributed response patterns. ► Decoding and encoding models can be used in a complementary manner to asses this information. ► Linking the output of these models with behavior is a critical means of validating new findings.
Introduction
The field of cognitive neuroscience seeks to establish and characterize links between neural modulations and behavioral measures that index latent processes such as perception, memory, and decision making. Articulating and critically testing these linking hypotheses is far from trivial, even when neural modulations are directly measured using single-unit recording techniques (deCharms & Zador, 2000). Sampling of individual neurons is inherently biased and caution must be exercised when generalizing beyond simple animal models, particularly when studying more abstract cognitive operations. Moreover, focusing on changes in spiking rates may not turn out to be the correct level of analysis to elucidate links between brain and behavior; perhaps lower or higher levels of analysis are more relevant (i.e. subthreshold changes in membrane potential or understanding the covariance structure of large neural populations: See Cohen and Maunsell, 2009, Cohen and Maunsell, 2010, Cohen and Maunsell, 2011, Mitchell, Sundberg, and Reynolds (2009)). At the other end of the spectrum, measuring the blood oxygenation level dependent (BOLD) signal in humans using fMRI provides a non-invasive and large-scale view of cortical activation while subjects perform arbitrarily complex cognitive tasks. However, the poorly understood relationship between single-unit neural activity and the BOLD signal places a hard constraint on the specificity of linking hypotheses that can be formulated, and makes it difficult to reconcile results obtained across the two domains even when similar paradigms are employed. This failure to make mutually constraining advances stems at least in part from a general reluctance (or inability) in the neuroimaging community to explicitly state hypothesized relationships between changes in neural activity, the BOLD signal, and the cognitive state of the observer. Instead, an implicit and overly simplistic assumption has come to dominate the field: a larger BOLD response implies that a region plays a more important role in task-related information processing.
This point is not brought up to attack the utility of using fMRI as a tool for investigating links between neural activity and cognition. Instead, the general lack of stated linking hypotheses highlights the inherent limitation of available imaging technologies, and also the fact that there is no viable analysis technique that circumvents all potential shortcomings. It is becoming increasingly clear, however, that the relative paucity of fMRI studies that evaluate specific a priori hypotheses about the link between BOLD signals and behavior is a major obstacle that must be overcome if we are to start realizing the type of strong-inference that characterizes the analysis and understanding of simpler animal model circuits (e.g. Briggman and Kristan, 2008, Field et al., 2010 for two recent examples). Following this agenda, we first provide a selective historical overview of fMRI methods that have been used over the last two decades. Then we critically discuss two complementary analysis techniques that have recently been applied to fMRI data: decoding approaches that utilize multi-voxel pattern analysis (MVPA) to infer and label stimuli or cognitive states, and complementary encoding approaches that use a priori models of neural activity to predict observed BOLD response patterns. In particular, encoding models hold promise as a means of evaluating as-yet untested ideas about the role of population codes in information processing, thus highlighting an area of inquiry for which fMRI might be well-suited to make key new discoveries. Ultimately, we argue that these new methods can be used to systematically link BOLD responses with behavior, thus placing empirical observations into a format that is more comparable with data gathered using complementary neuroscientific techniques. Although most of this review is based on studies carried out in visual cortex – primarily because so much neurophysiological data is available to constrain the interpretation of fMRI signals – the issues raised in each example should in principle generalize to other cortical areas and domains of inquiry.
Section snippets
Fundamental assumptions about the relationship between neural activity and the BOLD signal
The primary assumption behind all experiments seeking to understand the neural mechanisms of behavior using BOLD fMRI is that deflections in the magnitude of the BOLD signal are at least monotonically related to changes in the magnitude of underlying neural activity (Boynton et al., 1996, Heeger et al., 2000, Logothetis et al., 2001). However, the exact quantification of this relationship is challenging: the BOLD signal is an indirect measure of neural activity that reflects metabolic
Univariate neuroimaging techniques
Since the advent of BOLD neuroimaging, a vast majority of studies have focused on pinpointing the anatomical loci of neural mechanisms that putatively support a particular behavior or cognitive function. This approach is very much in the neuropsychological tradition of linking focal brain damage to specific behavioral deficits, albeit using a non-invasive imaging modality applied to intact volunteer subjects. In a typical fMRI brain mapping study, two or more experimental factors are
Limitations of univariate techniques
The main limitation of univariate methods is that different patterns of neural modulation can produce indistinguishable changes in the mean amplitude of the BOLD response, which can lead to two types of inferential error: (1) incorrectly attributing an increase in the BOLD response to a specific pattern of neural modulation, and (2) incorrectly concluding that an experimental manipulation had no influence on neural activity within a ROI. To be more concrete, consider a hypothetical experiment
Computational neuroimaging: combining univariate approaches with quantitative models
The general approach of using quantitative models to link changes in the BOLD signal with perception and cognition was introduced by investigators such as Brian Wandell, David Heeger and others within the computational neuroimaging tradition (see Wandell, 1999 for an early review). As opposed to mapping out networks of regions using whole-brain GLM analyses, computational neuroimaging focuses on examining parametric modulations within specific brain regions for which strong BOLD/behavior
Decoding using multi-voxel pattern analysis (MVPA)
Over a decade ago now, James Haxby and his coworkers published an influential study that demonstrated how the category of an object that a subject was viewing could be decoded based on the spatially distributed activation pattern across all voxels in inferior-temporal visual cortex (a region comprised of many smaller regions such as the lateral occipital complex, fusiform gyrus, parahippocampal gyrus, and early ventral visual areas such as human V4; Haxby et al., 2001). The insight that Haxby
Limitations of MVPA
Although MVPA is an elegant tool that is flexible enough for a wide range of experimental designs and research questions, this flexibility reduces the precision of the inferences that can be supported. The foremost issue is that an observation of increased classification accuracy does not clearly reveal how or why that increase occurred. For instance, the hypothetical observation that selective attention increases classification accuracy for decoding 45° from 135° oriented stimuli could arise
Encoding models of BOLD responses
In contrast to the decoding approach employed by MVPA studies, forward encoding models take the opposite approach by adopting a set of a priori assumptions about the important features or stimulus labels that can be distinguished using hemodynamic signals within an ROI (Dumoulin and Wandell, 2008, Gourtzelidis et al., 2005, Kay and Gallant, 2009, Kay et al., 2008, Mitchell et al., 2008, Naselaris et al., 2009, Schonwiesner and Zatorre, 2009, Thirion et al., 2006; reviewed in Naselaris, Kay,
Limitations of encoding models and comparison with decoding approaches
The main challenge of the encoding approach lies in generating a model that accurately characterizes neural activity within a ROI, particularly when examining higher cognitive functions that have not been subjected to intense psychophysical or single-unit physiology studies. However, the general approach provides a technique for developing multiple models – even in the absence of prior neurophysiological data as a guide – and then testing these models in at least two ways. The most intuitive
Concluding remarks
Beginning with the advent of computational neuroimaging in the late 1990s, a great deal of progress has been made with respect to testing quantitative models of perception and cognition using non-invasive methods such as fMRI. The recent explosion of MVPA and forward encoding approaches holds great promise, as these new tools have the potential to more precisely evaluate the information content of single ROIs as well as large-scale networks, and to refine our understanding of how the
Acknowledgements
We thank Gijs Brouwer and David Heeger for useful discussions and for help gathering figures. This work was supported by NIH grant MH092345 to J.T.S.
References (111)
- et al.
Overlapping mechanisms of attention and spatial working memory
Trends in Cognitive Sciences
(2001) Attention and visual perception
Current Opinion in Neurobiology
(2005)- et al.
Neuronal basis of contrast discrimination
Vision Research
(1999) - et al.
Using neuronal populations to study the mechanisms underlying spatial and feature attention
Neuron
(2011) - et al.
The reorienting system of the human brain: From environment to theory of mind
Neuron
(2008) - et al.
Functional magnetic resonance imaging (fMRI) “brain reading”: Detecting and classifying distributed patterns of fMRI activity in human visual cortex
Neuroimage
(2003) - et al.
What could underlie the trial-related signal?. A response to the commentaries by Drs. Kleinschmidt and Muller, and Drs. Handwerker and Bandettini
Neuroimage
(2011) - et al.
Combining multivariate voxel selection and support vector machines for mapping and classification of fMRI spatial patterns
Neuroimage
(2008) - et al.
Recurrent neuronal circuits in the neocortex
Current Biology: CB
(2007) - et al.
Population receptive field estimates in human visual cortex
Neuroimage
(2008)
Multivariate analysis of fMRI time series: Classification and regression of brain responses using machine learning
Magnetic Resonance Imaging
Sparse linear regression for reconstructing muscle activity from human cortical fMRI
Neuroimage
Is cortical vasculature functionally organized?
Neuroimage
Hemodynamic signals not predicted? Not so: A comment on Sirotin and Das (2009)
Neuroimage
Simple explanations before complex theories: Alternative interpretations of Sirotin and Das’ observations
Neuroimage
Decoding visual consciousness from human brain signals
Trends in Cognitive Sciences
Decoding seen and attended motion directions from activity in the human visual cortex
Current Biology: CB
The blind, the lame, and the poor signals of brain function– A comment on Sirotin and Das (2009)
Neuroimage
Analyzing for information, not activation, to exploit high-resolution fMRI
Neuroimage
Combining the tools: Activation- and information-based fMRI analysis
Neuroimage
Feature-based attention increases the selectivity of population responses in primate visual cortex
Current Biology: CB
Spatial attention decorrelates intrinsic activity fluctuations in macaque area V4
Neuron
Visual image reconstruction from human brain activity using a combination of multiscale local image decoders
Neuron
Encoding and decoding in fMRI
Neuroimage
Bayesian reconstruction of natural images from human brain activity
Neuron
Beyond mind-reading: Multi-voxel pattern analysis of fMRI data
Trends in Cognitive Sciences
The role of early visual cortex in visual short-term memory and visual attention
Vision Research
Against hyperacuity in brain reading: Spatial smoothing does not hurt multivariate fMRI analyses?
Neuroimage
Machine learning classifiers and fMRI: A tutorial overview
Neuroimage
Working memory as an emergent property of the mind and brain
Neuroscience
The normalization model of attention
Neuron
Attention increases sensitivity of V4 neurons
Neuron
Feature-based attentional modulations in the absence of direct visual stimulation
Neuron
Attention and perceptual cohence fields
Trends in Cognitive Sciences
Linking propositions
Vision Research
Cognitive neural prosthetics
Annual Review of Psychology
Imaging orientation selectivity: Decoding conscious perception in V1
Nature Neuroscience
Linear systems analysis of functional magnetic resonance imaging in human V1
Journal of Neuroscience
Multifunctional pattern-generating circuits
Annual Review of Neuroscience
Physiology of the retina and visual pathways
Decoding and reconstructing color from responses in human visual cortex
The Journal of Neuroscience: The Official Journal of the Society for Neuroscience
Introduction to functional magnetic resonance imaging: Principles and techniques
Do we know what the early visual system does?
The Journal of Neuroscience: The Official Journal of the Society for Neuroscience
Exploring functional connectivities of the human brain using multivariate information analysis
Attention improves performance primarily by reducing interneuronal correlations
Nature Neuroscience
A neuronal population measure of attention predicts behavioral performance on individual trials
The Journal of Neuroscience: The Official Journal of the Society for Neuroscience
Control of goal-directed and stimulus-driven attention in the brain
Nature Reviews. Neuroscience
From cognitive to neural models of working memory
Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences
Predicting neuronal responses during natural vision
Network
Cited by (87)
The role of alpha oscillations in spatial attention: limited evidence for a suppression account
2019, Current Opinion in PsychologyCitation Excerpt :Recently, multivariate analysis techniques have enabled a more refined quantification of the spatial information present in alpha activity. We and others have used an inverted encoding model (IEM) [8–10] to track the spatial and temporal dynamics of covert attention [11••,12•]. This approach (Figure 1) assumes that alpha power at each electrode reflects the combined activity of a number of spatially selective channels (or neuronal populations).
Strategic control of location and ordinal context in visual working memory
2023, Cerebral Cortex