Elsevier

Neuropsychologia

Volume 50, Issue 4, March 2012, Pages 435-446
Neuropsychologia

Computational advances towards linking BOLD and behavior

https://doi.org/10.1016/j.neuropsychologia.2011.07.013Get rights and content

Abstract

Traditionally, fMRI studies have focused on analyzing the mean response amplitude within a cortical area. However, the mean response is blind to many important patterns of cortical modulation, which severely limits the formulation and evaluation of linking hypotheses between neural activity, BOLD responses, and behavior. More recently, multivariate pattern classification analysis (MVPA) has been applied to fMRI data to evaluate the information content of spatially distributed activation patterns. This approach has been remarkably successful at detecting the presence of specific information in targeted brain regions, and provides an extremely flexible means of extracting that information without a precise generative model for the underlying neural activity. However, this flexibility comes at a cost: since MVPA relies on pooling information across voxels that are selective for many different stimulus attributes, it is difficult to infer how specific sub-sets of tuned neurons are modulated by an experimental manipulation. In contrast, recently developed encoding models can produce more precise estimates of feature-selective tuning functions, and can support the creation of explicit linking hypotheses between neural activity and behavior. Although these encoding models depend on strong – and often untested – assumptions about the response properties of underlying neural generators, they also provide a unique opportunity to evaluate population-level computational theories of perception and cognition that have previously been difficult to assess using either single-unit recording or conventional neuroimaging techniques.

Highlights

► Traditional analysis approaches that focus on mean response amplitudes discard useful information. ► Multivariate techniques exploit information contained in spatially distributed response patterns. ► Decoding and encoding models can be used in a complementary manner to asses this information. ► Linking the output of these models with behavior is a critical means of validating new findings.

Introduction

The field of cognitive neuroscience seeks to establish and characterize links between neural modulations and behavioral measures that index latent processes such as perception, memory, and decision making. Articulating and critically testing these linking hypotheses is far from trivial, even when neural modulations are directly measured using single-unit recording techniques (deCharms & Zador, 2000). Sampling of individual neurons is inherently biased and caution must be exercised when generalizing beyond simple animal models, particularly when studying more abstract cognitive operations. Moreover, focusing on changes in spiking rates may not turn out to be the correct level of analysis to elucidate links between brain and behavior; perhaps lower or higher levels of analysis are more relevant (i.e. subthreshold changes in membrane potential or understanding the covariance structure of large neural populations: See Cohen and Maunsell, 2009, Cohen and Maunsell, 2010, Cohen and Maunsell, 2011, Mitchell, Sundberg, and Reynolds (2009)). At the other end of the spectrum, measuring the blood oxygenation level dependent (BOLD) signal in humans using fMRI provides a non-invasive and large-scale view of cortical activation while subjects perform arbitrarily complex cognitive tasks. However, the poorly understood relationship between single-unit neural activity and the BOLD signal places a hard constraint on the specificity of linking hypotheses that can be formulated, and makes it difficult to reconcile results obtained across the two domains even when similar paradigms are employed. This failure to make mutually constraining advances stems at least in part from a general reluctance (or inability) in the neuroimaging community to explicitly state hypothesized relationships between changes in neural activity, the BOLD signal, and the cognitive state of the observer. Instead, an implicit and overly simplistic assumption has come to dominate the field: a larger BOLD response implies that a region plays a more important role in task-related information processing.

This point is not brought up to attack the utility of using fMRI as a tool for investigating links between neural activity and cognition. Instead, the general lack of stated linking hypotheses highlights the inherent limitation of available imaging technologies, and also the fact that there is no viable analysis technique that circumvents all potential shortcomings. It is becoming increasingly clear, however, that the relative paucity of fMRI studies that evaluate specific a priori hypotheses about the link between BOLD signals and behavior is a major obstacle that must be overcome if we are to start realizing the type of strong-inference that characterizes the analysis and understanding of simpler animal model circuits (e.g. Briggman and Kristan, 2008, Field et al., 2010 for two recent examples). Following this agenda, we first provide a selective historical overview of fMRI methods that have been used over the last two decades. Then we critically discuss two complementary analysis techniques that have recently been applied to fMRI data: decoding approaches that utilize multi-voxel pattern analysis (MVPA) to infer and label stimuli or cognitive states, and complementary encoding approaches that use a priori models of neural activity to predict observed BOLD response patterns. In particular, encoding models hold promise as a means of evaluating as-yet untested ideas about the role of population codes in information processing, thus highlighting an area of inquiry for which fMRI might be well-suited to make key new discoveries. Ultimately, we argue that these new methods can be used to systematically link BOLD responses with behavior, thus placing empirical observations into a format that is more comparable with data gathered using complementary neuroscientific techniques. Although most of this review is based on studies carried out in visual cortex – primarily because so much neurophysiological data is available to constrain the interpretation of fMRI signals – the issues raised in each example should in principle generalize to other cortical areas and domains of inquiry.

Section snippets

Fundamental assumptions about the relationship between neural activity and the BOLD signal

The primary assumption behind all experiments seeking to understand the neural mechanisms of behavior using BOLD fMRI is that deflections in the magnitude of the BOLD signal are at least monotonically related to changes in the magnitude of underlying neural activity (Boynton et al., 1996, Heeger et al., 2000, Logothetis et al., 2001). However, the exact quantification of this relationship is challenging: the BOLD signal is an indirect measure of neural activity that reflects metabolic

Univariate neuroimaging techniques

Since the advent of BOLD neuroimaging, a vast majority of studies have focused on pinpointing the anatomical loci of neural mechanisms that putatively support a particular behavior or cognitive function. This approach is very much in the neuropsychological tradition of linking focal brain damage to specific behavioral deficits, albeit using a non-invasive imaging modality applied to intact volunteer subjects. In a typical fMRI brain mapping study, two or more experimental factors are

Limitations of univariate techniques

The main limitation of univariate methods is that different patterns of neural modulation can produce indistinguishable changes in the mean amplitude of the BOLD response, which can lead to two types of inferential error: (1) incorrectly attributing an increase in the BOLD response to a specific pattern of neural modulation, and (2) incorrectly concluding that an experimental manipulation had no influence on neural activity within a ROI. To be more concrete, consider a hypothetical experiment

Computational neuroimaging: combining univariate approaches with quantitative models

The general approach of using quantitative models to link changes in the BOLD signal with perception and cognition was introduced by investigators such as Brian Wandell, David Heeger and others within the computational neuroimaging tradition (see Wandell, 1999 for an early review). As opposed to mapping out networks of regions using whole-brain GLM analyses, computational neuroimaging focuses on examining parametric modulations within specific brain regions for which strong BOLD/behavior

Decoding using multi-voxel pattern analysis (MVPA)

Over a decade ago now, James Haxby and his coworkers published an influential study that demonstrated how the category of an object that a subject was viewing could be decoded based on the spatially distributed activation pattern across all voxels in inferior-temporal visual cortex (a region comprised of many smaller regions such as the lateral occipital complex, fusiform gyrus, parahippocampal gyrus, and early ventral visual areas such as human V4; Haxby et al., 2001). The insight that Haxby

Limitations of MVPA

Although MVPA is an elegant tool that is flexible enough for a wide range of experimental designs and research questions, this flexibility reduces the precision of the inferences that can be supported. The foremost issue is that an observation of increased classification accuracy does not clearly reveal how or why that increase occurred. For instance, the hypothetical observation that selective attention increases classification accuracy for decoding 45° from 135° oriented stimuli could arise

Encoding models of BOLD responses

In contrast to the decoding approach employed by MVPA studies, forward encoding models take the opposite approach by adopting a set of a priori assumptions about the important features or stimulus labels that can be distinguished using hemodynamic signals within an ROI (Dumoulin and Wandell, 2008, Gourtzelidis et al., 2005, Kay and Gallant, 2009, Kay et al., 2008, Mitchell et al., 2008, Naselaris et al., 2009, Schonwiesner and Zatorre, 2009, Thirion et al., 2006; reviewed in Naselaris, Kay,

Limitations of encoding models and comparison with decoding approaches

The main challenge of the encoding approach lies in generating a model that accurately characterizes neural activity within a ROI, particularly when examining higher cognitive functions that have not been subjected to intense psychophysical or single-unit physiology studies. However, the general approach provides a technique for developing multiple models – even in the absence of prior neurophysiological data as a guide – and then testing these models in at least two ways. The most intuitive

Concluding remarks

Beginning with the advent of computational neuroimaging in the late 1990s, a great deal of progress has been made with respect to testing quantitative models of perception and cognition using non-invasive methods such as fMRI. The recent explosion of MVPA and forward encoding approaches holds great promise, as these new tools have the potential to more precisely evaluate the information content of single ROIs as well as large-scale networks, and to refine our understanding of how the

Acknowledgements

We thank Gijs Brouwer and David Heeger for useful discussions and for help gathering figures. This work was supported by NIH grant MH092345 to J.T.S.

References (111)

  • E. Formisano et al.

    Multivariate analysis of fMRI time series: Classification and regression of brain responses using machine learning

    Magnetic Resonance Imaging

    (2008)
  • G. Ganesh et al.

    Sparse linear regression for reconstructing muscle activity from human cortical fMRI

    Neuroimage

    (2008)
  • J.L. Gardner

    Is cortical vasculature functionally organized?

    Neuroimage

    (2010)
  • D.A. Handwerker et al.

    Hemodynamic signals not predicted? Not so: A comment on Sirotin and Das (2009)

    Neuroimage

    (2011)
  • D.A. Handwerker et al.

    Simple explanations before complex theories: Alternative interpretations of Sirotin and Das’ observations

    Neuroimage

    (2011)
  • J.D. Haynes

    Decoding visual consciousness from human brain signals

    Trends in Cognitive Sciences

    (2009)
  • Y. Kamitani et al.

    Decoding seen and attended motion directions from activity in the human visual cortex

    Current Biology: CB

    (2006)
  • A. Kleinschmidt et al.

    The blind, the lame, and the poor signals of brain function– A comment on Sirotin and Das (2009)

    Neuroimage

    (2010)
  • N. Kriegeskorte et al.

    Analyzing for information, not activation, to exploit high-resolution fMRI

    Neuroimage

    (2007)
  • N. Kriegeskorte et al.

    Combining the tools: Activation- and information-based fMRI analysis

    Neuroimage

    (2007)
  • J.C. Martinez-Trujillo et al.

    Feature-based attention increases the selectivity of population responses in primate visual cortex

    Current Biology: CB

    (2004)
  • J.F. Mitchell et al.

    Spatial attention decorrelates intrinsic activity fluctuations in macaque area V4

    Neuron

    (2009)
  • Y. Miyawaki et al.

    Visual image reconstruction from human brain activity using a combination of multiscale local image decoders

    Neuron

    (2008)
  • T. Naselaris et al.

    Encoding and decoding in fMRI

    Neuroimage

    (2011)
  • T. Naselaris et al.

    Bayesian reconstruction of natural images from human brain activity

    Neuron

    (2009)
  • K.A. Norman et al.

    Beyond mind-reading: Multi-voxel pattern analysis of fMRI data

    Trends in Cognitive Sciences

    (2006)
  • S. Offen et al.

    The role of early visual cortex in visual short-term memory and visual attention

    Vision Research

    (2009)
  • H.P. Op de Beeck

    Against hyperacuity in brain reading: Spatial smoothing does not hurt multivariate fMRI analyses?

    Neuroimage

    (2010)
  • F. Pereira et al.

    Machine learning classifiers and fMRI: A tutorial overview

    Neuroimage

    (2009)
  • B.R. Postle

    Working memory as an emergent property of the mind and brain

    Neuroscience

    (2006)
  • J.H. Reynolds et al.

    The normalization model of attention

    Neuron

    (2009)
  • J.H. Reynolds et al.

    Attention increases sensitivity of V4 neurons

    Neuron

    (2000)
  • J.T. Serences et al.

    Feature-based attentional modulations in the absence of direct visual stimulation

    Neuron

    (2007)
  • J.T. Serences et al.

    Attention and perceptual cohence fields

    Trends in Cognitive Sciences

    (2006)
  • D.Y. Teller

    Linking propositions

    Vision Research

    (1984)
  • R.A. Andersen et al.

    Cognitive neural prosthetics

    Annual Review of Psychology

    (2010)
  • G.M. Boynton

    Imaging orientation selectivity: Decoding conscious perception in V1

    Nature Neuroscience

    (2005)
  • G.M. Boynton et al.

    Linear systems analysis of functional magnetic resonance imaging in human V1

    Journal of Neuroscience

    (1996)
  • K.L. Briggman et al.

    Multifunctional pattern-generating circuits

    Annual Review of Neuroscience

    (2008)
  • G. Brindley

    Physiology of the retina and visual pathways

    (1960)
  • G.J. Brouwer et al.

    Decoding and reconstructing color from responses in human visual cortex

    The Journal of Neuroscience: The Official Journal of the Society for Neuroscience

    (2009)
  • Brouwer, G. J., & Heeger, D. J. (in press). Contrast suppression in human visual cortex. Journal of...
  • R.B. Buxton

    Introduction to functional magnetic resonance imaging: Principles and techniques

    (2002)
  • M. Carandini et al.

    Do we know what the early visual system does?

    The Journal of Neuroscience: The Official Journal of the Society for Neuroscience

    (2005)
  • B. Chai et al.

    Exploring functional connectivities of the human brain using multivariate information analysis

  • M.R. Cohen et al.

    Attention improves performance primarily by reducing interneuronal correlations

    Nature Neuroscience

    (2009)
  • M.R. Cohen et al.

    A neuronal population measure of attention predicts behavioral performance on individual trials

    The Journal of Neuroscience: The Official Journal of the Society for Neuroscience

    (2010)
  • M. Corbetta et al.

    Control of goal-directed and stimulus-driven attention in the brain

    Nature Reviews. Neuroscience

    (2002)
  • M. D’Esposito

    From cognitive to neural models of working memory

    Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences

    (2007)
  • S.V. David et al.

    Predicting neuronal responses during natural vision

    Network

    (2005)
  • Cited by (87)

    • The role of alpha oscillations in spatial attention: limited evidence for a suppression account

      2019, Current Opinion in Psychology
      Citation Excerpt :

      Recently, multivariate analysis techniques have enabled a more refined quantification of the spatial information present in alpha activity. We and others have used an inverted encoding model (IEM) [8–10] to track the spatial and temporal dynamics of covert attention [11••,12•]. This approach (Figure 1) assumes that alpha power at each electrode reflects the combined activity of a number of spatially selective channels (or neuronal populations).

    View all citing articles on Scopus
    View full text