Trends in Cognitive Sciences
ReviewDecoding human brain activity during real-world experiences
Introduction
Understanding how our brain makes sense of continuous and complex inputs from the external world represents a central challenge in cognitive neuroscience. Functional neuroimaging offers a means to address this key issue, but, because many studies use simplified static stimuli, surprisingly little is known about how the human brain operates during real-world experiences. Exploring brain function with dynamic naturalistic stimuli is important for several reasons. First, it is vital to verify whether results obtained in experiments that used simplified stimuli actually hold true under natural conditions, particularly because findings are often assumed to generalize. For example, do brain regions such as the fusiform face area also show selectivity when faces appear in the real world? Second, some research questions can only be addressed with naturalistic tasks where there is little temporal regularity. To understand properly the neural substrates of driving a vehicle or navigating in a city, for instance, it is sub-optimal to use static regularized stimuli.
The need to complement highly controlled experimental manipulations has been acknowledged in the field of functional neuroimaging; consequently, the use of dynamic stimuli such as movies and virtual reality environments is increasing 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18. However, most extant studies involving naturalistic stimuli have designs where activity associated with each task (e.g. navigation or a control task) was averaged across blocks of typically 30–60 s duration (Figure 1a). This approach not only lacks fine-grained temporal resolution but also reduces the correspondence to the real world, which is rarely organized in a ‘blocked’ and orderly manner. For a true appreciation of brain function during real-world experiences, one key element that it is vital to understand is how to segregate neural activity during specific events from the continuous stream of complex stimulation of which they are a part. However, this presents a significant challenge because the lack of discrete stimuli means that standard experimental designs and analyses cannot be used (Figure 1c). Dynamic, continuous stimuli can evoke both transient and sustained responses within the same brain region or in different brain regions simultaneously, making data interpretation difficult. The unconstrained nature of eye-movements, multiple features in any given scene, and the lack of a specific task can give rise to ambiguity about what subjects were attending to or thinking about during the experience.
Recently, however, researchers have adopted a range of innovative approaches to analysing fMRI data collected during naturalistic tasks. The analytical methods tend to fall within three broad categories: (i) those based on subjects’ classification of events; (ii) those based on subjects’ behaviour and verbal reports; and (3) stimulus ‘blind’ analyses, which in general do not require information about the stimuli to be known before the analysis. Several recent reviews have already discussed one of the analysis methods from the third category, namely multi-voxel pattern analysis (see Refs 19, 20). By contrast our focus here is different. First, we widen the scope considerably to explore work from the first two categories, as well as additional stimulus ‘blind’ approaches. Second, our interest is in how these innovations have been applied specifically to the analysis of fMRI data acquired during dynamic naturalistic tasks.
We will consider each category of analytical method in turn, reviewing the key studies. Although a diversity of cognitive domains and scientific agendas are represented, two common themes are evident. Initial studies have focussed on investigating whether patterns of brain activity observed previously using experimental stimuli are mirrored during naturalistic tasks. However, it is also apparent that experience with the techniques is growing, and the ability to combine these methods with naturalistic stimuli is beginning to permit novel insights that would be difficult to gain using more traditional approaches. Future developments, therefore, might lead to genuine conceptual advances. Given this potential, it is timely to consider the different techniques used, and their advantages and disadvantages, in order that researchers might identify new opportunities for experimental investigations involving naturalistic contexts that could add a new dimension to their work.
Section snippets
Analysis using subjects’ classification of events
We begin with arguably one of the most straightforward means of analysing naturalistic stimuli. Two recent studies examined brain activity during the passive viewing of commercial movies 21, 22. To identify moments in the movie when stimuli of interest (e.g. faces, voices and colours [21]) or humorous events [22] occurred, a separate group of subjects watched the movie and recorded these events. This record was incorporated into the analysis to examine the brain activity of subjects who were
Virtual reality, content analysis and verbal reports
Although studies that involve watching movies are beginning to advance our knowledge of brain dynamics, passive viewing remains distinct from much of our everyday activities, which generally involve engaging with the world around us. Trying to ascertain the neural correlates of real-world interactions represents a huge challenge, given the physical constraints of the MRI scanning environment (where a subject's head is immobilized in the bore of the scanner), and the variability in behaviour
Stimulus ‘blind’ analyses
Even when stimuli, a subject's behaviour and their verbal reports are analysed in detail, some patterns in an fMRI time series might still remain undetected. Several of the studies described so far, and indeed many fMRI datasets in general, are analysed using programmes such as statistical parametric mapping (SPM). In SPM, a linear combination of the effects of interest (e.g. the events) plus a residual error are used to model the data and test for significant relationships between the brain
Summary
Here we described how researchers have recently applied several innovative methods to explore brain responses measured with fMRI during naturalistic tasks. These methods involve either a stimulus-driven approach (using stimulus classifications, behaviour or verbal reports typically analysed with techniques such as SPM), or a stimulus ‘blind’ approach (e.g. ICA, reverse correlations, and MVPA, to extract hidden patterns in the fMRI signal). Although relatively few studies have been conducted so
Conclusions and future directions
The human brain evolved to function and survive in a highly stimulating, complex and fast-changing world. Attempting to ascertain the neural substrates of operating in naturalistic contexts represents a huge challenge. One productive approach has been to examine instead simplified or abstracted stimuli in controlled fMRI experimental designs. However, important insights might be missed by not examining thoughts and behaviours in the real-world setting where they typically take place. The new
Acknowledgements
H.J.S. and E.A.M. are supported by the Wellcome Trust.
References (70)
The well-worn route and the path less traveled: distinct neural bases of route following and wayfinding in humans
Neuron
(2003)Doing the right thing: a common neural circuit for appropriate violent or compassionate behavior
Neuroimage
(2006)Anterior prefrontal involvement in episodic retrieval reflects contextual interference
Neuroimage
(2005)A temporoparietal and prefrontal network for retrieving the spatial context of lifelike events
Neuroimage
(2001)Seeing or doing? Influence of visual and motor familiarity in action observation
Curr. Biol.
(2006)Watching social interactions produces dorsomedial prefrontal and medial parietal BOLD fMRI signal increases compared to a resting baseline
Neuroimage
(2004)A region of right posterior superior temporal sulcus responds to observed intentional actions
Neuropsychologia
(2004)Dynamics of male sexual arousal: distinct components of brain activation revealed by fMRI
Neuroimage
(2005)An interference effect of observed biological movement on action
Curr. Biol.
(2003)Inferring false beliefs from the actions of oneself and others: an fMRI study
Neuroimage
(2004)
Neural correlates of feeling sympathy
Neuropsychologia
Distinct neural substrates for the perception of real and virtual visual worlds
Neuroimage
Beyond mind-reading: multi-voxel pattern analysis of fMRI data
Trends Cogn. Sci.
Neural correlates of humor detection and appreciation
Neuroimage
Thoughts, behaviour, and brain dynamics during navigation in the real world
Neuroimage
Spontaneous mentalizing during an interactive real world task: an fMRI study
Neuropsychologia
Neural substrates of driving behaviour
Neuroimage
The neural correlates of driving performance identified using positron emission tomography
Brain Cogn.
Cognitive and emotional influences in anterior cingulate cortex
Trends Cogn. Sci.
Characterizing evoked hemodynamics with fMRI
Neuroimage
Analysis of fMRI time-series revisited
Neuroimage
Brain dynamics during natural viewing conditions–a new guide for mapping connectivity in vivo
Neuroimage
Towards natural stimulation in fMRI—issues of data analysis
Neuroimage
Unified SPM-ICA for fMRI analysis
Neuroimage
Predicting the stream of consciousness from activity in human visual cortex
Curr. Biol.
Reading hidden intentions in the human brain
Curr. Biol.
Decoding seen and attended motion directions from activity in the human visual cortex
Curr. Biol.
Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex
Neuroimage
Dynamic causal modelling
Neuroimage
Imaging the intentional stance in a competitive game
Neuroimage
Knowing where and getting there: a human navigation network
Science
Cognitive strategies dependent on the hippocampus and caudate nucleus in human navigation: variability and change with practice
J. Neurosci.
Action observation and acquired motor skills: an FMRI study with expert dancers
Cereb. Cortex
When strangers pass: processing of mutual and averted social gaze in the superior temporal sulcus
Psychol. Sci.
Brain activation and sexual arousal in healthy, heterosexual males
Brain
Cited by (131)
Functional organization of the caudal part of the human superior parietal lobule
2023, Neuroscience and Biobehavioral ReviewsDoes memory research have a realistic future?
2022, Trends in Cognitive SciencesThe Treachery of Images: How Realism Influences Brain and Behavior
2021, Trends in Cognitive Sciences