Disentangling visual imagery and perception of real-world objects☆
Highlights
► Both scene and imagined object identity can be decoded. ► Information was differentially distributed for imagined and seen objects. ► The structure of representations was more similar during imagery than perception. ► Imagery and perception involve different dynamics across the ventral visual pathway.
Introduction
Our everyday visual perception reflects the interaction of externally driven “bottom-up” sensory information and internally generated “top-down” signals, which guide interpretation of sensory input (Hsieh et al., 2010, Kastner et al., 1998). However, even in the absence of bottom-up signals it is still possible to generate internal visual representations using top-down signals only, commonly referred to as mental imagery. Here we investigate the extent to which imagery shares the same neural substrate and mechanisms with perception in the ventral visual pathway (Farah, 1999, Kosslyn et al., 2001, Pylyshyn, 2002).
Prior studies using psychophysics (Ishai and Sagi, 1995, Pearson et al., 2008, Winawer et al., 2010), functional brain imaging (Ganis et al., 2004, Kosslyn et al., 1997) and intracranial recordings (Kreiman et al., 2000) have suggested similar mechanisms for imagery and perception. For example, the global pattern of brain activation in both imagery and perception is remarkably similar (Ganis et al., 2004, Kosslyn et al., 1997). Further, imagery can elicit category specific responses in high-level visual areas (Ishai et al., 2000, O'Craven and Kanwisher, 2000, Reddy et al., 2010) and retinotopically specific activity in primary visual cortex (Klein et al., 2004, Slotnick et al., 2005, Thirion et al., 2006). Finally, a recent study reported that the identity of one of two possible imagined stimuli (‘X’ or ‘O’) could be decoded from the pattern of response elicited by seeing those same stimuli in high-level object-selective cortex, suggesting overlap in the representations evoked during imagery and perception (Stokes et al., 2009, Stokes et al., 2011).
Yet, despite these similarities, imagery and perception are clearly distinct. Seeing and imagining are very different phenomenologically, and studies of brain-damaged individuals suggest that imagery and perception can be dissociated (Bartolomeo, 2002, Bartolomeo, 2008, Behrmann, 2000). For example, the object agnosic patient CK who has damage in the ventral visual pathway (Behrmann et al., 1992, Behrmann et al., 1994) is unable to recognize objects but can reproduce detailed drawings from memory and has preserved visual imagery. Conversely, deficits in visual imagery have been reported in the absence of agnosia, low-level perceptual deficits or disruption of imagery in other modalities (Farah et al., 1988, Moro et al., 2008). Thus, imagery and perception seem to share a neural substrate but are nevertheless dissociable. Therefore the critical question is what differentiates the utilization of the tissue by the two processes along the ventral visual pathway.
Here, we investigated the specificity (discrimination of objects), distribution (comparison of visual areas from V1 to high-level visual cortex), and differences between visual representations during perception and imagery of 10 individual real-world object images. While most previous studies have used stimuli (e.g. Gabor filters) or comparisons (e.g. category) tailored for specific regions of the ventral visual pathway, we ensured that perceptual decoding was possible across the entire pathway, allowing a systematic comparison of imagery and perception both within and across visual areas. We find that while there are similarities in the representations of seen and imagined objects, allowing decoding during imagery throughout the ventral pathway, there are critical differences. In particular, the distribution of information from V1 to high-level visual cortex showed opposite gradients. Further, the structure of representations (pattern of correlations between object images) across visual areas was more similar during imagery than during perception. Thus, while imagery and perception share at least some of the same neural substrate, they engage distinct neural mechanisms and involve different network dynamics along the ventral visual pathway, providing a potential resolution to the contradiction between prior imaging and neuropsychological studies.
Section snippets
Participants
Eleven neurologically intact, right-handed participants (5 males, 6 females, age 25 ± 1 years) took part in this study (2 additional participants were excluded due to a failure to localize early visual cortical regions). All participants provided written informed consent for the procedure in accordance with protocols approved by the NIH institutional review board.
Stimuli
We used ten images of common objects: bag, car, chair, clock, jet, lamp, necklace, pen, umbrella and violin (Fig. 1A). These objects
Results
Prior to scanning, participants were trained (see Material and methods) to remember and imagine the details of 10 full color object images (Fig. 1A). These object images were chosen to be distinct both retinotopically and categorically, maximizing the potential contribution of both early visual and object-selective cortex. The training required participants to process and retain as much detail from the object images as possible, so that their later imagery of the object images would be precise.
Discussion
In the present study, we used fMRI to conduct a detailed comparison of visual representations during imagery and perception. First, we examined the specificity of representations, and found that during both perception and imagery, the patterns of response in visual areas could be used to decode an individual seen or imagined object (out of 10 possible object images), though decoding was much weaker during imagery than perception. Further, there was enough correspondence between imagery and
Conclusions
In summary, our findings show that the relationship between imagery and perception is complex, with overlap in the neural substrates involved, but differences in how that tissue is utilized. We suggest that these differences directly result from the removal of bottom-up input during imagery and changes in the neural dynamics across regions.
Acknowledgments
This work was supported by the NIMH Intramural Research Program. Thanks to V. Elkis, S. Truong and J. Arizpe for help with data collection, Z. Saad for help with data analysis and A. Martin, M. Behrmann, A. Harel and members of the Laboratory of Brain and Cognition, NIMH for helpful comments and discussion.
References (50)
- et al.
Negative BOLD differentiates visual imagery and perception
Neuron
(2005) The relationship between visual perception and visual mental imagery: a reappraisal of the neuropsychological evidence
Cortex
(2002)The neural correlates of visual mental imagery: an ongoing debate
Cortex
(2008)- et al.
Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex
Neuroimage
(2003) - et al.
Vividness of mental imagery: individual variability can be measured objectively
Vision Res.
(2007) - et al.
Untangling invariant object recognition
Trends Cogn. Sci.
(2007) - et al.
A case study of a mental imagery deficit
Brain Cogn.
(1988) - et al.
Brain areas underlying visual mental imagery and visual perception: an fMRI study
Brain Res. Cogn. Brain Res.
(2004) - et al.
The lateral occipital complex and its role in object recognition
Vision Res.
(2001) - et al.
Accurate prediction of V1 location from cortical folds in a surface coordinate system
Neuroimage
(2008)
Distributed neural systems for the generation of visual images
Neuron
A brief thought can modulate activity in extrastriate visual areas: top-down effects of refreshing just-seen visual stimuli
Neuroimage
Shared and dissociated cortical regions for object and letter processing
Brain Res. Cogn. Brain Res.
Retinotopic organization of visual mental images as revealed by functional magnetic resonance imaging
Brain Res. Cogn. Brain Res.
Neural systems shared by visual imagery and visual perception: a positron emission tomography study
Neuroimage
Selective deficit of mental visual imagery with intact primary visual cortex and visual perception
Cortex
The functional impact of mental imagery on conscious perception
Curr. Biol.
Category selectivity in the ventral visual pathway confers robustness to clutter and diverted attention
Curr. Biol.
Reading the mind's eye: decoding category information during mental imagery
Neuroimage
Imagery for shapes activates position-invariant representations in human visual cortex
Neuroimage
Inverse retinotopy: inferring the visual content of images from brain activation patterns
Neuroimage
A motion aftereffect from visual imagery of motion
Cognition
The mind's eye mapped onto the brain's matter
Curr. Psychol. Sci.
Dissociation between mental imagery and object recognition in a brain-damaged patient
Nature
Intact visual imagery and impaired visual perception in a patient with visual agnosia
J. Exp. Psychol. Hum. Percept. Perform.
Cited by (0)
- ☆
No conflicts of interest.