Elsevier

NeuroImage

Volume 59, Issue 4, 15 February 2012, Pages 4064-4073
NeuroImage

Disentangling visual imagery and perception of real-world objects

https://doi.org/10.1016/j.neuroimage.2011.10.055Get rights and content

Abstract

During mental imagery, visual representations can be evoked in the absence of “bottom-up” sensory input. Prior studies have reported similar neural substrates for imagery and perception, but studies of brain-damaged patients have revealed a double dissociation with some patients showing preserved imagery in spite of impaired perception and others vice versa. Here, we used fMRI and multi-voxel pattern analysis to investigate the specificity, distribution, and similarity of information for individual seen and imagined objects to try and resolve this apparent contradiction. In an event-related design, participants either viewed or imagined individual named object images on which they had been trained prior to the scan. We found that the identity of both seen and imagined objects could be decoded from the pattern of activity throughout the ventral visual processing stream. Further, there was enough correspondence between imagery and perception to allow discrimination of individual imagined objects based on the response during perception. However, the distribution of object information across visual areas was strikingly different during imagery and perception. While there was an obvious posterior–anterior gradient along the ventral visual stream for seen objects, there was an opposite gradient for imagined objects. Moreover, the structure of representations (i.e. the pattern of similarity between responses to all objects) was more similar during imagery than perception in all regions along the visual stream. These results suggest that while imagery and perception have similar neural substrates, they involve different network dynamics, resolving the tension between previous imaging and neuropsychological studies.

Highlights

► Both scene and imagined object identity can be decoded. ► Information was differentially distributed for imagined and seen objects. ► The structure of representations was more similar during imagery than perception. ► Imagery and perception involve different dynamics across the ventral visual pathway.

Introduction

Our everyday visual perception reflects the interaction of externally driven “bottom-up” sensory information and internally generated “top-down” signals, which guide interpretation of sensory input (Hsieh et al., 2010, Kastner et al., 1998). However, even in the absence of bottom-up signals it is still possible to generate internal visual representations using top-down signals only, commonly referred to as mental imagery. Here we investigate the extent to which imagery shares the same neural substrate and mechanisms with perception in the ventral visual pathway (Farah, 1999, Kosslyn et al., 2001, Pylyshyn, 2002).

Prior studies using psychophysics (Ishai and Sagi, 1995, Pearson et al., 2008, Winawer et al., 2010), functional brain imaging (Ganis et al., 2004, Kosslyn et al., 1997) and intracranial recordings (Kreiman et al., 2000) have suggested similar mechanisms for imagery and perception. For example, the global pattern of brain activation in both imagery and perception is remarkably similar (Ganis et al., 2004, Kosslyn et al., 1997). Further, imagery can elicit category specific responses in high-level visual areas (Ishai et al., 2000, O'Craven and Kanwisher, 2000, Reddy et al., 2010) and retinotopically specific activity in primary visual cortex (Klein et al., 2004, Slotnick et al., 2005, Thirion et al., 2006). Finally, a recent study reported that the identity of one of two possible imagined stimuli (‘X’ or ‘O’) could be decoded from the pattern of response elicited by seeing those same stimuli in high-level object-selective cortex, suggesting overlap in the representations evoked during imagery and perception (Stokes et al., 2009, Stokes et al., 2011).

Yet, despite these similarities, imagery and perception are clearly distinct. Seeing and imagining are very different phenomenologically, and studies of brain-damaged individuals suggest that imagery and perception can be dissociated (Bartolomeo, 2002, Bartolomeo, 2008, Behrmann, 2000). For example, the object agnosic patient CK who has damage in the ventral visual pathway (Behrmann et al., 1992, Behrmann et al., 1994) is unable to recognize objects but can reproduce detailed drawings from memory and has preserved visual imagery. Conversely, deficits in visual imagery have been reported in the absence of agnosia, low-level perceptual deficits or disruption of imagery in other modalities (Farah et al., 1988, Moro et al., 2008). Thus, imagery and perception seem to share a neural substrate but are nevertheless dissociable. Therefore the critical question is what differentiates the utilization of the tissue by the two processes along the ventral visual pathway.

Here, we investigated the specificity (discrimination of objects), distribution (comparison of visual areas from V1 to high-level visual cortex), and differences between visual representations during perception and imagery of 10 individual real-world object images. While most previous studies have used stimuli (e.g. Gabor filters) or comparisons (e.g. category) tailored for specific regions of the ventral visual pathway, we ensured that perceptual decoding was possible across the entire pathway, allowing a systematic comparison of imagery and perception both within and across visual areas. We find that while there are similarities in the representations of seen and imagined objects, allowing decoding during imagery throughout the ventral pathway, there are critical differences. In particular, the distribution of information from V1 to high-level visual cortex showed opposite gradients. Further, the structure of representations (pattern of correlations between object images) across visual areas was more similar during imagery than during perception. Thus, while imagery and perception share at least some of the same neural substrate, they engage distinct neural mechanisms and involve different network dynamics along the ventral visual pathway, providing a potential resolution to the contradiction between prior imaging and neuropsychological studies.

Section snippets

Participants

Eleven neurologically intact, right-handed participants (5 males, 6 females, age 25 ± 1 years) took part in this study (2 additional participants were excluded due to a failure to localize early visual cortical regions). All participants provided written informed consent for the procedure in accordance with protocols approved by the NIH institutional review board.

Stimuli

We used ten images of common objects: bag, car, chair, clock, jet, lamp, necklace, pen, umbrella and violin (Fig. 1A). These objects

Results

Prior to scanning, participants were trained (see Material and methods) to remember and imagine the details of 10 full color object images (Fig. 1A). These object images were chosen to be distinct both retinotopically and categorically, maximizing the potential contribution of both early visual and object-selective cortex. The training required participants to process and retain as much detail from the object images as possible, so that their later imagery of the object images would be precise.

Discussion

In the present study, we used fMRI to conduct a detailed comparison of visual representations during imagery and perception. First, we examined the specificity of representations, and found that during both perception and imagery, the patterns of response in visual areas could be used to decode an individual seen or imagined object (out of 10 possible object images), though decoding was much weaker during imagery than perception. Further, there was enough correspondence between imagery and

Conclusions

In summary, our findings show that the relationship between imagery and perception is complex, with overlap in the neural substrates involved, but differences in how that tissue is utilized. We suggest that these differences directly result from the removal of bottom-up input during imagery and changes in the neural dynamics across regions.

Acknowledgments

This work was supported by the NIMH Intramural Research Program. Thanks to V. Elkis, S. Truong and J. Arizpe for help with data collection, Z. Saad for help with data analysis and A. Martin, M. Behrmann, A. Harel and members of the Laboratory of Brain and Cognition, NIMH for helpful comments and discussion.

References (50)

  • A. Ishai et al.

    Distributed neural systems for the generation of visual images

    Neuron

    (2000)
  • M.R. Johnson et al.

    A brief thought can modulate activity in extrastriate visual areas: top-down effects of refreshing just-seen visual stimuli

    Neuroimage

    (2007)
  • J.E. Joseph et al.

    Shared and dissociated cortical regions for object and letter processing

    Brain Res. Cogn. Brain Res.

    (2003)
  • I. Klein et al.

    Retinotopic organization of visual mental images as revealed by functional magnetic resonance imaging

    Brain Res. Cogn. Brain Res.

    (2004)
  • S.M. Kosslyn et al.

    Neural systems shared by visual imagery and visual perception: a positron emission tomography study

    Neuroimage

    (1997)
  • V. Moro et al.

    Selective deficit of mental visual imagery with intact primary visual cortex and visual perception

    Cortex

    (2008)
  • J. Pearson et al.

    The functional impact of mental imagery on conscious perception

    Curr. Biol.

    (2008)
  • L. Reddy et al.

    Category selectivity in the ventral visual pathway confers robustness to clutter and diverted attention

    Curr. Biol.

    (2007)
  • L. Reddy et al.

    Reading the mind's eye: decoding category information during mental imagery

    Neuroimage

    (2010)
  • M. Stokes et al.

    Imagery for shapes activates position-invariant representations in human visual cortex

    Neuroimage

    (2011)
  • B. Thirion et al.

    Inverse retinotopy: inferring the visual content of images from brain activation patterns

    Neuroimage

    (2006)
  • J. Winawer et al.

    A motion aftereffect from visual imagery of motion

    Cognition

    (2010)
  • M. Behrmann

    The mind's eye mapped onto the brain's matter

    Curr. Psychol. Sci.

    (2000)
  • M. Behrmann et al.

    Dissociation between mental imagery and object recognition in a brain-damaged patient

    Nature

    (1992)
  • M. Behrmann et al.

    Intact visual imagery and impaired visual perception in a patient with visual agnosia

    J. Exp. Psychol. Hum. Percept. Perform.

    (1994)
  • Cited by (0)

    No conflicts of interest.

    View full text