Elsevier

NeuroImage

Volume 162, 15 November 2017, Pages 322-343
NeuroImage

Intracranial markers of conscious face perception in humans

https://doi.org/10.1016/j.neuroimage.2017.08.074Get rights and content

Highlights

  • ECoG study combining a masked and an unmasked tasks.

  • We decoded physical properties of stimuli or perceptual states from spectrograms.

  • Both lateral and ventral areas discriminate seen vs. unseen faces in the masked task.

  • Both discriminate upright faces vs. other categories in the unmasked task.

  • Only ventral areas discriminate upright vs. inverted faces in the unmasked task.

Abstract

Investigations of the neural basis of consciousness have greatly benefited from protocols that involve the presentation of stimuli at perceptual threshold, enabling the assessment of the patterns of brain activity that correlate with conscious perception, independently of any changes in sensory input. However, the comparison between perceived and unperceived trials would be expected to reveal not only the core neural substrate of a particular conscious perception, but also aspects of brain activity that facilitate, hinder or tend to follow conscious perception. We take a step towards the resolution of these confounds by combining an analysis of neural responses observed during the presentation of faces partially masked by Continuous Flash Suppression, and those responses observed during the unmasked presentation of faces and other images in the same subjects. We employed multidimensional classifiers to decode physical properties of stimuli or perceptual states from spectrotemporal representations of electrocorticographic signals (1071 channels in 5 subjects). Neural activity in certain face responsive areas located in both the fusiform gyrus and in the lateral-temporal/inferior-parietal cortex discriminated seen vs. unseen faces in the masked paradigm and upright faces vs. other categories in the unmasked paradigm. However, only the former discriminated upright vs. inverted faces in the unmasked paradigm. Our results suggest a prominent role for the fusiform gyrus in the configural perception of faces, and possibly other objects that are holistically processed. More generally, we advocate comparative analysis of neural recordings obtained during different, but related, experimental protocols as a promising direction towards elucidating the functional specificities of the patterns of neural activation that accompany our conscious experiences. Keywords: Consciousness, Visual awareness, Face perception, ECoG, Brain decoding, Visual masking

Introduction

In the last couple of decades, the relationships between brain activity and the contents of perceptual consciousness have been investigated using a variety of experimental techniques operating at different spatial and temporal scales, from single-unit, multi-unit and local field potential recordings in monkeys (Logothetis and Schall, 1989, Leopold and Logothetis, 1996, Wilke et al., 2006, Wilke et al., 2009, Maier et al., 2007), to non-invasive neuroimaging techniques such as EEG, MEG and fMRI in humans (e.g., Tong et al., 1998, Grill-Spector et al., 2000, Dehaene et al., 2001, van Aalderen-Smeets et al., 2006, Liu et al., 2012, Schurger et al., 2015)) (see (Rees et al., 2002, Tononi and Koch, 2008, Dehaene and Changeux, 2011, Boly et al., 2013, Panagiotaropoulos et al., 2014) for reviews).

The scientific investigation of perceptual states presents a unique challenge, since it requires the objective measurement of subjective states. In particular, accuracy in reports of subjective states is a critical prerequisite for this investigation. With sufficient amount of training and careful experimental design (Leopold et al., 2003), monkeys (and potentially other animals) can be trained to report their perceptual states in a reliable manner (see, for example, (Leopold and Logothetis, 1996)). However, the investigation of the neural correlates of conscious awareness in human subjects constitutes a great advantage, since they can provide accurate reports of their perceptual states with minimal training following verbal instructions from the experimenter. This is critical, especially if graded levels of perceptual awareness are considered, as in the current study.

In humans, non-invasive neural recordings have been extensively employed in the search of the neural correlates of consciousness. Here, we recorded electrocorticography (ECoG) from subdural electrodes implanted on the ventral and lateral surface of the temporal lobes in five epileptic patients undergoing pre-surgical seizure monitoring while they engaged in visual perception tasks. Intracranial recordings from human subjects undergoing pre-surgical monitoring constitute a precious opportunity to advance our understanding of the neural correlates of conscious perception (e.g. (Kreiman et al., 2002, Gaillard et al., 2009, Fisch et al., 2009, Aru et al., 2012a, Willenbockel et al., 2012, Quiroga et al., 2014), see (Lachaux et al., 2003, Engel et al., 2005, Jacobs and Kahana, 2010, Mukamel and Fried, 2012) for reviews), due to the direct measure of electrophysiological responses as well as their high spatial and temporal resolution in comparison with non-invasive modalities. In particular, the high temporal resolution afforded by ECoG recordings enables one to accurately assess high-frequency broadband neural activity (>70 Hz), which has been shown to be highly informative and more related to neuronal spiking activity than lower frequency bands (e.g., (Manning et al., 2009, Ray and Maunsell, 2011, Miller et al., 2014)), while much less contaminated by non-neural (e.g., muscular) activity in comparison with non-invasive recording modalities.

Several techniques have been proposed to investigate the neural correlates of conscious visual perception (Kim and Blake, 2005). These techniques enable the dissociation between retinal images and subjective perception.

Previous intracranial recording studies have investigated the neural correlates of conscious visual perception using stimuli that are perceptually degraded by a technique known as Backward Masking (BM) (Gaillard et al., 2009, Fisch et al., 2009, Quiroga et al., 2008). In a typical BM paradigm, a target image is presented briefly, followed by a masking image after a variable delay, known as Stimulus Onset Asynchrony (SOA). Short SOAs prevent the target image from being consciously perceived, while long SOAs allow the target image to emerge to consciousness reliably. At intermediate SOAs, conscious visibility fluctuates across trials. While most BM studies investigated the neural correlates of consciousness by comparing trials that differed markedly in either stimulus configuration (e.g., (Dehaene et al., 2001)) or other covariates, such as subject training (e.g., (Grill-Spector et al., 2000)), some recent studies aimed to more subtle contrasts that could more specifically expose the neural correlates of consciousness (Gaillard et al., 2009, Fisch et al., 2009). However, even these latter studies compared visible and invisible conditions in response to similar, but not identical, input stimuli, due to the experimental difficulty of adjusting SOA at perceptual threshold (but see (Quiroga et al., 2008, Del Cul et al., 2007) for examples where the contrast between seen and unseen targets at threshold SOA was possible for a subset of subjects). Thus, studies using BM may generally confound neural activity related to different perceptual outcome with neural activity related to different visual stimulation.

Here, we employed a different masking technique, known as Continuous Flash Suppression (CFS). This technique is based on the presentation of rapidly changing Mondrian patterns to one eye, while a static image (the target) is presented to the other eye ((Tsuchiya and Koch, 2005), see (Yang et al., 2014, Sterzer et al., 2014) for recent reviews). Depending on the contrast of the target image and the Mondrian masks, the target image can be completely invisible, clearly visible, or visible only in a subset of the trials. The latter condition is of special interest, since the contrast between neural activity corresponding to trials with different visibility outcomes, in conditions of equal stimulus contrast, enables us to assess the neural correlates of visibility in the absence of any change in the physical properties of the stimulus.

Even when comparing trials corresponding to identical physical stimuli, but different perceptual outcomes, the resulting differences cannot be unambiguously considered as core neural correlates of phenomenal conscious perception, or NCC-core (Aru et al., 2012b, de Graaf et al., 2012, Miller, 2007, van Boxtel and Tsuchiya, 2015, Tsuchiya et al., 2015), because they likely reflect additional processes that also differ between the conditions. In fact, the comparison between seen and unseen trials can also reveal brain states that facilitate (e.g., attentional mechanisms) or hinder (e.g., mind-wandering) the perceptual awareness of threshold stimuli. In addition to this, conscious visual perception of target stimuli can trigger a cascade of neural processes related to memory formation, generation of associations and motor preparation for the ensuing response. Aru et al. and de Graaf et al. conveniently termed the potential confounds belonging to the former category as NCC-prs, or prerequisites, and the latter as NCC-cos, or consequences, of the conscious perceptual experience.

In this work, we take a step towards the dissociation between the neural correlates of core aspects of conscious visual experience (NCC-core) and their prerequisites and consequences by combining different, albeit related, experimental protocols. In particular, we considered a masked visual task, where stimuli were made partially invisible by Continuous Flash Suppression (CFS); and an unmasked visual task, where stimuli were clearly visible. Each of these tasks expose different cognitive processes: while the partially masked visual targets in the CFS task seemed to require some effort to be seen, unmasked images were clearly seen without effort.

Importantly, some of the stimuli used in the unmasked task (photographs of human faces) belong to the same category as the target stimuli in the masked task. Human faces constitute a stimulus category of exceptional behavioral and ecological relevance, and are known to be processed in specific circuits in ventral and lateral regions of the temporal lobe, most evidently in the Fusiform Gyrus (FG) and in the Superior Temporal Sulcus (STS) (Allison et al., 1994, Puce et al., 1995, Kanwisher et al., 1997, Haxby et al., 2000, Kanwisher and Yovel, 2006, Tsuchiya et al., 2008, Kawasaki et al., 2012).

Critically, the image categories used in the unmasked task comprise inverted faces in addition to upright faces and non-face objects such as houses and tools. The comparison between neural activity in response to upright versus inverted faces is expected to reveal features of neural processing that are specific to configural or holistic perception, that is, a gestalt perception where the whole face is perceived in a qualitatively different manner from the sum of its parts (e.g., (Rossion and Gauthier, 2002)). This phenomenon can be measured behaviorally, for example via reaction times in recognition (e.g. same/different judgment) tasks.

The comparison between specific neural markers in the masked and unmasked tasks enables one to discard neural markers that could otherwise be considered as putative NCC-core if only the contrast between visible and invisible trials at threshold were considered. More generally, this work paves the way for a new promising set of methodological approaches in consciousness research based on the comparison between similar experimental protocols, which differ in specific aspects that expose the key differences that enable one to disentangle the different aspects of the conscious visual experience.

Section snippets

Data acquisition

We recorded intracranially with electrocorticographic (ECoG) electrodes from 5 epilepsy patients undergoing pre-surgical monitoring. Sampling rate for the ECoG signal was 2034.5 Hz. Electrode location was based solely on clinical criteria. Patient age, gender, handedness, ocular and language dominance, and locations of seizure foci are reported in Table 1. We did not record for 12 h after any generalized seizure event. We did not perform any explicit artifact rejection. Voltage traces from each

Neural responses to objective and subjective attributes of visual stimuli

Our electrophysiological data set comprised 1071 bipolar channels (187, 171, 219, 228, 266 from subject 147, 153, 154, 168, 178) from the ventral and lateral temporal cortex of 5 subjects. Out of these, 271 channels were face-responsive (82, 57, 53, 53, 26 from subject 147, 153, 154, 168, 178, see subsection “Face responsiveness” in Methods for the definition of face responsiveness).

A subset of face-responsive channels exhibited spectral power responses that differed between categories in the

Discussion

Traditionally, the closely related aspects of conscious face perception (e.g., contrasting a visible with an invisible face) and holistic face perception (e.g., contrasting an upright with an inverted face) have been assessed separately. However, the search for the neural correlates of consciousness consists in assessing the patterns of neural activation that covary with the phenomenal experience, regardless of the precise experimental protocol employed. By combining similar, but different,

Conclusion

We have combined a set of decoding analyses of either physical properties of stimuli or subjective phenomenology in a masked task, with a set of decoding analyses of stimulus category in an unmasked task. Our results in the masked task show that subjective phenomenology is better decodable than physical properties of stimuli (such as contrast) in the high level visual areas considered in this study, i.e. the ventral and lateral sides of the temporal lobe.

While we were able to decode subjective

Competing interests

NT collaborates with Dr Ryota Kanai, CEO of a venture company, called Araya Brain Imaging (araya.org) based in Tokyo, Japan. In October 2015, Araya received a grant from Japan Science and Technology (JST; approximately 3 million yen for 5 years), which aims to potentially commercial projects related to artificial consciousness. NT is part of this grant and he is supposed to host several students/research assistants/postdocs hired at Araya until March 2020. However, his institute (Monash

Author contributions

Conceived the idea of the study: FB NT. Conceived and designed the experiments: HK CKK MAH RA NT. Performed neurosurgery: HK MAH. Collected the data: HK CKK NT. Conducted preliminary analyses including anatomical localization of electrodes: HK HO NT. Analyzed the data: FB JvK NT. Wrote the manuscript: FB NT. Commented critically on the manuscript: FB JvK HK CKK HO MAH RA NT.

Acknowledgments

We thank all subjects for their participation in the study, the medical team at University of Iowa Hospital for their assistance, Haiming Chen for his technical assistance. We thank Andrew Haun for generating Fig. 1. NT is supported by Australian Research Council (ARC) Future Fellowship (FT120100619) and Discovery Project (DP130100194). NT and FB were supported by Japan Science and Technology Agency (JST) PRESTO fellowship 3630 to NT. CKK, HK, HO, and MAH were supported by NIH grants R01

References (160)

  • C. Destrieux et al.

    Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature

    NeuroImage

    (2010)
  • J. Dubois et al.

    Building a science of individual differences from fMRI

    Trends cognitive Sci.

    (2016)
  • J.H. Duyn

    The future of ultra-high field MRI and fMRI for study of the human brain

    NeuroImage

    (2012)
  • A.K. Engel et al.

    Temporal binding, binocular rivalry, and consciousness

    Conscious. Cognit.

    (1999)
  • A.D. Engell et al.

    Facial expression and gaze-direction in human superior temporal sulcus

    Neuropsychologia

    (2007)
  • L. Fisch et al.

    Neural “ignition”: enhanced activation linked to perceptual awareness in human ventral stream visual cortex

    Neuron

    (2009)
  • M.A. Frost et al.

    Measuring structural-functional correspondence: spatial variability of specialised brain regions after macro-anatomical alignment

    NeuroImage

    (2012)
  • M.A. Frost et al.

    Functionally informed cortex based alignment: an integrated approach for whole-cortex macro-anatomical and ROI-based functional alignment

    NeuroImage

    (2013)
  • J.V. Haxby et al.

    The distributed human neural system for face perception

    Trends cognitive Sci.

    (2000)
  • J.V. Haxby et al.

    A common, high-dimensional model of the representational space in human ventral temporal cortex

    Neuron

    (2011)
  • J. Jacobs et al.

    Direct brain recordings fuel advances in cognitive electrophysiology

    Trends cognitive Sci.

    (2010)
  • C.Y. Kim et al.

    Psychophysical magic: rendering the visible 'invisible'

    Trends Cognit. Sci.

    (2005)
  • C.K. Kovach et al.

    Manifestation of ocular-muscle EMG contamination in human intracranial recordings

    NeuroImage

    (2011)
  • J.P. Lachaux et al.

    Intracranial EEG and human brain mapping

    J. Physiol. Paris

    (2003)
  • J.P.P. Lachaux et al.

    High-frequency neural activity and human cognition: past, present and possible future of intracranial EEG research

    Prog. Neurobiol.

    (2012)
  • T. Landis et al.

    Are unilateral right posterior cerebral lesions sufficient to cause prosopagnosia? Clinical and radiological findings in six additional patients

    Cortex

    (1986)
  • S.L. Macknik

    Visual masking approaches to visual awareness

    Prog. brain Res.

    (2006)
  • B. Maniscalco et al.

    A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings

    Conscious. cognition

    (2012)
  • D. Maurer et al.

    The many faces of configural processing

    Trends cognitive Sci.

    (2002)
  • K.J. Miller et al.

    Broadband changes in the cortical surface potential track activation of functionally diverse neuronal populations

    NeuroImage

    (2014)
  • M.B. Miller et al.

    Individual variability in brain activations associated with episodic retrieval: a role for large-scale databases

    Int. J. Psychophysiol.

    (2007)
  • M.B. Ahrens et al.

    Whole-brain functional imaging at cellular resolution using light-sheet microscopy

    Nat. Methods

    (2013)
  • T. Allison et al.

    Face recognition in human extrastriate cortex

    J. Neurophysiol.

    (1994)
  • T. Allison et al.

    Electrophysiological studies of human face perception. I: potentials generated in occipitotemporal cortex by face and non-face stimuli

    Cereb. Cortex

    (1999)
  • J. Almeida et al.

    Unconscious processing dissociates along categorical lines

    Proc. Natl. Acad. Sci. U. S. A.

    (2008)
  • S. Anzellotti et al.

    From parts to identity: invariance and sensitivity of face representations to different face halves

    Cereb. Cortex

    (2015)
  • S. Anzellotti et al.

    Decoding representations of face identity that are tolerant to rotation

    Cereb. Cortex

    (2014)
  • J. Aru et al.

    Local category-specific gamma band responses in the visual cortex do not reflect conscious perception

    J. Neurosci.

    (2012)
  • A.P. Atkinson et al.

    The neuropsychology of face perception: beyond simple dissociations and functional selectivity

    Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci.

    (2011)
  • G. Avidan et al.

    Detailed exploration of face-related processing in congenital prosopagnosia: 2. Functional neuroimaging findings

    J. Cognitive Neurosci.

    (2005)
  • V. Axelrod et al.

    Successful decoding of famous faces in the fusiform face area

    PloS One

    (2015)
  • A.K. Bansal et al.

    Temporal stability of visually selective responses in intracranial field potentials recorded from human occipital and temporal lobes

    J. Neurophysiol.

    (2012)
  • A.B. Barrett et al.

    Measures of metacognition on signal-detection theoretic models

    Psychol. Methods

    (2013)
  • Y. Benjamini et al.

    Controlling the false Discovery rate: a practical and powerful approach to multiple testing

    J. R. Stat. Soc. Ser. B Methodol.

    (1995)
  • M. Bilalić et al.

    Many faces of expertise: fusiform face area in chess experts and novices

    J. Neurosci.

    (2011)
  • M. Boly et al.

    Consciousness in humans and non-human animals: recent advances and future directions

    Front. Psychol.

    (2013)
  • D.H. Brainard

    The psychophysics toolbox

    Spat. Vis.

    (1997)
  • G. Buzsáki et al.

    The origin of extracellular fields and currents EEG, ECoG, LFP and spikes

    Nat. Rev. Neurosci.

    (2012)
  • A.M. Chan et al.

    First-pass selectivity for semantic categories in human anteroventral temporal lobe

    J. Neurosci.

    (2011)
  • A.W.Y. Chan et al.

    An exploration of face selectivity in human inferior frontal cortex

    J. Vis.

    (2006)
  • Cited by (0)

    View full text