Intracranial markers of conscious face perception in humans
Graphical abstract
Introduction
In the last couple of decades, the relationships between brain activity and the contents of perceptual consciousness have been investigated using a variety of experimental techniques operating at different spatial and temporal scales, from single-unit, multi-unit and local field potential recordings in monkeys (Logothetis and Schall, 1989, Leopold and Logothetis, 1996, Wilke et al., 2006, Wilke et al., 2009, Maier et al., 2007), to non-invasive neuroimaging techniques such as EEG, MEG and fMRI in humans (e.g., Tong et al., 1998, Grill-Spector et al., 2000, Dehaene et al., 2001, van Aalderen-Smeets et al., 2006, Liu et al., 2012, Schurger et al., 2015)) (see (Rees et al., 2002, Tononi and Koch, 2008, Dehaene and Changeux, 2011, Boly et al., 2013, Panagiotaropoulos et al., 2014) for reviews).
The scientific investigation of perceptual states presents a unique challenge, since it requires the objective measurement of subjective states. In particular, accuracy in reports of subjective states is a critical prerequisite for this investigation. With sufficient amount of training and careful experimental design (Leopold et al., 2003), monkeys (and potentially other animals) can be trained to report their perceptual states in a reliable manner (see, for example, (Leopold and Logothetis, 1996)). However, the investigation of the neural correlates of conscious awareness in human subjects constitutes a great advantage, since they can provide accurate reports of their perceptual states with minimal training following verbal instructions from the experimenter. This is critical, especially if graded levels of perceptual awareness are considered, as in the current study.
In humans, non-invasive neural recordings have been extensively employed in the search of the neural correlates of consciousness. Here, we recorded electrocorticography (ECoG) from subdural electrodes implanted on the ventral and lateral surface of the temporal lobes in five epileptic patients undergoing pre-surgical seizure monitoring while they engaged in visual perception tasks. Intracranial recordings from human subjects undergoing pre-surgical monitoring constitute a precious opportunity to advance our understanding of the neural correlates of conscious perception (e.g. (Kreiman et al., 2002, Gaillard et al., 2009, Fisch et al., 2009, Aru et al., 2012a, Willenbockel et al., 2012, Quiroga et al., 2014), see (Lachaux et al., 2003, Engel et al., 2005, Jacobs and Kahana, 2010, Mukamel and Fried, 2012) for reviews), due to the direct measure of electrophysiological responses as well as their high spatial and temporal resolution in comparison with non-invasive modalities. In particular, the high temporal resolution afforded by ECoG recordings enables one to accurately assess high-frequency broadband neural activity (>70 Hz), which has been shown to be highly informative and more related to neuronal spiking activity than lower frequency bands (e.g., (Manning et al., 2009, Ray and Maunsell, 2011, Miller et al., 2014)), while much less contaminated by non-neural (e.g., muscular) activity in comparison with non-invasive recording modalities.
Several techniques have been proposed to investigate the neural correlates of conscious visual perception (Kim and Blake, 2005). These techniques enable the dissociation between retinal images and subjective perception.
Previous intracranial recording studies have investigated the neural correlates of conscious visual perception using stimuli that are perceptually degraded by a technique known as Backward Masking (BM) (Gaillard et al., 2009, Fisch et al., 2009, Quiroga et al., 2008). In a typical BM paradigm, a target image is presented briefly, followed by a masking image after a variable delay, known as Stimulus Onset Asynchrony (SOA). Short SOAs prevent the target image from being consciously perceived, while long SOAs allow the target image to emerge to consciousness reliably. At intermediate SOAs, conscious visibility fluctuates across trials. While most BM studies investigated the neural correlates of consciousness by comparing trials that differed markedly in either stimulus configuration (e.g., (Dehaene et al., 2001)) or other covariates, such as subject training (e.g., (Grill-Spector et al., 2000)), some recent studies aimed to more subtle contrasts that could more specifically expose the neural correlates of consciousness (Gaillard et al., 2009, Fisch et al., 2009). However, even these latter studies compared visible and invisible conditions in response to similar, but not identical, input stimuli, due to the experimental difficulty of adjusting SOA at perceptual threshold (but see (Quiroga et al., 2008, Del Cul et al., 2007) for examples where the contrast between seen and unseen targets at threshold SOA was possible for a subset of subjects). Thus, studies using BM may generally confound neural activity related to different perceptual outcome with neural activity related to different visual stimulation.
Here, we employed a different masking technique, known as Continuous Flash Suppression (CFS). This technique is based on the presentation of rapidly changing Mondrian patterns to one eye, while a static image (the target) is presented to the other eye ((Tsuchiya and Koch, 2005), see (Yang et al., 2014, Sterzer et al., 2014) for recent reviews). Depending on the contrast of the target image and the Mondrian masks, the target image can be completely invisible, clearly visible, or visible only in a subset of the trials. The latter condition is of special interest, since the contrast between neural activity corresponding to trials with different visibility outcomes, in conditions of equal stimulus contrast, enables us to assess the neural correlates of visibility in the absence of any change in the physical properties of the stimulus.
Even when comparing trials corresponding to identical physical stimuli, but different perceptual outcomes, the resulting differences cannot be unambiguously considered as core neural correlates of phenomenal conscious perception, or NCC-core (Aru et al., 2012b, de Graaf et al., 2012, Miller, 2007, van Boxtel and Tsuchiya, 2015, Tsuchiya et al., 2015), because they likely reflect additional processes that also differ between the conditions. In fact, the comparison between seen and unseen trials can also reveal brain states that facilitate (e.g., attentional mechanisms) or hinder (e.g., mind-wandering) the perceptual awareness of threshold stimuli. In addition to this, conscious visual perception of target stimuli can trigger a cascade of neural processes related to memory formation, generation of associations and motor preparation for the ensuing response. Aru et al. and de Graaf et al. conveniently termed the potential confounds belonging to the former category as NCC-prs, or prerequisites, and the latter as NCC-cos, or consequences, of the conscious perceptual experience.
In this work, we take a step towards the dissociation between the neural correlates of core aspects of conscious visual experience (NCC-core) and their prerequisites and consequences by combining different, albeit related, experimental protocols. In particular, we considered a masked visual task, where stimuli were made partially invisible by Continuous Flash Suppression (CFS); and an unmasked visual task, where stimuli were clearly visible. Each of these tasks expose different cognitive processes: while the partially masked visual targets in the CFS task seemed to require some effort to be seen, unmasked images were clearly seen without effort.
Importantly, some of the stimuli used in the unmasked task (photographs of human faces) belong to the same category as the target stimuli in the masked task. Human faces constitute a stimulus category of exceptional behavioral and ecological relevance, and are known to be processed in specific circuits in ventral and lateral regions of the temporal lobe, most evidently in the Fusiform Gyrus (FG) and in the Superior Temporal Sulcus (STS) (Allison et al., 1994, Puce et al., 1995, Kanwisher et al., 1997, Haxby et al., 2000, Kanwisher and Yovel, 2006, Tsuchiya et al., 2008, Kawasaki et al., 2012).
Critically, the image categories used in the unmasked task comprise inverted faces in addition to upright faces and non-face objects such as houses and tools. The comparison between neural activity in response to upright versus inverted faces is expected to reveal features of neural processing that are specific to configural or holistic perception, that is, a gestalt perception where the whole face is perceived in a qualitatively different manner from the sum of its parts (e.g., (Rossion and Gauthier, 2002)). This phenomenon can be measured behaviorally, for example via reaction times in recognition (e.g. same/different judgment) tasks.
The comparison between specific neural markers in the masked and unmasked tasks enables one to discard neural markers that could otherwise be considered as putative NCC-core if only the contrast between visible and invisible trials at threshold were considered. More generally, this work paves the way for a new promising set of methodological approaches in consciousness research based on the comparison between similar experimental protocols, which differ in specific aspects that expose the key differences that enable one to disentangle the different aspects of the conscious visual experience.
Section snippets
Data acquisition
We recorded intracranially with electrocorticographic (ECoG) electrodes from 5 epilepsy patients undergoing pre-surgical monitoring. Sampling rate for the ECoG signal was 2034.5 Hz. Electrode location was based solely on clinical criteria. Patient age, gender, handedness, ocular and language dominance, and locations of seizure foci are reported in Table 1. We did not record for 12 h after any generalized seizure event. We did not perform any explicit artifact rejection. Voltage traces from each
Neural responses to objective and subjective attributes of visual stimuli
Our electrophysiological data set comprised 1071 bipolar channels (187, 171, 219, 228, 266 from subject 147, 153, 154, 168, 178) from the ventral and lateral temporal cortex of 5 subjects. Out of these, 271 channels were face-responsive (82, 57, 53, 53, 26 from subject 147, 153, 154, 168, 178, see subsection “Face responsiveness” in Methods for the definition of face responsiveness).
A subset of face-responsive channels exhibited spectral power responses that differed between categories in the
Discussion
Traditionally, the closely related aspects of conscious face perception (e.g., contrasting a visible with an invisible face) and holistic face perception (e.g., contrasting an upright with an inverted face) have been assessed separately. However, the search for the neural correlates of consciousness consists in assessing the patterns of neural activation that covary with the phenomenal experience, regardless of the precise experimental protocol employed. By combining similar, but different,
Conclusion
We have combined a set of decoding analyses of either physical properties of stimuli or subjective phenomenology in a masked task, with a set of decoding analyses of stimulus category in an unmasked task. Our results in the masked task show that subjective phenomenology is better decodable than physical properties of stimuli (such as contrast) in the high level visual areas considered in this study, i.e. the ventral and lateral sides of the temporal lobe.
While we were able to decode subjective
Competing interests
NT collaborates with Dr Ryota Kanai, CEO of a venture company, called Araya Brain Imaging (araya.org) based in Tokyo, Japan. In October 2015, Araya received a grant from Japan Science and Technology (JST; approximately 3 million yen for 5 years), which aims to potentially commercial projects related to artificial consciousness. NT is part of this grant and he is supposed to host several students/research assistants/postdocs hired at Araya until March 2020. However, his institute (Monash
Author contributions
Conceived the idea of the study: FB NT. Conceived and designed the experiments: HK CKK MAH RA NT. Performed neurosurgery: HK MAH. Collected the data: HK CKK NT. Conducted preliminary analyses including anatomical localization of electrodes: HK HO NT. Analyzed the data: FB JvK NT. Wrote the manuscript: FB NT. Commented critically on the manuscript: FB JvK HK CKK HO MAH RA NT.
Acknowledgments
We thank all subjects for their participation in the study, the medical team at University of Iowa Hospital for their assistance, Haiming Chen for his technical assistance. We thank Andrew Haun for generating Fig. 1. NT is supported by Australian Research Council (ARC) Future Fellowship (FT120100619) and Discovery Project (DP130100194). NT and FB were supported by Japan Science and Technology Agency (JST) PRESTO fellowship 3630 to NT. CKK, HK, HO, and MAH were supported by NIH grants R01
References (160)
- et al.
Distinct representations for facial identity and changeable aspects of faces in the human temporal lobe
NeuroImage
(2004) - et al.
Distilling the neural correlates of consciousness
Neurosci. Biobehav. Rev.
(2012) - et al.
Visual areas exert feedforward and feedback influences through distinct frequency channels
Neuron
(2015) - et al.
Tools for probing local circuits: high-density silicon probes combined with optogenetics
Neuron
(2015) - et al.
The fusiform face area is tuned for curvilinear patterns with more high-contrasted elements in the upper part
NeuroImage
(2006) - et al.
Inter-subject alignment of human cortical anatomy using functional connectivity
NeuroImage
(2013) - et al.
The ’correlates' in neural correlates of consciousness
Neurosci. Biobehav. Rev.
(2012) - et al.
Prosopagnosia can be associated with damage confined to the right hemisphere–an MRI and PET study and a review of the literature
Neuropsychologia
(1994) - et al.
Experimental and theoretical approaches to conscious processing
Neuron
(2011) - et al.
An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest
NeuroImage
(2006)