Abstract
The perception of facial affect engages a distributed cortical network. We used functional magnetic resonance imaging and dynamic causal modeling to characterize effective connectivity during explicit (conscious) categorization of affective stimuli in the human brain. Specifically, we examined the modulation of connectivity from posterior regions of the face-processing network to the lateral ventral prefrontal cortex (VPFC) during affective categorization and we tested for a potential role of the amygdala (AMG) in mediating this modulation. We found that explicit processing of facial affect led to prominent modulation (increase) in the effective connectivity from the inferior occipital gyrus (IOG) to the VPFC, while there was less evidence for modulation of the afferent connections from fusiform gyrus and AMG to VPFC. More specifically, the forward connection from IOG to the VPFC exhibited a selective increase under anger (as opposed to fear or sadness). Furthermore, Bayesian model comparison suggested that the modulation of afferent connections to the VPFC was mediated directly by facial affect, as opposed to an indirect modulation mediated by the AMG. Our results thus suggest that affective information is conveyed to the VPFC along multiple parallel pathways and that AMG activity is not sufficient to account for the gating of information transfer to the VPFC during explicit emotional processing.
Introduction
Emotions are core aspects of mental life. Several meta-analyses (Phan et al., 2002; Murphy et al., 2003; Fusar-Poli et al., 2009; Vytal and Hamann, 2010) have attempted to formulate models of the neural circuitry underlying emotional processing. The majority of studies included in these meta-analyses used facial affect as a probe for emotional processing. Facial affect processing involves a number of functionally and anatomically connected cortical and subcortical brain structures, including the inferior occipital gyrus (IOG) (Haxby et al., 2000), the fusiform gyrus (FG) (Hoffman and Haxby, 2000; Haxby et al., 2002), the amygdala (AMG), and the ventral prefrontal cortex (VPFC) (Adolphs, 2002; Fairhall and Ishai, 2007). Within this network, the AMG is thought to play a key role in the rapid detection of facial affect and in biasing behavioral responses accordingly (LeDoux, 1998; Rolls, 1999). In contrast, the VPFC is thought to be involved in a more detailed evaluation of emotional stimuli and their contextual significance (Iidaka et al., 2001; Ochsner and Gross, 2005; Quirk and Beer, 2006). Crucially, the processing of facial features in this network is context sensitive. The AMG may be more engaged when facial expressions are processed outside the focus of attention (implicit processing) or when attention is directed toward nonaffective cues (e.g., gender or facial features) (Critchley et al., 2000; Hariri et al., 2003). In contrast, when subjects are engaged in facial affect labeling (explicit processing), AMG activation may be reduced (Critchley et al., 2000; Hariri et al., 2000; Dyck et al., 2011). In addition, responses to emotional faces are generally enhanced within visual cortices (Vuilleumier and Driver, 2007). This phenomenon has been attributed to modulation of early visual processing by prefrontal regions via mechanisms of selective attention (Armony and Dolan, 2002), although some models highlight the role of the AMG as the source of signal amplification both for sensory processing and prefrontal engagement (Anderson and Phelps, 2001; Pessoa et al., 2002). Given such conflicting accounts, we wanted to characterize the functional interrelationships among the structures involved in facial affect processing and infer the direction of effects mediated by critical pathways in the network.
In this study, we combined conventional statistical parametric mapping (SPM) with dynamic causal modeling (DCM) of functional magnetic resonance imaging (fMRI) data (Friston et al., 2003) to investigate effective connectivity within the face perception network and its modulation by affect in 40 healthy adults performing a face affect categorization task. The aims of the study were twofold. First, we hypothesized that the explicit processing or categorization of affective stimuli would increase the connectivity from posterior regions of the face processing network to the VPFC. Second, we investigated the role of the AMG during conscious processing of facial affect by testing the hypothesis implied by previous work (Anderson and Phelps, 2001; Pessoa et al., 2002) that AMG gates prefrontal connectivity during affect processing.
Materials and Methods
Subjects
Forty healthy adults (Table 1) were recruited via advertisement in the local press and were included if they (1) were aged 18–65 years, (2) had no personal lifetime history of mental health problems, substance use, head injury, or medical disorders [as assessed following personal interview with trained psychiatrists using the Structured Interview for DSM-IV-TR Axis I Disorders, Non-patient Edition (First et al., 2002)], (3) did not take any, prescription medication, and (4) were of self-reported British white ancestry. An estimate of current intellectual function (IQ) was obtained using the Wechsler Adult Intelligence Scale–Revised (Wechsler, 1981). Educational level was scored on a five-point scale, where 1 indicated no formal qualifications and 5 indicated postgraduate-level qualifications.
The study was approved by the Ethics Committee of the Institute of Psychiatry and the South London and Maudsley National Health Service Trust. Written informed consent was obtained from all subjects.
Experimental design
We studied three negative emotions (anger, fear, and sadness) in three separate event-related facial affect recognition fMRI experiments conducted in a single acquisition session in a randomized order. Each experiment lasted for 5 min. In each experiment, 10 different facial identities (six female, four male; www.paulekman.com) were presented; depending on the experiment, they either depicted a negative emotion (anger, fear, or sadness) or a neutral facial expression. The facial identities were manipulated by computer software to depict 150% intensity of the emotion. The 150% level of intensity was chosen to minimize ambiguity and uncertainty about the nature of the stimuli (Calder et al., 1997; Phillips et al., 1997). Faces were presented in alternation with a fixation cross in a pseudorandom order. The neutral faces and affective faces were each displayed for 2 s and repeated 20 times, giving a total of 60 images (Fig. 1). The interstimulus interval followed a Poisson distribution and was varied between 3 and 9 s (mean interval, 5 s). Participants were instructed to press the right or left button with their dominant hand on an MRI-compatible response box to indicate whether the face was emotional or neutral. Subjects were familiarized with the task off-line 1 h before the scan. Response time and accuracy data were collected.
Image acquisition
Gradient echo planar MR images were acquired using a 1.5 T GE Sigma MR system (General Electric) fitted with 40 mT/m high-speed gradients. Foam padding and a forehead strap were used to limit head motion, and a quadrature birdcage head coil was used for radio frequency transmission and reception. In each of the 16 noncontiguous planes parallel to the intercommissural (AC–PC) plane, T2*-weighted MR images reporting BOLD contrast were acquired (TR = 2000 ms, TE = 40 ms, flip angle = 70°, slice thickness = 7 mm, slice skip = 0.7 mm, matrix size = 64 × 64, voxel dimensions = 3.75 × 3.75 × 7.7 mm). For each participant 3 × 150 fMRI images were acquired.
During the same session, a high-resolution T1-weighted structural image was acquired in the axial plane (inversion recovery prepared, spoiled gradient-echo sequence; TR = 18 ms, TE = 5.1 ms, TI = 450 ms, flip angle = 20°, slice thickness = 1.5 mm, matrix size = 256 × 192, FOV = 240 × 180 mm, voxel dimensions = 0.9375 × 0.9375 × 1.5 mm, number of excitations = 1) for subsequent coregistration.
Image processing
Preprocessing.
For image preprocessing and GLM analysis, we used the SPM8 software package (Wellcome Trust Centre for Neuroimaging, London, UK; http://www.fil.ion.ucl.ac.uk); for effective connectivity analyses, DCM8 was used. Preprocessed images were realigned to correct for movement and normalized into MNI space using each subject's structural MRI image. The spatially normalized data were smoothed with an isotropic Gaussian filter (8 mm full-width half-maximum) to compensate for normal variation in structural and functional anatomy across subjects.
First level (within-subject) analysis.
For each subject, the data from the three experiments (emotions) were concatenated and modeled with a general linear (convolution) model, with additional regressors representing potential confounds. Vectors of onset representing the correctly identified angry, fearful, and sad faces and correctly identified neutral faces were convolved with a canonical hemodynamic response function. Serial correlations were removed using a first-order autoregressive model and a high-pass filter (128 s) was applied to remove low-frequency noise. An explicit mask was used to ensure only voxels within the brain were included in the analysis. Six movement parameters were also entered as nuisance covariates. Additionally, the means of the three sessions were modeled as well as the transition at the end of each session. Contrast images of brain activations associated with correct categorization of emotional faces (angry, fearful, and sad) compared with neutral faces were produced for each participant.
Second level (between-subject) analysis.
Group-level analyses were based on random-effects analyses of the single-subject contrast images using the summary statistic approach. For facial affect, one-sample t tests were used to investigate the main effect of task (correctly identified emotional faces > correctly identified neutral faces). The statistical threshold was adjusted to provide a FWE of p = 0.05 (based on the spatial extent of clusters of voxels thresholded at p < 0.001), corrected for multiple comparisons across the whole brain. For all analyses, results are reported in the MNI coordinate system.
Volumes of interest
We selected a priori volumes of interest (VOIs) within a right-hemispheric network of regions implicated in face processing and its modulation by affect, following previous work (Fairhall and Ishai, 2007). These VOIs comprised the IOG (x = 44, y = −78, z = −6), the FG (x = 24, y = −56, z = −12), the AMG (x = 20, y = −2, z = −16), and the inferior frontal gyrus within the VPFC (x = 52, y = 20, z= −6). The coordinates for the visual regions, IOG and FG, were based on the group maxima from the contrast of all faces (minus crosshair), while the coordinates for the emotional regions, AMG and IFG, were specified from the contrast of emotional minus neutral faces. For each subject, we chose subject-specific maxima (in the appropriate SPM) in these regions that were within 4 mm of the group maxima and within the same anatomical area, as defined by the PickAtlas toolbox (Maldjian et al., 2003). Regional time series were summarized with the first eigenvariate of all activated (at p < 0.01) voxels within 5 mm of the subject-specific maxima.
Dynamic causal modeling
DCM (Friston et al., 2003) is a Bayesian model comparison procedure used to infer effective connectivity between brain regions. DCM estimates directed interactions within neural systems. Crucially, it models these interactions at the neuronal level and distinguishes between endogenous coupling and context-specific coupling, while accounting for the effects of experimentally controlled network perturbations (cf. stimulus-locked coupling) (Friston et al., 2003; Penny et al., 2004).
A four-area DCM was specified for all subjects with bidirectional endogenous connection between all regions (IOG, FG, AMG, VPFC) and the main effect of “all faces” as the driving input entering the IOG, the visual input region of our model. This base model (Fig. 2a) was then elaborated systematically to produce seven alternative variants. These variations were guided by our primary aim to define the role of the VPFC in facial affect recognition and the modulation of afferent connections to VPFC by affect (anger, fear, and sadness). Figure 2b shows the seven variants of four-area models, which include all possible combinations of how facial affect could modulate the forward connections to the VPFC. Note that while these figures show, for clarity, a single modulatory term labeled “Facial Affect,” the models contained distinct modulatory inputs for anger, fear, and sadness, allowing us to test the modulatory effects of these emotions (on connectivity) separately.
An additional (bilinear) model was constructed, where affective stimuli directly entered AMG, to test whether affective information transfer to the VPFC is mediated by the AMG (Fig. 2b, Model 8). Furthermore, three additional nonlinear models were constructed that allowed for multiplicative interactions of postsynaptic inputs (Stephan et al., 2008). In these models, affective stimuli drove AMG activity directly, which then modulated the forward connections (from IOG and/or FG) to the VPFC (Fig. 2c, Models 9, 10, 11). These nonlinear models replace the direct modulatory effect of experimentally defined facial affect with a vicarious physiological influence that is mediated by synaptic connections from the AMG. All 11 models (Fig. 2) were constructed, fitted, and compared for each of the 40 subjects in this study.
Note that we limited our model comparisons to the right hemisphere, investigating the modulation of forward (afferent) connections to VPFC. This was motivated by the study by Fairhall and Ishai (2007), who highlighted the predominance of the right hemisphere in the processing of emotional faces. Additionally, we assumed that visual input entered the IOG (as the lowest visual area). These constraints made it possible to define a relatively small model space (11 models per subject), in which we systematically investigated which of the forward connections was subject to modulation by facial affect and whether this modulation was best modeled by the direct effect of facial affect (bilinear models) or could be explained indirectly via by AMG modulation (nonlinear models) (Fig. 2).
Model comparison.
Model comparison was implemented using random-effects (RFX) Bayesian model selection (BMS) in DCM8 to compute exceedance and posterior probabilities at the group level (Stephan et al., 2009). The exceedance probability of a given model denotes the probability that this model is more likely than any other model tested, given the data. Additionally, we made inferences about families of models (Penny et al., 2010; Stephan et al., 2010). Specifically, our models were divided into a bilinear (models 1–8) and a nonlinear (models 9–11) family to test whether or not AMG activity could provide a sufficient account of the modulation of connections to VPFC. All models were included in the BMS procedure, both when comparing individual models and when comparing model families. Finally, to summarize the strength of effective connectivity and its modulation quantitatively, we used random effects Bayesian model averaging (BMA) to obtain average connectivity estimates (weighted by their posterior model probability) across all models and all subjects (Penny et al., 2010).
Correlation with behavioral measures.
To examine correlations between effective connectivity and behavior, we applied BMA on a subject-by-subject basis across all models and then extracted the resulting posterior means for each subject. These were entered into a subsequent correlation analysis with behavioral measures (accuracy and response time) with Bonferroni correction for multiple comparisons (alpha = 0.002).
Finally, we subjected the subject-specific BMA parameter estimates to one-sample tests to assess their significance in a classical sense (i.e., consistency across subjects). Behavioral and demographic data and parameter estimates were analyzed using SPSS 15 (SPSS) using one-sample t tests, or appropriate nonparametric tests (one-sample Kolmogorov–Smirnov test) when data were not normally distributed based on the Kolmogorov–Smirnov criterion, with α = 0.05. Since we tested 21 parameters of interest (i.e., all endogenous connections and all bilinear modulations), we applied Bonferroni correction for multiple comparisons, resulting in an adjusted threshold of α = 0.002.
Results
Behavioral data
Subjects identified all facial affects with a high rate of accuracy, as seen in Table 1.
SPM analysis
In accordance with previous meta-analytic studies (Phan et al., 2002; Murphy et al., 2003; Fusar-Poli et al., 2009; Vytal and Hamann, 2010), robust activation in response to emotional faces (relative to neutral faces) was evident in the visual association cortices, the temporal gyrus as well as in frontal areas. Details of the regional maxima are provided in Figures 3 and 4 and Table 2.
DCM analysis
Comparing the individuals models
Model 1 outperformed all other models with an exceedance probability of 38% (Fig. 5). This optimal Model 1 contained reciprocal endogenous connections between all four areas (IOG, FG, AMG, VPFC), with affect only modulating the forward connection from the IOG to the VPFC. Model 3 was the second-best model with an exceedance probability of 20%. In Model 3, affect modulated the forward connection from AMG to the VPFC.
Comparing the bilinear and nonlinear families
In a next step, we applied random-effects BMS at the family level to clarify whether or not the AMG played a sufficient role in modulating forward connections to VPFC during processing of facial affect. Comparison between the bilinear (models 1–8) and nonlinear (models 9–11) families showed that the bilinear family was superior to the nonlinear family, with an exceedance probability of 100%. This suggests that the AMG alone does not provide a sufficient account for the gating of transfer of facial affect to VPFC. Note that Bayesian model comparison eschews null results. In other words, we can be nearly 100% confident that the (direct) modulatory effects of facial affect provide a better model of empirical responses than a model in which the equivalent modulation is mediated (indirectly) by the AMG.
Bayesian model averaging
The results from the BMA across all subjects and across all 11 models are shown in Table 3. The implementation of RFX BMA in SPM8 employs an Occam's window for computational efficiency (Penny et al., 2010), excluding from the average those models whose probability ratio (compared with the best model) is <0.05. In our case, this applied to model 8 and the three nonlinear models (models 9–11).
The BMA results highlight the importance of the efferent connections from the IOG to FG, AMG, and VPFC in the network. When BMA parameter estimates of the endogenous connections originating from IOG were tested for consistency across subjects with classical t tests, they were found to be highly significant, even when correcting for multiple comparisons. Furthermore, the IOG→VPFC connection was strongly and significantly modulated by anger (p < 0.001; Table 3). It also showed a substantial modulation by fear (p < 0.017), although this did not quite survive Bonferroni correction (α = 0.002). By contrast, sadness did not change this connection significantly (p = 0.076). Altogether, these results emphasized the importance of this direct connection to VPFC for the categorization of facial affect (Fig. 6).
Behavioral correlations
We performed correlations between response time and accuracy for angry, fearful, and sad faces, with the modulatory parameters from BMA on a subject-by-subject basis over models. Mean response time to angry faces showed a significant negative correlation with anger modulation of the forward connections from FG (r = −0.336, p = 0.034) and from AMG to the VPFC (r = −0.337, p = 0.034). Mean response time to fearful faces showed a significant positive correlation for fear modulation on the forward connection from the IOG to the VPFC (r = 0.376, p = 0.017). However, these correlations did not survive Bonferroni correction for multiple tests.
Discussion
In this study, we assessed effective connectivity during facial affect processing using DCM as an established method for inferring effective connectivity from fMRI data (Stephan et al., 2010). Our aim was to examine the modulation of connectivity between posterior regions of the face-processing network and the VPFC during affect categorization and to evaluate the role of the AMG in mediating this modulation. There are three key findings from our study. First, we confirmed our hypothesis that facial affect significantly increased effective connectivity between posterior regions of the face network and the VPFC. Our data were best explained by a model in which facial affect modulated the connection from the IOG to the VPFC. Second, our results suggest that affect may modulate connections to the VPFC above and beyond any putative modulation by the AMG. This implies that the AMG is not necessarily involved in gating prefrontal connectivity during explicit affective processing. Third, the effective connectivity between the IOG and VPFC showed evidence for differential modulation according to valence, with the strongest modulation observed for anger.
These findings are timely in view of the current revaluation of existing models of affective stimuli processing in the brain (Pessoa and Adolphs, 2010). Our results contribute to this debate by highlighting the role of the prefrontal cortex during affective processing and by questioning the prevailing amygdalocentric model of affective processing (LeDoux, 1996; Davis and Whalen, 2001).
Across all subjects, our modeling results suggest that the VPFC receives information directly from regions within the face network. Crucially, however, it was the coupling from the IOG to the VPFC that emerged as the most significant effective connection within this network, which was further enhanced during the processing of facial affect. A large body of literature has established that attention enhances neural responses primarily within extrastriate visual cortices (Moran and Desimone, 1985; Buffalo et al., 2010). A similar enhancement has also been noted in response to emotional valence in general (Lane et al., 1999) and during presentation of emotional, particularly negative (Vuilleumier et al., 2004), facial expressions compared with neutral ones (Critchley et al., 2000; Vuilleumier et al., 2001; Winston et al., 2003). Electrical recordings have shown that during attentive viewing, prefrontal and visual cortices show increased gamma frequency synchrony suggestive of greater functional coupling between these regions (Gregoriou et al., 2009). Our results therefore add support to the notion of increased coupling between prefrontal and visual cortical regions during visual attention and suggest that such coupling may be further increased when attention is directed to the emotional valence of the stimuli.
Research in affective neuroscience has traditionally emphasized the role of the AMG in emotional processing. The AMG is considered a core component of a subcortical pathway for the rapid detection of emotions during visual processing (LeDoux, 1998; Morris et al., 1999; Rolls, 1999) and is thought to influence behavior through the modulation of PFC activity (Miller and Cohen, 2001). There are innumerable examples from multiple lines of research showing that the AMG modulates activity in the ventral visual pathway in response to emotional signals (Sugase et al., 1999), including facial expressions of affect (Pessoa et al., 2002; Vytal and Hamann, 2010). Although the contribution of the AMG to emotional processing is indisputable, our results, together with those of others (Tsuchiya et al., 2009; Pessoa and Adolphs, 2010; Piech et al., 2010), suggest that the amygdalocentric model of affect processing overlooks the significant contribution of other brain regions and the complexity of their interactions. Our modeling results imply that the VPFC receives information about facial affect directly from three distinct brain regions—the IOG, the FG, and the AMG—and that AMG modulation of the connections from either the FG or the IOG to the VPFC is not a complete or sufficient explanation for changes in coupling. While the results of the present study do not disclose the sources of this modulation, they imply that AMG activity does not necessarily gate the transfer of facial affect information toward the VPFC. Instead, they support a model in which affective information from visual stimuli proceeds simultaneously along parallel channels, creating multiple waves of activation across the face-processing network (Pessoa and Adolphs, 2010). One may speculate that the emotional modulation of these channels may have arisen from multiple, and possibly diffuse, sources; these, however, were not visible in the statistical parametric maps of our present fMRI study. Finally, the implausibility of an amygdalocentric explanation of our data, as indicated by our model comparisons, may be due to the fact that the effective connectivity (and its affect-sensitive changes) assessed by our paradigm mediated the explicit or conscious processing of affective stimuli. This contrasts with the implicit and automatic processes normally associated with AMG processing.
The inclusion of facial identities with three distinct negative affects allowed us to explore the effect of valence on effective connectivity within the face-processing network. Our results suggest that anger showed the most potent modulation of the coupling from the IOG to the VPFC. Anger is considered particularly salient compared with other emotions, as it signals the need for immediate action in response to perceived threat. The VPFC is thought to exert a regulatory role in the expression of anger as damage to the VPFC can increase violent and aggressive behaviors (Damasio et al., 1994; Grafman et al., 1996). Our findings complement results from a recent meta-analytic review of the discrete neural correlates of basic emotions, which suggested that the rapid engagement of the VPFC during exposure to anger stimuli may serve to avert potential overreaction, such as unrestrained rage (Vytal and Hamann, 2010).
Finally, concerning the AMG, our results based on Bayesian model averaging suggest that the differential responses to affective versus neutral faces in the AMG arise via two afferent connections (Table 3), i.e., the IOG→AMG connection (p < 0.0001) and the VPFC→AMG connection (p < 0.018, not quite surviving Bonferroni correction at α = 0.002).
Our study focused on effective connectivity within the face-processing network during attentive viewing and categorization of facial affect. Since the ensuing estimates of effective connectivity are context- and paradigm-dependent, further studies are required to explore the functional architecture of the facial-affect-processing network during tasks that make additional cognitive demands involving induction or suppression of emotional responses. Our findings have significant implications for pathophysiological models of affective dysfunction in psychiatric disorders, particularly mood disorders where current amygdalocentric approaches may need to be reevaluated.
In summary, our analyses have identified that during the processing of facial affect, effective connectivity to the VPFC is increased not only from the AMG but also from other regions in the ventral visual stream, namely the FG and IOG. AMG modulation of this coupling does not appear to be sufficient to account for affect-dependent changes during explicit (conscious) processing. Finally, the functional coupling of the IOG to the VPFC plays a major role in the processing of facial affect and suggests a greater contribution of visual cortical–prefrontal pathways to affect processing than previously considered.
Footnotes
This work was supported by a National Alliance for Research on Schizophrenia and Depression Independent Investigator Award 2008 (S.F.) and the Neurochoice project of SystemsX.ch (K.E.S.).
The authors declare no competing financial interests.
- Correspondence should be addressed to Dr. Danai Dima, Section of Neurobiology of Psychosis, Department of Psychosis Studies, Institute of Psychiatry PO66, King's College London, De Crespigny Park, London SE5 8AF, UK. danai.dima{at}kcl.ac.uk