Article Figures & Data
Figures
- Figure 1.
Task modulation of neural encoding of social traits. A, Task. Each face was presented for 1.5 s, followed by participants’ judgment of trustworthiness/dominance within 2 s. The overall intertrial interval was jittered between 1 and 2 s. B, Correlation between the ratings from our neurosurgical patients with consensus ratings from Oosterhof and Todorov (2008). Tru: trustworthiness. Dom: dominance. Error bars denote 1 SEM across sessions. Asterisks indicate a significant difference from 0 using two-tailed paired t test; ***p < 0.001. C, D, Example neurons that showed a significant correlation between the normalized FR and the z-scored rating for (C) the trustworthiness trait and (D) the dominance trait. Each dot represents a face image, and the gray line denotes the linear fit. Sample face images with a range of consensus trustworthiness/dominance ratings are illustrated below the correlation plots, and the corresponding consensus ratings (z-scored) are shown under each sample face image. E, F, Scatterplot of correlation coefficient between congruent versus incongruent conditions. Each circle represents a neuron, and the gray line denotes the linear fit. E, Trustworthiness trait. F, Dominance trait. G–J, Correlation between neural response and social judgment ratings. Bars show the mean correlation coefficient under each task instruction. Error bars denote ±SEM across neurons. Asterisks indicate a significant difference from 0 (two-tailed paired t test) or between conditions (two-tailed two-sample t test); *p < 0.05, **p < 0.01, ****p < 0.0001. For each trait (trustworthiness or dominance), the left bars show the congruent conditions (i.e., the evaluated traits were the same as the task instruction) and the right bars show the incongruent conditions (i.e., the evaluated traits were different to the task instruction). G, All neurons (n = 630). H, The same neurons that were recorded in both trustworthiness and dominance judgment tasks (n = 143). I, Amygdala neurons only (n = 328). J, Hippocampal neurons only (n = 302). Extended Data Figure 1-1 shows assessment of spike sorting and recording quality. Extended Data Figure 1-2 shows control analyses for choices.
- Figure 2.
RSA for social trait and neural response. A, The social trait DM. B, E, The neural response DM from (B) the trustworthiness judgment task and (E) the dominance judgment task. Color coding shows dissimilarity values. C, F, Observed versus permuted correlation coefficient between DMs. C, Correlation between the social DM and the neural DM from the trustworthiness judgment task. F, Correlation between the social DM and the neural DM from the dominance judgment task. The correspondence between DMs was assessed using permutation tests with 1000 runs. The magenta line indicates the observed correlation coefficient between DMs. The null distribution of correlation coefficients (shown in gray histogram) was calculated by permutation tests of shuffling the faces (1000 runs). D, Bootstrap distribution of DM correspondence for each task instruction. Tru: trustworthiness judgment task (blue). Dom: dominance judgment task (red). Violin plots present the median value as the white circle and the interquartile range as the gray vertical bars. The neural response from the trustworthiness judgment task had a stronger correspondence with the social trait space compared with the neural response from the dominance judgment task. G–I, RSA for neural response and low-level features. G, The low-level feature DM. H, I, Observed versus permuted correlation coefficient between DMs. H, Correlation between the low-level feature DM and the neural DM from the trustworthiness judgment task. I, Correlation between the low-level feature DM and the neural DM from the dominance judgment task. The correspondence between DMs was assessed using permutation tests with 1000 runs. The magenta line indicates the observed correlation coefficient between DMs. The null distribution of correlation coefficients (shown in gray histogram) was calculated by permutation tests of shuffling the faces (1000 runs). J, Correlation between the DNN feature DM and the neural DM. Null distribution was estimated using 1000 permutation runs. Red, trustworthiness judgment task; blue, dominance judgment task. Shaded area denotes the statistical significance threshold (95% interval of the null distribution).
- Figure 3.
Task modulation of neural encoding of low-level facial features. A–F, Axis-based feature coding. G–M, Region-based feature coding. A, B, A simple linear regression model. C, D, A classic face model. E, F, A PLS regression model. A, C, E, The number of neurons showing a significant face model. Black dots show the chance number of significant neurons (5% of all neurons). B, D, F, Model assessment. For the classic face model and PLS regression model, model predictability was assessed using the Pearson correlation between the predicted and actual neural response in the test dataset. Error bars denote ±SEM across neurons. G, Feature space constructed by the first shape and tone/texture features that were used to generate the stimuli. Note that face shape varied along feature dimension 1 and skin color varied along feature dimension 2. H, I, Example neurons demonstrating region-based feature coding. The size of the squares indicates the FR. The blue/red outlines delineate the tuning regions of the neurons in the feature space. H, An example neuron from the trustworthiness judgment task. I, An example neuron from the dominance judgment task. J, Percentage of neurons (all neurons, amygdala neurons, and hippocampal neurons) demonstrating region-based feature coding. K, L, The aggregated tuning regions of the neuronal population. Color bars show the counts of overlap between individual tuning regions. K, Trustworthiness judgment task. L, Dominance judgment task. M, The percentage of the feature space covered by the tuning regions did not differ between the trustworthiness judgment task and the dominance judgment task for all neurons and amygdala neurons. Error bars denote ±SEM across neurons. Asterisk indicates a significant difference between conditions (two-tailed two-sample t test); *p < 0.05. N, Group difference in the aggregated tuning regions of the neuronal population (trustworthiness − dominance). Yellow, higher counts in the trustworthiness judgment task; blue, higher counts in the dominance judgment task; green, no difference between tasks. O, Statistical map shows areas that had a significant difference in the proportion of neurons (i.e., the number of neurons that encoded a particular pixel divided by the total number of region-coding neurons) between tasks (yellow; χ2 test at each pixel, p < 0.05 uncorrected for multiple comparisons). Extended Data Figure 3-1 shows axis-based coding of low-level features analyzed separately for shape features and texture features. Extended Data Figure 3-2 shows region-based coding of low-level features analyzed separately for shape features and texture features.
- Figure 4.
Task modulation of eye movement onto faces. A, Percentage of fixations for each ROI. Error bars denote ±SEM across sessions. B, Fixation density maps to quantify eye movements for different tasks. Each map shows the probability of fixating a given location within the entire stimulus period. The scale bar (color bar) is in arbitrary units. The ROIs (eye, mouth, nose) used for analysis are shown in red (not shown to patients). C, Mean distance to the stimulus center (in pixels). Tru: trustworthiness. Dom: dominance. D, Group difference density map (trustworthiness − dominance) shows areas that patients fixated more in the trustworthiness judgment task (yellow), and vice versa (blue), with green meaning there was no difference between tasks. E, Statistical map shows areas that had a significant difference in density maps between tasks (red; two-tailed two-sample t test between individual density maps at each pixel, p < 0.05 uncorrected).
- Figure 5.
Task modulation of face part selectivity. A–E, Trustworthiness judgment task. F–J, Dominance judgment task. A, F, Neurons that had a significantly greater FR when fixating on the eyes compared with the mouth (selection by two-tailed t test in a time window of −200 ms before fixation onset to 200 ms after fixation offset). B, G, Neurons that had a significantly greater FR when fixating on the mouth compared with the eyes. Fixations are sorted by fixation duration (black line shows start of the next saccade). Fixation onset is t = 0. Asterisks indicate a significant difference between fixations on the eyes and mouth in that bin (p < 0.05, two-tailed t test, after Bonferroni correction; bin size = 100 ms). C–E, H–J, Population summary of all eye-mouth-selective. C, H, Average normalized FR of eyes-preferring neurons. D, I, Average normalized FR of mouth-preferring neurons. Shaded area denotes ±SEM across neurons. Asterisks indicate a significant difference between the fixation categories in that bin (p < 0.05, two-tailed t test, after Bonferroni correction). E, J, Single-fixation analysis using the FSI (Materials and Methods). Shown is the cumulative distribution of the single-fixation response of fixation-aligned eyes-preferring and mouth-preferring neurons for fixations on the eyes and mouth.
- Figure 6.
Summary of each type of neuron in the amygdala and hippocampus. A, Trustworthiness judgment task. B, Dominance judgment task.
Tables
ID Age Sex Race Epilepsy diagnosis Trustworthiness task Dominance task Number of
amygdala
neuronsNumber of
hippocampal
neuronsNumber of
amygdala
neuronsNumber of
hippocampal
neuronsTotal Left Right Total Left Right Total Left Right Total Left Right P6 33 F White Left posterior neocortical
extratemporal/parietal6 6 0 20 20 0 9 9 0 27 27 0 P7 28 F White Right mesial temporal 3 3 0 23 21 2 8 8 0 36 36 0 P10 47 F White Right mesial temporal and
neocortical temporal22 0 22 0 0 0 3 0 3 0 0 0 P11 33 F White Right mesial temporal and
extratemporal17 0 17 0 0 0 15 0 15 0 0 0 7 0 7 0 0 0 12 0 12 20 0 20 5 0 5 12 0 12 3 0 3 11 0 11 P14 26 M White Bilateral amygdylar/
hippocampal22 18 4 7 5 2 23 18 5 7 5 2 23 20 3 8 6 2 27 22 5 8 6 2 20 17 3 32 31 1 21 18 3 41 40 1 11 9 2 6 4 2 12 10 2 5 3 2 P15 37 M White Left amygdylar/
hippocampal11 11 0 3 0 3 22 22 0 13 0 13 12 12 0 20 0 20 14 14 0 3 0 3 Sum 159 96 63 131 87 44 169 121 48 171 117 54 Each row of neurons represents a separate recording session. Total: all neurons recorded from an area that had a FR greater than 0.15 Hz. Left, Neurons that were recorded from the left side of an area. Right, Neurons that were recorded from the right side of an area.
Extended Data Figure 1-1
Spike sorting and recording quality assessment. A, Histogram of the number of units identified on each active wire (only wires with at least one unit identified are counted). The average yield per wire with at least one unit was 2.54 ± 1.62 (mean ± SD). B, Histogram of mean FRs. C, Histogram of proportion of interspike intervals (ISIs), which are shorter than 3 ms. The large majority of clusters had less than 0.5% of such short ISIs. D, Histogram of the signal-to-noise ratio (SNR) of the mean waveform peak of each unit. E, Histogram of the SNR of the entire waveform of all units. F, Pairwise distance between all possible pairs of units on all wires where more than one cluster was isolated. Distances are expressed in units of SD after normalizing the data such that the distribution of waveforms around their mean is equal to 1. G, Isolation distance of all units for which this metric was defined (n = 630, median = 14.56). H, Neurons recorded in the trustworthiness judgment task versus dominance judgment task did not differ significantly in isolation distance (t(534) = 0.76, p = 0.45). Download Figure 1-1, EPS file.
Extended Data Figure 1-2
Control analyses for choices. A, Distribution of choices across tasks. Blue, trustworthiness judgment task; red, dominance judgment task. Error bars denote 1 SEM across sessions. There was no significant difference between tasks for each choice (all ps > 0.1). B, An example neuron showing a significant correlation with choices in the trustworthiness judgment task. C, An example neuron showing a significant correlation with choices in the dominance judgment task. Error bars denote ±SEM across images. D, Percentage of neurons showing a significant correlation with choices (trustworthiness judgment task: n = 21, 7.24%, binomial p = 0.04; dominance judgment task: n = 34, 10%, binomial p < 0.0001). The percentages did not differ significantly between tasks (χ2 test: p = 0.22). E, Overlap of choice neurons between tasks with neurons recorded from both tasks. The overlap between choice neurons across tasks was not above chance (χ2 test; trustworthiness judgment task: p = 0.14; dominance judgment task: p = 0.60). F, Correlation between neural response and patients’ social judgment ratings. G, Correlation between neural response and social judgment ratings after excluding choice neurons. Bars show the mean correlation coefficient under each task instruction. Error bars denote ±SEM across neurons. Asterisks indicate a significant difference from 0 (two-tailed paired t test) or between conditions (two-tailed two-sample t test); *p < 0.05 and **p < 0.01. For each trait (trustworthiness or dominance), the left bars show the congruent conditions (i.e., the evaluated traits were the same as the task instruction) and the right bars show the incongruent conditions (i.e., the evaluated traits were different to the task instruction). Download Figure 1-2, EPS file.
Extended Data Figure 3-1
Axis-based coding of low-level features analyzed separately for shape features (A–C, G–I) and texture features (D–F, J–L). Legend conventions as in Figure 3A–F. Download Figure 3-1, EPS file.
Extended Data Figure 3-2
Region-based coding of low-level features analyzed separately for shape features (A–D) and texture features (E–H). Legend conventions as in Figure 3G–O. Download Figure 3-2, EPS file.
In this issue
