Article Figures & Data
Figures
- Figure 1.
MEG-fMRI RSA analysis. A, Examples from our 92-image set (B) MEG analysis and MEG RDMs. From the MEG signals, the complex TF transform was computed for each of the 306 MEG sensors. The amplitude and phase (separated into cosine and sine) values were extracted from the complex number at each TF coordinate, and a MEG RDM was constructed, reflecting the distance between oscillatory activation patterns for every pair of images (i,j; for details, see Materials and Methods). As a result, we obtained a power and phase MEG RDM at each TF coordinate for each participant. C, fMRI RDMs were obtained from (Cichy et al., 2014). Two ROIs were defined: V1 and IT and one fMRI RDM was obtained for each ROI, and each participant, reflecting the distance between BOLD activation patterns for every pair of images (i,j). D, RSA consists in comparing two (or more) RDMs. The MEG power or phase RDMs were compared with the fMRI RDMs (V1 or IT) by computing the partial Pearson’s R. This step was performed at each TF coordinate, resulting in an RSA map of R values at each TF coordinate, for each subject and ROI.
- Figure 2.
K-means clustering analysis procedure. Starting from the MEG RDM representations (top left, as described in Fig. 1A), we flatten each RDM data point into a vector. The entire set of vectors (across all TF coordinates and power/phase conditions) is entered into a K-means clustering algorithm (right), resulting in N clusters and their centroids. By measuring the distance of these centroids to all initial RDM data points, we obtain TF maps of “distance to centroid” (bottom left) that capture the main TF components (across both power and phase) of each cluster.
- Figure 3.
A, B, t-distributed stochastic neighbor embedding (t-SNE) visualizations of 500 ImageNet samples and the 92-image stimulus-set used in this study, across representative layers of two networks (A, AlexNet; B, EfficientNet). To obtain these visualizations, the feature values of all images were subjected to a principal component analysis (PCA) of which only the first 100 dimensions were retained (so as to limit computational demands); then, t-SNE was applied, as implemented in the scikit-learn Python library, with parameters: [perplexity = 30, n_iter = 1000, learning_rate = 1.0, min_grad_norm = 0]. The DNNs used in this study had been trained on images from ImageNet, which consists of millions of photographs of one or more objects in natural backgrounds. In contrast, our 92-image stimulus set consists of cut-out images on a gray background. The DNNs learn optimal representations for the training images from ImageNet, i.e., different images from different categories are mapped to different regions of the representation space, and the whole space tends to be equally occupied by the training samples. However, as the t-SNE visualizations show, our 92 images are all projected into a remote corner of this space, meaning that the RDM distances between the 92 images are confounded by the mean vector (the pairwise Pearson distance depends more on the alignment with the mean vector, and less on the true physical distance between points). Inset images show the most stereotypical image of our 92-stimulus set (highlighted in green), the closest image from the ImageNet set (characterized, as expected, by an empty gray background), as well as one ImageNet sample near the space origin, and one on the opposite side of the feature space. To circumvent this problem, we used a re-centering approach as described in Materials and Methods. C, The same layer of AlexNet as shown in A, after re-centering. The 92-stimulus set is now closer to the center of the feature space. D, Systematic measurement of the distance between the centroid of our 92-stimulus set and the space origin (normalized by the SD across our 92 images), for each layer of each DNN. The two DNN layers depicted in A, B are labeled (a) and (b) on the corresponding curves. As a baseline, the dashed lines reflect the same distance measure, applied to the 500 ImageNet samples.
- Figure 4.
Results of the 2 × 2 RSA comparisons (MEG power/phase × fMRI V1/IT), averaged over all subjects. (A) MEG-Power x fMRI V1, (B) MEG-Power x fMRI-IT, (C) MEG-Phase x fMRI-V1, (D) MEG-Phase x fMRI-IT. The purple contours mark those regions in the maps that are significantly different from zero (paired t test against 0 across N = 15 subjects, FDR correction, α = 0.05). Note that the absolute latencies are not directly comparable across frequencies, because of different smoothing windows applied at the different frequencies when performing the TF transform (hence, the x-axis is labeled as uncorrected time).
- Figure 5.
Profile of the results of the RSA with V1 (green lines) and IT (black lines) in oscillatory power (top row) and oscillatory phase (bottom row) in different frequency bands. To examine the RSA maps in more detail, we extracted their time courses in different traditional frequency bands: α (8–13 Hz), low-β (13–20 Hz), and high-β (20–32 Hz). In each of these frequency bands, we computed the average R values. Since the TF decomposition induces temporal smearing, and the amount of smearing differs for different frequencies, to interpret the latencies of the representational similarities, we corrected for this smearing effect. Specifically, to avoid underestimating the onset latencies, we corrected time by adding half the wavelet window duration at each frequency. Note that the same correction was applied to the two curves compared in each plot. Solid lines are the means across subjects, and the shaded areas correspond to the SEM across subjects.
- Figure 6.
Clustering analysis. K-means clustering was performed on the MEG power and phase RDMs. Each TF plot shows the distance of each RDM from the centroid of the corresponding cluster. The purple lines correspond to the cluster boundaries as returned by the k-means algorithm, indicating that all points within the purple lines are assigned to this specific cluster based on their distance to the different cluster centroids. The distance to centroid (color scale) reflects how “stereotypical” each RDM is for the corresponding cluster (i.e., how close to the cluster centroid), a continuous scale that complements the discrete cluster assignment. For example, although cluster 3 simultaneously encompasses oscillatory power across many frequencies from 10 to 65 Hz, we can see that low-β frequencies (13–20 Hz) are the most stereotypical for this cluster. The insets show the relative degree of RSA between the cluster centroid and V1/IT (top), or the cluster centroid and the DNN layer hierarchy (bottom). For the DNNs, the layer with maximum RSA, normalized by the number of layers in the DNN hierarchy, and averaged across the seven DNN types (colored ticks), was taken as the layer that corresponded to each cluster centroid (black arrowhead).
- Figure 7.
Clustering results for clusters 5–7. We identify these clusters as noise components because (1) their distance to the cluster centroid is typically higher than for other clusters, and (2) they mainly map onto prestimulus oscillatory activity. Prestimulus oscillations, while accounting for a sizeable portion of the (notoriously noisy) MEG signal variance, cannot possibly encode the identity of a stimulus that has not been presented yet. Prestimulus α is a well-studied oscillatory component reflecting the attention state of the observer, and whose phase is known to modulate the subsequent ERP amplitudes and latencies; as such, it is not surprising that the phase of this oscillatory component would induce a separate cluster of RDM patterns (cluster 5). Similarly, prestimulus α-β power (cluster 6) and γ phase (cluster 7) could reflect preparatory attention or motor signals (including muscular artifacts) not related to stimulus identity. Notations as in Figure 6. Note that for consistency with the previous figure, we continue to report the V1-IT RSA scaling value in the insets; however, the corresponding correlation values were systematically lower for these clusters, and should thus be interpreted with caution (V1 partial correlations for clusters 5–7: [0.04, 0.00, 0.08], IT partial correlations: [0.00, 0.04, 0.03]). In comparison, V1 partial correlations for clusters 1–4 ranged from 0.14 to 0.43, and IT partial correlations from 0.12 to 0.44. Similarly, we also report the DNN layer for which the correlation to the cluster centroid was maximal; however, this maximal correlation was consistently lower than in the previous figure, as expected for a prestimulus component (peak correlations averaged across DNNs for clusters 5–7: [0.02, 0.03, 0.16], compared with values ranging from 0.24 to 0.45 for clusters 1–4).
- Figure 8.
Illustrative summary. The different clusters identified are plotted schematically as a function of time, and their main oscillatory characteristics (frequency band, power/phase) are indicated, together with the corresponding brain region (V1/IT) and the corresponding DNN layer (low/mid/high-complexity).
In this issue
