Article Figures & Data
Figures
- Figure 1.
Motivation for using automatic MaqFACS to analyze facial expressions. A, The stereotypical facial expressions in macaque monkeys include the “neutral,” “lip-smacking,” “threat,” “alert,” and “fear grimace” expressions (Altmann, 1962; Hinde and Rowell, 1962). B, Some of the facial expressions that monkeys produce during the experiments that require head immobilization match the stereotypical expressions produced during natural behaviors (e.g., the three images with blue frames on the left correspond to the neutral, lip-smacking, and threat expressions). We have also observed facial expressions that were less frequently described in the literature (two images with yellow frames on the right). C, A comparison between the neutral and lip-smacking facial expression shows that the lip-smacking example contains AU1 + 2 (Brow Raiser) in the upper face, AU25 + 26 + 18i (Lips part, Jaw drop, and True Pucker) in the lower face, and EAU3 (Ear Flattener) in the ear region. D, The proportion of each upper face AU in the FD test set. Bars with the solid outline (first three highest bars) represent the most frequent AUs, which were chosen for the analysis in this work. UpperNone - no coded action in the upper face, AU1+2 - brow raiser, AU43_5 - eye closure, AU6 - cheek raiser, AU41- glabella lowerer. E, Same as D, but for lower face. First five most frequent AUs were chosen for the analysis. F, Proportion matrix of AU combinations in the FD test set, for the most frequent AUs. Cells inside the magenta (bottom left) and green frames (top right) represent the combinations of upper face and lower face AUs, correspondingly. AUs that frequently occurred in combination with other AUs (in the upper face or the lower face, separately) are denoted by “+.” Cell values were calculated as the ratio between the number of frames containing the combination of the two AUs and the total number of frames containing the less frequent AU. G, Left, Images of upper face AUs from the FD test set. UpperNone, No coded action in the upper face; AU1 + 2, Brow Raiser; AU43_5, Eye closure. Right, The difference of the images from the neutral face image. H, Same as G, but for lower face. AU25 + 26, Lips part and Jaw drop; AU25 + 26 + 16, Lips part, Jaw drop, and Lower lip depressor; AU25 + 26 + 18i, Lips part, Jaw drop, and True Pucker.
- Figure 2.
Monkey–intruder behavioral paradigm. Monkey–intruder block, The subject monkey is sitting behind a closed shutter. The intruder monkey is brought into the room and seated behind the shutter, which remains closed. The shutter opens and closes 18 times, and the monkeys are able to see each other while it is open. The subject monkey can not see any part of the intruder unless the shutter is open. At the end of the block, the shutter closes and the intruder monkey is taken out from the room (Extended Data Figs. 2-1, 2-2, 2-3, examples of monkey interactions).
- Figure 3.
Diagram of the automatic MaqFACS AUs recognition system pipeline. A, Alignment of frames from the original video stream (example of two videos from two different RD monkeys). Seven landmark points were manually selected on the mean of all neutral frames of each video. In the next step, these points were mapped to corresponding predefined positions (reference landmarks, common for all videos). The resulting affine transformation for each video was then applied to all its frames. For more examples, see Extended Data Figure 3-1. B, Manual definition of upper face and lower face ROIs on the mean of all neutral frames. Magenta, Upper face ROI; green, lower face ROI. The “All neutral frames mean” image in this scheme was calculated from all RD videos. C, Cropping of all the frames according to upper face and lower face ROIs. D, Generation of δ-images by subtracting the optimal neutral frame of each video from all its frames. The contrast and the color map of the grayscale images were adjusted for a better representation. E, Construction of lower face and upper face δ-images databases, consisting of two-dimensional matrices where each row corresponds to one image. F, Eigenface extraction from the training images and projection of the training and test images onto the eigenspace (following the desired training and test sets construction).
- Figure 4.
Eigenfaces analysis. A, Example of eigenfaces: six first eigenfaces (PCs) of one of the upper face training sets, containing all five Rhesus subjects from RD. The grayscale values were normalized to the 0–1 range, and the image contrast and color map were adjusted for a better representation. The color bar corresponds to pixel grayscale values. B, Same as A, but for lower face. C, Example of the information coded by the first two eigenfaces. Top, The image sequence demonstrates the first eigenface from A, added to the mean image (MeanImg) and varied. Middle, Mean image of the training set (described in A), with the first eigenface added after being weighted by its mean weight (
- Figure 5.
Results for parameters selection and model performance. A, Top, Example of parameter selection for upper face KNN classifier, trained for generalization across species. The training set in the example is the one described in Figure 4A, the test set is monkey D frames from FD, and the distance metric is set to be Euclidean. The surface represents the performance of KNN classifiers with two parameters varied: k (number of nearest neighbors, varied from 1 to 12), and the percentage of the training set variance explained by the eigenfaces (pcExplVar, varied from 50% to 95%). The z-axis is the average sensitivity value of each model (i.e., average of the sensitivity values for the classification of three upper face AUs). The red dot denotes the highest point on the surface and hence the parameters yielding the best performance. With the selected parameters k = 2 and pcExplVar = 93%, the model average sensitivity value is 0.86. Bottom, Same as the top but for the lower face. The training set is one of the lower face training sets, containing all five Rhesus subjects from RD, and the test set is monkey D frames from FD. The distance metric is set to be Euclidean. The selected model has the average sensitivity of 0.84 with the following parameters: k = 9, pcExplVar = 60%. B, The curves demonstrate the number of eigenfaces that should be used to cumulatively capture a given percentage of the dataset variance. The red asterisk denotes the pcExplVar parameter value selected in A. Left, The curve corresponds to the dataset described in A, top. To express 93% of the dataset variance, at least 264 vectors (eigenfaces) should span the eigenspace. Right, Same as left but regarding A, bottom. To express 60% of the dataset variance, at least 15 vectors (eigenfaces) should span the eigenspace. C, Best performance of KNN classification for each generalization type. Each bar group contains five bars (from left to right), as follows: three bars describing the classifier’s sensitivity for single AUs; sensitivity averaged for three classified AUs; and the total accuracy of the classifier. The mean and the error are calculated regarding the recognition performance on a new subject. The horizontal dashed line denotes the chance level. The first bar group demonstrates the results for generalization of the classification within the same Rhesus subject [within subject (Rhesus): training on videos of a subject and testing on a new video of the same subject]. The second group shows the generalization performance of a classifier to new Rhesus subjects [across subjects (Rhesus): training on videos from several subjects and testing on videos of a new subject]. The blue lines denote the performance of the classifier across subjects using the parameters selected in the within-subject (Rhesus) case. The third group displays the generalization performance to new Fascicularis subjects (across species: training on videos from several Rhesus subjects and testing on videos of a new Fascicularis subject). In this case, the parameters should be tuned for each Fascicularis subject, and the results are the mean performance of two parameter sets (for the two Fascicularis subjects). Top, Performance for upper face. Bottom, Performance for lower face. D, Averaged confusion matrices of the KNN best performance results (of the three cases presented in C). The columns in each matrix represent the true labels, and the rows stand for the predicted labels. Top, Upper face confusion matrices. Bottom, Lower face confusion matrices (Extended Data Fig. 5-1, confusion matrix of inter-rater variability). E, Example of the KNN classification performance demonstrating correctly recognized frames along with some recognition errors. Each data point denotes a frame in a video. The classified AUs (magenta and green lines) are shown in comparison with the ground truth labels (black lines). Video time is displayed in the x-axis. Sample frames of the original video stream (after alignment and ROI cropping) are shown above the lines. The video for the example is taken from FD. Top, Output example for upper face video. Bottom, Output example for lower face video.
- Figure 6.
Examples of the method applications. A, Example of the final system output for monkey B from FD. Classification labels are presented on the y-axis, while the frame time of the video stream is on the X. “Other_upper” and “other_lower” labels are for video frames that were not part of the task of the classifier but exist in the original video and were labeled manually. Frames of the original video (with no preprocessing) are shown on the bottom, and the dashed lines denote their corresponding timing. The magenta and green lines demonstrate the outputs from the upper face and lower face algorithms, respectively. Images above the output lines exhibit the frames as they were processed in the algorithm, after alignment and ROI cropping. The estimated locations of the ROIs, comprising the full facial expressions, are illustrated in frames on the bottom by magenta and green rectangles (the positions are not precise since the original images on the bottom are not aligned). B, Facial expression analysis following classification of frames. Bars demonstrate the proportion of a specific facial configuration in monkey B (from FD) elicited during one block of the experiment described in Figure 2. This value is calculated as the ratio between frames containing the combination of AUs and the total frames per trial. Yellow bars denote the block part when the intruder monkey enters and exists the room, the blue bar is for phases with the closed shutter (after the first shutter opening and before its last closure), and the orange bars stand for periods of open shutter. An example image of the analyzed expression is shown on the right (taken from the examples in B). Top, Proportions of cooing facial expression events composed of UpperNone AU for the upper face and AU25 + 26 + 18i for the lower face. Bottom, Same as in top, but for “alert” facial expression: upper face, AU1 + 2; lower face, AU25 + 25 (Extended Data Fig. 6-1a, analysis following classification by human coders; **p < 1e-2; ***p < 1e-3). C, Same as A but for monkey D from FD. D, Same as B but for monkey D from FD and lip-smacking facial expression with upper face AU1 + 2 and lower face AU25+26 + 18i (Extended Data Fig. 6-1b, analysis following classification by human coders). E, PSTHs and raster plots of one neuron in the amygdala and one in the ACC, temporally locked to the socially associated AU25 + 26 + 18i, during monkey intruder block.
Tables
AU1 + 2 AU43_5 UpperNone Undersampled per class Total balanced training set Upper face frames, n 1213 ∼19,500 ∼150,000 1213 3639 AU25 + 26 + 16 AU25 + 26 + 18i AU25 + 26 Undersampled per class Total balanced training set Lower face frames, n 310 ∼15,000 ∼15,000 310 930 In the upper face, the smallest category was AU1 + 2 with only 1213 frames (in total, from all RD subjects). On the contrary, AU43_5 category had ∼19,500 frames (after eliminating RD AU45 frames because of time synchronization errors), and UpperNone class included >150,000 images. Consequently, balanced training sets were generated with each including all of the AU1 + 2 frames, and randomly selected 1213 frames from AU43_5 along with 1213 randomly selected UpperNone frames. Therefore, the upper face balanced training sets were each composed of 3639 frames. The same was done for the lower face, where the smallest category was AU25 + 26 + 16 with only 310 frames. Categories AU25 + 26 + 18i and AU25 + 26 contained >15,000 images each. Accordingly, each lower face balanced training set included 930 frames.
Figure 2-1
Lip-smacking interactions. Examples of dynamics and progression of lip-smacking interactions captured during the monkey–intruder experiment, where the subject monkey is the first to initiate the movement. Each sequence demonstrates sample frames of the Fascicularis subject D with his head fixed (first row), along with the corresponding frames of the intruder Fascicularis monkey (second row). The subject monkey D was filmed using the facial camera (see Materials and Methods). The intruder monkey was filmed using another monitoring camera, from the direction of the subject monkey and through the opened shutter (hence, the reflections on the screen). The time presented relative to the first frame in the sequence, which starts with a neutral expression of the subject monkey. Yellow arrows indicate the change in the movement of brows, ears, and lips at the onset of the lip-smacking movement (for the subject and the intruder monkeys) and the offset of the movement (for the intruder monkey). In the example, sequence with intruder monkey B. Download Figure 2-1, TIF file.
Figure 2-2
Lip-smacking interactions. Same setup as in Figure 2-1, but with intruder monkey P. Download Figure 2-2, TIF file.
Figure 2-3
Lip-smacking interactions. Same setup as in Figure 2-1, but with intruder monkey N. Download Figure 2-3, TIF file.
Figure 3-1
Motivation for alignment. Seven reference landmark points (yellow, predefined and common for all videos) displayed on sample neutral frames of original video streams. A, Sample neutral frames from five different videos of each of the five Rhesus monkeys (K, L, M, Q, R). B, Sample neutral frames from one video of Rhesus monkey K. C, Sample neutral frames of the two Fascicularis monkeys (D and B). Download Figure 3-1, TIF file.
Figure 5-1
Confusion matrix: inter-rater variability. Confusion matrix for the inter-rater variability between two experienced human coders, for a video from FD. “Other Upper” and “Other Lower” represent all the upper face and lower face labels that were not part of the task of the automatic classifier. Download Figure 5-1, DOCX file.
Extended Data 1
The archive “autoMaqFACS_code.zip” contains MATLAB code for autoMaqFACS classification. Download Extended Data 1, ZIP file
Figure 6-1
Facial expression analysis from ground truth labeling. Facial expressions analysis following frames classification by a human coder. Same as in Figure 5, C and E, but deduced from ground-truth labels. a, Monkey B from FD. b, Monkey D from FD. Download Figure 6-1, TIF file.
In this issue
