Sex-discriminating facial features are examples of visual information involved in guiding social behavior. We used functional magnetic resonance imaging (fMRI) to assess brain responses in face-relevant brain areas in men and women during exposure to neutral male and female faces. An increased fMRI signal was found in the left amygdala and adjacent anterior temporal regions in men, but not in women, during exposure to faces of the opposite versus the same sex. These data indicate that the relationship between the sex of the subject and the sex of the face affects activity in the inferior temporal lobe. The sex-differential nature of this activation pattern may reflect sex differences in cognitive style and attentional processes when confronting faces of the opposite sex.