How the brain represents different aspects of faces remains controversial. Here we presented subjects with stimuli drawn from morph continua between pairs of famous faces. In the paired presentations, a second face could be identical to the first, could share perceived identity but differ physically (30% along the morph continuum), or could differ physically by the same distance along the continuum (30%) but in the other direction. We show that, behaviorally, subjects are more likely to classify face pairs in the third paired presentation as different and that this effect is more pronounced for subjects who are more familiar with the faces. In functional magnetic resonance imaging (fMRI), inferior occipital gyrus (IOG) shows sensitivity to physical rather than to identity changes, whereas right fusiform gyrus (FFG) shows sensitivity to identity rather than to physical changes. Bilateral anterior temporal regions show sensitivity to identity change that varies with the subjects' pre-experimental familiarity with the faces. These findings provide neurobiological support for a hierarchical model of face perception.