Previous neurophysiological and neuroimaging studies have shown that a cortical network involving the inferior frontal gyrus (IFG), inferior parietal lobe (IPL) and cortical areas in and around the posterior superior temporal sulcus (pSTS) region is employed in action understanding by vision and audition. However, the brain regions that are involved in action understanding by touch are unknown. Lederman et al. (2007) recently demonstrated that humans can haptically recognize facial expressions of emotion (FEE) surprisingly well. Here, we report a functional magnetic resonance imaging (fMRI) study in which we test the hypothesis that the IFG, IPL and pSTS regions are involved in haptic, as well as visual, FEE identification. Twenty subjects haptically or visually identified facemasks with three different FEEs (disgust, neutral and happiness) and casts of shoes (shoes) of three different types. The left posterior middle temporal gyrus, IPL, IFG and bilateral precentral gyrus were activated by FEE identification relative to that of shoes, regardless of sensory modality. By contrast, an inferomedial part of the left superior parietal lobule was activated by haptic, but not visual, FEE identification. Other brain regions, including the lingual gyrus and superior frontal gyrus, were activated by visual identification of FEEs, relative to haptic identification of FEEs. These results suggest that haptic and visual FEE identification rely on distinct but overlapping neural substrates including the IFG, IPL and pSTS region.