It has been claimed that semantic dementia (SD), the temporal variant of fronto-temporal dementia, is characterized by an across-the-board deficit affecting all types of conceptual knowledge. We here confirm this generalized deficit but also report differential degrees of impairment in processing specific semantic word categories in a case series of SD patients (N = 11). Within the domain of words with strong visually grounded meaning, the patients' lexical decision accuracy was more impaired for color-related than for form-related words. Likewise, within the domain of action verbs, the patients' performance was worse for words referring to face movements and speech acts than for words semantically linked to actions performed with the hand and arm. Psycholinguistic properties were matched between the stimulus groups entering these contrasts; an explanation for the differential degrees of impairment must therefore involve semantic features of the words in the different conditions. Furthermore, this specific pattern of deficits cannot be captured by classic category distinctions such as nouns versus verbs or living versus nonliving things. Evidence from previous neuroimaging research indicates that color- and face/speech-related words, respectively, draw most heavily on anterior-temporal and inferior-frontal areas, the structures most affected in SD. Our account combines (a) the notion of an anterior-temporal amodal semantic "hub" to explain the profound across-the-board deficit in SD word processing, with (b) a semantic topography model of category-specific circuits whose cortical distributions reflect semantic features of the words and concepts represented.