Dedifferentiation, or decreased processing specificity, has been suggested to represent a ubiquitous characteristic of cognitive aging. In this study, we examined both age-related differences and intra-group differences in neural specificity in the ventral visual cortex for color, words, faces and places. Our results demonstrated that neural dedifferentiation was not ubiquitous across stimulus categories. Neural dedifferentiation was also relatively stable, across age, in a group of older adults. Older adults with more overall gray matter showed less neural dedifferentiation in the visual cortex. However, regional gray matter volume was not associated with neural dedifferentiation. We illustrate these effects using a discriminability metric, a signal detection theory measure, for neural dedifferentiation that takes into account both magnitude and variance of brain activation. The dedifferentiation measure provides a quantitative means to examine activation patterns and individual difference factors associated with neural dedifferentiation, and to test theories of behavioral dedifferentiation in cognitive aging literature.