Compelling evidence shows that there is decreased neural specialization in the ventral visual cortex in older adults under passive viewing conditions. We assessed whether such specialization would be maintained on a working memory task and whether decreased specialization co-occurred with increased prefrontal activations. Participants encoded three faces or houses and maintained the set across an 8-s interval. Results provided evidence for less specialization of faces and houses in older adults than in younger adults, while the middle and inferior frontal cortex showed increased activation in elders compared with the young.