It is notoriously difficult to name odours. Without the benefit of non-olfactory information, even common household smells elude our ability to name them. The neuroscientific basis for this olfactory language 'deficit' is poorly understood, and even basic models to explain how odour inputs gain access to transmodal representations required for naming have not been put forward. This study used patients with primary progressive aphasia, a clinical dementia syndrome characterized by primary deficits in language, to investigate the interactions between olfactory inputs and lexical access by assessing behavioural performance of olfactory knowledge and its relationship to brain atrophy. We specifically hypothesized that the temporal pole would play a key role in linking odour object representations to transmodal networks, given its anatomical proximity to olfactory and visual object processing areas. Behaviourally, patients with primary progressive aphasia with non-semantic subtypes were severely impaired on an odour naming task, in comparison with an age-matched control group. However, with the availability of picture cues or word cues, odour matching performance approached control levels, demonstrating an inability to retrieve but not to recognize the name and nature of the odorant. The magnitude of cortical thinning in the temporal pole was found to correlate with reductions in odour familiarity and odour matching to visual cues, whereas the inferior frontal gyrus correlated with both odour naming and matching. Volumetric changes in the mediodorsal thalamus correlated with the proportion of categorical mismatch errors, indicating a possible role of this region in error-signal monitoring to optimize recognition of associations linked to the odour. A complementary analysis of patients with the semantic subtype of primary progressive aphasia, which is associated with marked temporopolar atrophy, revealed much more pronounced impairments of odour naming and matching. In identifying the critical role of the temporal pole and inferior frontal gyrus in transmodal linking and verbalization of olfactory objects, our findings provide a new neurobiological foundation for understanding why even common odours are hard to name.