Semantic memory underpins our understanding of objects, people, places, and ideas. Anomia, a disruption of semantic memory access, is the most common residual language disturbance and is seen in dementia and following injury to temporal cortex. While such anomia has been well characterized by lesion symptom mapping studies, its pathophysiology is not well understood. We hypothesize that inputs to the semantic memory system engage a specific heteromodal network hub that integrates lexical retrieval with the appropriate semantic content. Such a network hub has been proposed by others, but has thus far eluded precise spatiotemporal delineation. This limitation in our understanding of semantic memory has impeded progress in the treatment of anomia. We evaluated the cortical structure and dynamics of the lexical semantic network in driving speech production in a large cohort of patients with epilepsy using electrocorticography (n = 64), functional MRI (n = 36), and direct cortical stimulation (n = 30) during two generative language processes that rely on semantic knowledge: visual picture naming and auditory naming to definition. Each task also featured a non-semantic control condition: scrambled pictures and reversed speech, respectively. These large-scale data of the left, language-dominant hemisphere uniquely enable convergent, high-resolution analyses of neural mechanisms characterized by rapid, transient dynamics with strong interactions between distributed cortical substrates. We observed three stages of activity during both visual picture naming and auditory naming to definition that were serially organized: sensory processing, lexical semantic processing, and articulation. Critically, the second stage was absent in both the visual and auditory control conditions. Group activity maps from both electrocorticography and functional MRI identified heteromodal responses in middle fusiform gyrus, intraparietal sulcus, and inferior frontal gyrus; furthermore, the spectrotemporal profiles of these three regions revealed coincident activity preceding articulation. Only in the middle fusiform gyrus did direct cortical stimulation disrupt both naming tasks while still preserving the ability to repeat sentences. These convergent data strongly support a model in which a distinct neuroanatomical substrate in middle fusiform gyrus provides access to object semantic information. This under-appreciated locus of semantic processing is at risk in resections for temporal lobe epilepsy as well as in trauma and strokes that affect the inferior temporal cortex-it may explain the range of anomic states seen in these conditions. Further characterization of brain network behaviour engaging this region in both healthy and diseased states will expand our understanding of semantic memory and further development of therapies directed at anomia.