How does ventrolateral prefrontal cortex (VLPFC) control mnemonic processing? Alternative models propose that VLPFC guides top-down (controlled) retrieval of knowledge from long-term stores or selects goal-relevant products of retrieval from among competitors. A paucity of evidence supports a retrieval/selection distinction, raising the possibility that these models reduce to a common mechanism. Here, four manipulations varied semantic control demands during fMRI: judgment specificity, cue-target-associative strength, competitor dominance, and number of competitors. Factor analysis revealed evidence for a metafactor that accounted for common behavioral variance across manipulations and for functional variance in left mid-VLPFC. These data support a generalized control process that selects relevant knowledge from among competitors. By contrast, left anterior VLPFC and middle temporal cortex were sensitive to cue-target-associative strength, but not competition, consistent with a control process that retrieves knowledge stored in lateral temporal cortex. Distinct PFC mechanisms mediate top-down retrieval and postretrieval selection.