The utilization of symbols such as words and numbers as mental tools endows humans with unrivalled cognitive flexibility. In the number domain, a fundamental first step for the acquisition of numerical symbols is the semantic association of signs with cardinalities. We explored the primitives of such a semantic mapping process by recording single-cell activity in the monkey prefrontal and parietal cortices, brain structures critically involved in numerical cognition. Monkeys were trained to associate visual shapes with varying numbers of items in a matching task. After this long-term learning process, we found that the responses of many prefrontal neurons to the visual shapes reflected the associated numerical value in a behaviorally relevant way. In contrast, such association neurons were rarely found in the parietal lobe. These findings suggest a cardinal role of the prefrontal cortex in establishing semantic associations between signs and abstract categories, a cognitive precursor that may ultimately give rise to symbolic thinking in linguistic humans.