The surface area of early visual cortices varies several fold across healthy adult humans and is genetically heritable. But the functional consequences of this anatomical variability are still largely unexplored. Here we show that interindividual variability in human visual cortical surface area reflects a tradeoff between sensitivity to visual details and susceptibility to visual context. Specifically, individuals with larger primary visual cortices can discriminate finer orientation differences, whereas individuals with smaller primary visual cortices experience stronger perceptual modulation by global orientation contexts. This anatomically correlated tradeoff between discrimination sensitivity and contextual modulation of orientation perception, however, does not generalize to contrast perception or luminance perception. Neural field simulations based on a scaling of intracortical circuits reproduce our empirical observations. Together our findings reveal a feature-specific shift in the scope of visual perception from context-oriented to detail-oriented with increased visual cortical surface area.