The effects of early-onset blindness on the development of the visual system have been explained traditionally by the stabilization of transient connections through Hebbian competition. Although many of the findings from congenital cataract and congenital blindness are consistent with that view, there is inconsistent evidence from studies of visual cognition in children treated for visual deprivation from cataract, case reports of recovery of vision in adults, and studies of visual reorganization after late-onset blindness. Collectively, the data from congenital cataract and congenital blindness indicate that early visual experience sets up the infrastructure for later learning involving both the dorsal ("where") and ventral ("what") streams. Nevertheless, there is surprising residual plasticity in adulthood that can be revealed if vision is lost either temporarily or permanently. This has important implications for understanding the role of early visual experience in shaping visual cognitive development.