Cross-modal reorganization after sensory deprivation is a model for understanding brain plasticity. Although it is a well-documented phenomenon, we still know little of the mechanisms underlying it or the factors that constrain and promote it. Using fMRI, we identified visual motion-related activity in 17 early-deaf and 17 hearing adults. We found that, in the deaf, the posterior superior temporal gyrus (STG) was responsive to visual motion. We compared functional connectivity of this reorganized cortex between groups to identify differences in functional networks associated with reorganization. In the deaf more than the hearing, the STG displayed increased functional connectivity with a region in the calcarine fissure. We also explored the role of hearing aid use, a factor that may contribute to variability in cross-modal reorganization. We found that both the cross-modal activity in STG and the functional connectivity between STG and calcarine cortex correlated with duration of hearing aid use, supporting the hypothesis that residual hearing affects cross-modal reorganization. We conclude that early auditory deprivation alters not only the organization of auditory regions but also the interactions between auditory and primary visual cortex and that auditory input, as indexed by hearing aid use, may inhibit cross-modal reorganization in early-deaf people.