Cortical cross-modal re-organization, or recruitment of auditory cortical areas for visual processing, has been well-documented in deafness. However, the degree of sensory deprivation necessary to induce such cortical plasticity remains unclear. We recorded visual evoked potentials (VEP) using high-density electroencephalography in nine persons with adult-onset mild-moderate hearing loss and eight normal hearing control subjects. Behavioral auditory performance was quantified using a clinical measure of speech perception-in-noise. Relative to normal hearing controls, adults with hearing loss showed significantly larger P1, N1, and P2 VEP amplitudes, decreased N1 latency, and a novel positive component (P2') following the P2 VEP. Current source density reconstruction of VEPs revealed a shift toward ventral stream processing including activation of auditory temporal cortex in hearing-impaired adults. The hearing loss group showed worse than normal speech perception performance in noise, which was strongly correlated with a decrease in the N1 VEP latency. Overall, our findings provide the first evidence that visual cross-modal re-organization not only begins in the early stages of hearing impairment, but may also be an important factor in determining behavioral outcomes for listeners with hearing loss, a finding which demands further investigation.