Blood oxygen-level-dependent signal decreases relative to baseline (deactivations) can occur with stimulation of an opposing sensory modality. Here, we show the importance of the difficulty of an auditory task on the deactivation of visual cortical areas. Participants performed an auditory temporal-order judgment task in conjunction with sparse-sampling functional MRI at both moderate and high levels of difficulty (adjusted for each individual's own threshold). With moderate difficulty, small deactivations were observed not only in parietal and cingulate cortex, but occipital cortex as well. When the same task was more difficult, deactivations increased significantly to include a greater extent of functionally defined visual cortex. Together, these results suggest that cross-modal deactivations occur in compensation for task difficulty, perhaps acting as an intrinsic filter for nonrelevant information.