Improvement of visual contrast detection by a simultaneous sound

Abstract

Combining input from multiple senses is essential for successfully mastering many real-world situations. While several studies demonstrate that the presentation of a simultaneous sound can enhance visual detection performance or increase the perceived luminance of a dim light, the origin of these effect remains disputed. The suggestions range from early multisensory integration to changes in response bias and cognitive influences—implying that these effects could either result from relatively low-level, hard-wired connections of early sensory areas or from associations formed higher in the processing stream. To address this question, we quantified the effect of a simultaneous sound in various contrast detection tasks. A completely redundant sound did not alter detection rates, but only speeded reaction times. An informative sound, which reduced the uncertainty about the timing of the visual display, significantly improved detection rates, which manifested as a significant shift of the contrast detection curve. Surprisingly, this improvement occurred only in a paradigm were there was a consistent timing relation between sound and target and disappeared when subjects were not aware of the fact that the sound offered information about the visual stimulus. Altogether our findings suggest that cross-modal influences in such simple detection tasks are not exclusively mediated by hard-wired sensory integration but rather point to a prominent role for cognitive and attention-like effects.

Introduction

During everyday experience, auditory and visual stimuli are not separated into independent modalities but usually appear in close coordination. A snake wriggling through the grass makes a typical rustling sound, and thunderstorms impress both by lightning and thunder. In general, combining sensory information can enhance perceptual clarity and reduce ambiguity about the sensory environment (Ernst and Bulthoff, 2004, Stein and Meredith, 1993). For example, it has been demonstrated that combined sensory information can speed reaction times (Gielen et al., 1983, Hershenson, 1962, Posner et al., 1976), facilitate learning (Seitz et al., 2006) and change the qualitative sensory experience (Jousmaki and Hari, 1998, McGurk and MacDonald, 1976, Shams et al., 2000). Although many of these cross-modal phenomena are attributed to high-level cognitive processes, others are thought to arise from early and hard-wired sensory integration (Stein, 1998).

In particular, such early sensory integration is thought to mediate cross-modal improvement of low-level detection tasks. For example, a simultaneous tone improved detection of a dimly flashed light (Frassinetti et al., 2002a, Frassinetti et al., 2002b, McDonald et al., 2000, Teder-Salejarvi et al., 2005), enhanced the discriminability of briefly flashed visual patterns (Vroomen and de Gelder, 2000) or increased the perceived luminance of light (Stein et al., 1996). While these studies suggest that early sensory integration serves as basis for the improved visual performance, other studies propose that the observed effects result from biases of the cognitive decision process related to the particular paradigms employed (Doyle and Snowden, 2001, Odgaard et al., 2003).

One could conceive that cross-connections between early sensory areas, as for example demonstrated from the auditory to the visual cortex (Falchier et al., 2002, Rockland and Ojima, 2003), facilitate processing in one sense by input from another. It could also be that the superior colliculus, a subcortical structure containing many neurons responding to bi- or trimodal stimuli, is mediating cross-modal improvements in simple detection tasks (Stein, 1988, Stein and Meredith, 1993). However, many behavioral protocols used previously do not allow clear dissociation between early sensory integration and cognitive effects related to changes in decision making (Odgaard et al., 2003). For example, subjects could explicitly combine the information they gather from each sense and adjust their behavioral strategy depending on whether or not it seems advantageous on a cognitive level.

To address this controversy, we systematically quantified the effect of a simultaneous sound on a contrast detection task. We compared different paradigms based on the following reasoning: An early and automatic auditory influence on vision should occur regardless of whether the sound provides additional information about the visual stimulus or is redundant with the visual display. In addition, such an influence should not depend on the subjects' knowledge about the informative relation between both stimuli. A cognitive effect, however, should manifest only when the sound provides additional information over the visual stimulus, and even then, only when subjects are aware of the additional information.

To distinguish between these two possibilities, we manipulated the informative content of the sound. In different paradigms the temporal uncertainty of the visual stimulus was reduced by either the sound (“informative sound”), or by a visual cue that appeared simultaneously with target and which made the sound redundant (“redundant sound”). Additionally, we manipulated the subjects' knowledge about the informative content of the sound by randomizing the stimulus onset asynchrony. Our results demonstrate that a behavioral benefit of the sound occurs only in the “informative sound” condition, and only when the sound has a reliable and fixed timing relative to the visual target.