Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging

Abstract

Though commonly held that multisensory experiences enrich our memories and that memories influence ongoing sensory processes, their neural mechanisms remain unresolved. Here, electrical neuroimaging shows that auditory–visual multisensory experiences alter subsequent processing of unisensory visual stimuli during the same block of trials at early stages poststimulus onset and within visual object recognition areas. We show this with a stepwise analysis of scalp-recorded event-related potentials (ERPs) that statistically tested (1) ERP morphology and amplitude, (2) global electric field power, (3) topographic stability of and changes in the electric field configuration, and (4) intracranial distributed linear source estimations. Subjects performed a continuous recognition task, discriminating repeated vs. initial image presentations. Corresponding, but task-irrelevant, sounds accompanied half of the initial presentations during a given block of trials. On repeated presentations within a block of trials, only images appeared, yielding two situations—the image's prior presentation was only visual or with a sound. Image repetitions that had been accompanied by sounds yielded improved memory performance accuracy (old or new discrimination) and were differentiated as early as ∼ 60–136 ms from images that had not been accompanied by sounds through generator changes in areas of the right lateral–occipital complex (LOC). It thus appears that unisensory percepts trigger multisensory representations associated with them. The collective data support the hypothesis that perceptual or memory traces for multisensory auditory–visual events involve a distinct cortical network that is rapidly activated by subsequent repetition of just the unisensory visual component.

Introduction

Neurophysiological investigation of when and where in the brain one's memories or past experiences first affect responses to incoming stimuli has predominantly focused on the influence of unisensory memories; that is, stimulation of one sensory modality later influencing the responses to stimuli within the same modality. For example, studies of repetition priming have shown that behavior and brain responses change with repeated exposure to the same or similar stimuli (e.g., Doniger et al., 2001, Wiggs and Martin, 1998 for a recent review). More recently, investigations have begun examining how experiences in one or multiple senses alter later processing of stimuli of another sensory modality. These studies provide evidence that brain regions involved in an experience's encoding are also involved during its subsequent active retrieval (e.g., James et al., 2002, Nyberg et al., 2000, Wheeler et al., 2000). Subjects learned auditory–visual or visual–visual associations during a study session and later classified visual stimuli according to the sensory modality with which it initially appeared Nyberg et al., 2000, Wheeler et al., 2000. That auditory regions were active in response to visual stimuli that were first presented with sounds was taken as support for the theory of ‘redintegration’ (Hamilton, 1859), wherein a component part is sufficient to (re)activate the whole experience's consolidated representation. Intracranial microelectrode recordings in monkeys provide similar evidence by demonstrating selective delay activity during a delayed match-to-sample task with visual–visual, somatosensory–visual, and auditory–visual paired associates (e.g., Colombo and Gross, 1994, Gibson and Maunsell, 1997, Haenny et al., 1988, Maunsell et al., 1991). In these studies, responses were elicited in cortical areas involved in visual object recognition (i.e., areas V4 and IT) by nonvisual stimuli and were selective for specific associations among the learned set. One implication of these collective data is that prior multisensory experiences can influence and be part of memory functions such that when an association is formed between stimuli of different modalities, presentation of one stimulus can alter the activity in regions typically implicated in the processing of the modality of the other stimulus. That is, responses to an incoming stimulus may vary, either in terms of their pattern within a region or overall activated network, according to whether it is part of a multisensory or unisensory memory. However, it is not clear where or when (either in terms of time poststimulus or in terms of levels of processing) such effects first occur. These previous studies either lacked adequate temporal resolution or had limited spatial sampling to provide such information. Identification of the earliest effects can be used to place critical limits on the mechanisms of multisensory memory retrieval. It is similarly unclear whether such effects depend either on extensive training with the stimulus associations or on active classification of stimuli according to past experiences.

The aim of the present study was to determine the time course and initial locus of incidental effects of past multisensory experiences on current unisensory processes when subjects neither studied (through prolonged or repeated exposure) multisensory image–sound pairs nor later classified images according to the sense(s) initially stimulated. We applied high-density electrical neuroimaging techniques to this aim. We show that visual stimuli are rapidly differentiated according to their multisensory or unisensory association as early as ∼ 60–136 ms poststimulus onset in regions of the right lateral–occipital complex (LOC), a system of areas of the ventral visual cortical pathway involved in object recognition (e.g., Malach et al., 1995, Murray et al., 2002). This observation indicates that multisensory memories first alter visual sensory responses at early processing stages and provides evidence of the functional efficacy of prior multisensory experiences.