Memorable audiovisual narratives synchronize sensory and supramodal neural responses

Abstract

Our brains integrate information across sensory modalities to generate perceptual experiences and form memories. However, it is difficult to determine the conditions under which multisensory stimulation will benefit or hinder the retrieval of everyday experiences. We hypothesized that the determining factor is the reliability of information processing during stimulus presentation, which can be measured through inter-subject correlation of stimulus evoked activity. We therefore presented biographical auditory narratives and visual animations to 72 human subjects either visually, auditorily, or combined while neural activity was recorded using electroencephalography. Memory for the narrated information, contained in the auditory stream, was tested three weeks later. While the visual stimulus alone led to no meaningful retrieval, this related stimulus improved memory when it was combined with the story, even when it was temporally incongruent with the audio. Further, individuals with better subsequent memory elicited neural responses during encoding that were more correlated with their peers. Surprisingly, portions of this predictive synchronized activity were present regardless of the sensory modality of the stimulus. These data suggest that the strength of sensory and supramodal activity is predictive of memory performance after three weeks and that neural synchrony may explain the mnemonic benefit of the functionally uninformative visual context observed for these real-world stimuli.

Significance Statement: Although multisensory integration is an important part of daily life, the mnemonic influence of one modality on another is not well established. Cross-modal cues may either strengthen or interfere with memory for information imparted through another sensory modality. We establish that during the encoding of a naturalistic auditory stimulus the cross-subject synchrony of neural processing predicts memory performance regardless of stimulus modality. The dominant neural signature of enhanced encoding is supramodal in that it is largely independent of the modality of stimulus presentation. The level of synchrony that a story elicits may help predict the extent to which adding extraneous information benefits memory.