Single-trial multisensory memories affect later auditory and visual object discrimination
Introduction
A substantial body of work suggests that multisensory interactions can already occur at early latencies and within primary or near-primary cortices (reviewed in Murray et al., 2012, Van Atteveldt et al., 2014). Moreover, these interactions have been correlated with behavior (Cappe et al., 2012, Romei et al., 2007, Thelen et al., 2014, Van den Brink et al., 2013, Van der Burg et al., 2011). Cappe et al. (2012) found that increases in neuronal response strength at early latencies were positively correlated with multisensory gains in a motion discrimination task. Similarly, Romei et al. (2007) found correlations between multisensory events and the impact of a TMS pulse delivered over the occipital pole on auditory detection response speed. In another study, Van der Burg et al. (2011) showed auditory facilitation effects in a visual search task modulating activity within parieto-occipital cortices. Following up on the latter results, Van den Brink et al. (2013) found that this facilitation was predicted by the strength of anatomical connections between sub-cortical and cortical auditory structures.
While these and similar data reveal much about the instantaneous interactions between the senses, other studies have focused on how multisensory interactions taking place at one point in time have an impact on subsequent unisensory processing. For example, a large number of studies have investigated how unisensory stimulus discrimination and perceptual learning are affected by prior multisensory experiences (Gottfried et al., 2004, Nyberg et al., 2000, Shams and Seitz, 2008, Shams et al., 2011, von Kriegstein and Giraud, 2006, Wheeler et al., 2000). Likewise, Meylan and Murray (2007) showed that occipital cortical activation, due to the processing of visual stimuli was significantly attenuated when these stimuli were preceded by a multisensory stimulus. Our group has therefore specifically focused on how multisensory contexts may exert their influences in a more implicit manner and via single-trial exposures (Lehmann and Murray, 2005, Murray et al., 2005, Murray and Sperdin, 2010, Murray et al., 2004, Thelen et al., 2012, Thelen and Murray, 2013, Thelen et al., 2014). These studies show that visual object recognition is improved when the initial multisensory context had been semantically congruent and can be impaired if this context was either semantically incongruent or meaningless, when compared to recognition of visual stimuli only encountered in a unisensory visual context. More generally, these “single-trial” memories (i.e. memories that form after a single, initial pairing of a semantically congruent image and sound) of multisensory object associations are formed incidentally (i.e. parenthetically) and despite many intervening stimuli, are distinguishable from encoding processes, and promote distinct object representations that manifest as differentiable brain networks whose activity is correlated with recognition performance (Thelen & Murray, 2013).
Despite these advances in our understanding of multisensory memory and its impact on visual recognition, it is still not clear whether or not auditory object discrimination also benefits from (single-trial) multisensory memories. Some research would emphatically contend that auditory memory is grossly inferior to visual memory (Cohen, Horowitz, & Wolfe, 2009). Memory performance in a recognition task was impaired for sounds that had been paired with a corresponding image during the preceding study phase, as well as when the stimuli for the task were either speech stimuli or clips of music, which were considered to be richer in their content. The only situation wherein recognition memory for sounds was better than that for images was when the images were highly degraded. In terms of a putative explanation, Cohen et al. went so far as to suggest the following: “…auditory memory might be fundamentally different/smaller than visual memory. We might simply lack the capacity to remember more than a few auditory objects, however memorable, when they are presented one after another in rapid succession.” (p. 6010 of Cohen et al., 2009).
By this account, benefits of multisensory contexts on subsequent unisensory auditory discrimination may not be expected. If true, this would dramatically curtail potential applications of this paradigm to remediation or training situations; a central issue for the development of multisensory rehabilitation strategies across the lifespan (Johansson, 2012, White-Traut et al., 2013). By contrast, an alternative interpretation of the results of Cohen et al. (2009) may be warranted. This is based on an extension of the principle of inverse effectiveness (Altieri et al., 2013, Stein and Meredith, 1993, Stevenson et al., 2014). This interpretation would instead suggest that greater benefits would be observed in the sensory modality wherein information is less effective in eliciting a given behavior. If memory is generally less efficient in the auditory modality, then relatively greater gains from multisensory contexts would be expected. In accordance, Yuval-Greenberg and Deouell (2009) observed that visual information has a greater impact on auditory object identification than vice-versa. Likewise, selective delay-period activity on a delayed match-to-sample task was observed in intracellular recordings from monkey infero-temporal neurons not only when the animal performed a visual-to-visual task, but also when it performed either a visual-to-auditory or auditory-to-visual task (Gibson & Maunsell, 1997). This kind of neural response provides an indication that memory representations can be formed across the senses, and can also be activated by input from either sense alone. Likewise, functional imaging in humans is increasingly documenting the involvement of visual cortices in the categorical processing of sounds either via predictive coding (Vetter, Smith, & Muckli, 2014) or multisensory learning (von Kriegstein & Giraud, 2006; see also Schall et al., 2013, Sheffert and Olson, 2004).
It thus remains to be established (1) if auditory object discrimination is affected by single-trial multisensory memories and if so whether this is to the same degree as that observed in the visual modality, and (2) if there is a systematic relationship between memory performance in the visual and auditory modalities. Given these outstanding issues, the present study assessed the efficacy of multisensory exposures on auditory object discrimination during the completion of a continuous recognition task requiring the discrimination of initial from repeated sound object presentations. On the one hand, establishing such an effect will reveal whether or not auditory object processing has access to (and potentially benefits from) visual object representations, even when such information is task-irrelevant and occurred during initial object encoding. On the other hand and given the preponderance of auditory functional deficits following stroke (e.g. Griffiths, 2002), determining the ability of multisensory learning contexts to improve auditory memory functions in an incidental manner confers potential clinical applicability. By having the same set of participants also perform the task in the visual modality, we were able to compare the relative impact of single-trial and task-irrelevant multisensory contexts on subsequent unisensory memory functions (see also Cohen et al., 2009). This would reveal potential coupling and/or independence between the senses in terms of memory functions and by extension potential common resources.
Section snippets
Participants
The experiment included 26 adults (6 men) aged 17–41 years (mean age ± SD = 26 ± 6.16 years). 24 subjects were right-handed, according to the Edinburgh Inventory (Oldfield, 1971). No subject had a history of neurological or psychiatric illness, and all subjects had normal or corrected-to-normal vision and reported normal hearing. Subjects were either undergraduate students enrolled in psychology at the University of Lausanne (N = 13), who received course credit in exchange or were unpaid volunteers (N =
Gain/cost indices
The gain/cost index describes the relative percentage of accuracy enhancement or impairment for objects initially encountered in a multisensory vs. unisensory context, independently of general sensory modality related differences. These values were entered into a 2 × 3 repeated-measures ANOVA. There was no main effect of Task Modality (overall gain/cost ± s.e.m.: visual blocks = −1.44 ± 1.01%; vs. auditory blocks = −1.63 ± 1.37%; F(1,25) = 0.021; p = 0.885; ηp2 = 0.001), indicative of similar magnitudes of
Discussion
The present study demonstrated that the discrimination of objects presented in an auditory manner is affected by prior, single-trial multisensory experiences. In what follows we discuss results of the auditory recognition task in light of our prior and present findings in the visual modality with a particular focus on the potential inter-independence of multisensory influences on visual and auditory object discrimination. Further, since similar patterns of performance were observed for
Conclusions
Taken together, the present study shows that memory traces formed after single-trial multisensory encounters impact subsequent auditory object discrimination. To our knowledge this is the first demonstration of such effects. Moreover, we demonstrate there to be generally similar effects of prior multisensory contexts on both auditory and visual object discrimination in the same group of participants. This was the case even though raw performance was generally poorer in the auditory than visual
Acknowledgements
We thank Dr. Pawel Matusz for comments and discussion during the revision of this manuscript. Financial support was provided by the Swiss National Science Foundation (Grants 151771 to AT and 149982 to MMM). Additional support was provided by the Swiss Brain League (2014 Research Prize to MMM).
References (75)
- et al.
Conflict monitoring and anterior cingulate cortex: An update
Trends in Cognitive Sciences
(2004) - et al.
Auditory perceptual decision-making based on semantic categorization of environmental sounds
NeuroImage
(2012) - et al.
Remembrance of odors past: Human olfactory cortex in cross-modal recognition memory
Neuron
(2004) - et al.
Repetition priming for multisensory stimuli: Task-irrelevant and task-relevant stimuli are associated if semantically related but with no advantage over uni-sensory stimuli
Brain Research
(2009) - et al.
The role of multisensory memories in unisensory object discrimination
Brain Research. Cognitive Brain Research
(2005) - et al.
Auditory–visual multisensory interactions attenuate subsequent visual responses in humans
NeuroImage
(2007) - et al.
The costs of crossing paths and switching tasks between audition and vision
Brain and Cognition
(2009) - et al.
The brain uses single-trial multisensory memories to discriminate without awareness
NeuroImage
(2005) - et al.
Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging
NeuroImage
(2004) The assessment and analysis of handedness: The Edinburgh inventory
Neuropsychologia
(1971)
Early auditory sensory processing of voices is facilitated by visual mechanisms
NeuroImage
Enhanced EEG gamma-band activity reflects multisensory semantic matching in visual-to-auditory object priming
NeuroImage
Benefits of multisensory learning [Review]
Trends in Cognitive Sciences
Learning-induced plasticity in human audition: Objects, time, and space
Hearing Research
Electrical neuroimaging of memory discrimination based on single-trial multisensory learning
NeuroImage
Multisensory context portends object memory
Current Biology
Multisensory integration: Flexible use of general operations
Neuron
Early multisensory interactions affect the competition among multiple visual objects
NeuroImage
Decoding sound and imagery content in early visual cortex
Current Biology: CB
Mother–infant interaction improves with a developmental intervention for mother-preterm infant dyads
Infant Behavior & Development
Multisensory conflict modulates the spread of visual attention across a multisensory object
NeuroImage
Learning to associate auditory and visual stimuli: Behavioral and neural mechanisms
Brain Topography
Recent models and findings in visual backward masking: A comparison, review, and update
Perception & Psychophysics
When seeing a dog activates the bark: Multisensory generalization and distinctiveness effects
Experimental Psychology
The sensory nature of episodic memory: Sensory priming effects due to memory trace activation
Journal of Experimental Psychology: Learning, Memory, and Cognition
Looming signals reveal synergistic principles of multisensory interactions
Journal of Neuroscience
Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity
Journal of Experimental Psychology: Human Perception and Performance
Auditory recognition memory is inferior to visual recognition memory
Proceedings of the National Academy of Sciences of the United States of America
The fastest saccadic responses escape visual masking
PLoS One
A temporal hierarchy for conspecific vocalization discrimination in humans
The Journal of Neuroscience: The Official Journal of the Society for Neuroscience
Perceptual and semantic contributions to repetition priming of environmental sounds
Cerebral Cortex (New York, N.Y.: 1991)
Visual working memory capacity and stimulus categories: A behavioral and electrophysiological investigation
Experimental Brain Research
Effects of salience are short-lived
Psychological Science
The rapid distraction of attentional resources toward the source of incongruent stimulus input during multisensory conflict
Journal of Cognitive Neuroscience
Congruency sequence effects and cognitive control
Cognitive, Affective, & Behavioral Neuroscience
Sensory modality specificity of neural activity related to memory in visual cortex
Journal of Neurophysiology
Central auditory disorders: Toward a neuropsychology of auditory objects
Current Opinion in Neurology
Cited by (72)
Enriched learning: behavior, brain, and computation
2023, Trends in Cognitive SciencesCitation Excerpt :This early FFA response is followed by an increase in auditory cortex activity that peaks ~200 ms post-auditory onset and correlates with voice recognition performance, indicating that crossmodal responses are evoked during early stages of sensory processing. In addition, sounds that have been paired with pictures during prior learning elicit a distinct EEG time course as early as 35 ms post-auditory onset relative to sounds learned without pictures [72]. Source localization suggests that these differences are associated with increased responses in visual brain regions.
Does multisensory study benefit memory for pictures and sounds?
2022, CognitionCitation Excerpt :The full set of conditions with number of items in each condition is presented in Table 1. The experimental procedure was closely based on that of Thelen et al. (2015). Participants were tested individually.
Distinct multisensory perceptual processes guide enhanced auditory recognition memory in older cochlear implant users
2022, NeuroImage: ClinicalCitation Excerpt :This indicates that a multisensory stimulus context allows more accurate access to memory representations which can be utilized for subsequent top-down unisensory stimulus recognition. Consistent with this view, it has been previously shown that an audio-visual context – when compared to a unisensory context – facilitates the later recognition of visual (Murray et al., 2005, 2004; Thelen et al., 2012) and auditory objects in normal-hearing (NH) listeners (Matusz et al., 2015; Thelen et al., 2015). It is currently not well understood how multisensory facilitation in the context of auditory object recognition is affected by auditory deprivation and the use of hearing devices.
Audiovisual working memory and association with resting-state regional homogeneity
2021, Behavioural Brain ResearchCitation Excerpt :The results showed that subsequent recognition for sound stimuli was better during congruent audiovisual stimuli than neutral audiovisual stimuli or sounds presented in isolation. Similarly, other studies have found that recognition for visual stimuli is improved when they are initially presented with semantically congruent sounds compared to when the stimuli are presented with semantically incongruent sounds [6,7]. These studies mentioned above may suggest that audiovisual object encoding, particularly congruent audiovisual object encoding in working memory, can facilitate subsequent recognition of objects from a single sensory modality.