Elsevier

Cognition

Volume 138, May 2015, Pages 148-160
Cognition

Single-trial multisensory memories affect later auditory and visual object discrimination

https://doi.org/10.1016/j.cognition.2015.02.003Get rights and content

Abstract

Multisensory memory traces established via single-trial exposures can impact subsequent visual object recognition. This impact appears to depend on the meaningfulness of the initial multisensory pairing, implying that multisensory exposures establish distinct object representations that are accessible during later unisensory processing. Multisensory contexts may be particularly effective in influencing auditory discrimination, given the purportedly inferior recognition memory in this sensory modality. The possibility of this generalization and the equivalence of effects when memory discrimination was being performed in the visual vs. auditory modality were at the focus of this study. First, we demonstrate that visual object discrimination is affected by the context of prior multisensory encounters, replicating and extending previous findings by controlling for the probability of multisensory contexts during initial as well as repeated object presentations. Second, we provide the first evidence that single-trial multisensory memories impact subsequent auditory object discrimination. Auditory object discrimination was enhanced when initial presentations entailed semantically congruent multisensory pairs and was impaired after semantically incongruent multisensory encounters, compared to sounds that had been encountered only in a unisensory manner. Third, the impact of single-trial multisensory memories upon unisensory object discrimination was greater when the task was performed in the auditory vs. visual modality. Fourth, there was no evidence for correlation between effects of past multisensory experiences on visual and auditory processing, suggestive of largely independent object processing mechanisms between modalities. We discuss these findings in terms of the conceptual short term memory (CSTM) model and predictive coding. Our results suggest differential recruitment and modulation of conceptual memory networks according to the sensory task at hand.

Introduction

A substantial body of work suggests that multisensory interactions can already occur at early latencies and within primary or near-primary cortices (reviewed in Murray et al., 2012, Van Atteveldt et al., 2014). Moreover, these interactions have been correlated with behavior (Cappe et al., 2012, Romei et al., 2007, Thelen et al., 2014, Van den Brink et al., 2013, Van der Burg et al., 2011). Cappe et al. (2012) found that increases in neuronal response strength at early latencies were positively correlated with multisensory gains in a motion discrimination task. Similarly, Romei et al. (2007) found correlations between multisensory events and the impact of a TMS pulse delivered over the occipital pole on auditory detection response speed. In another study, Van der Burg et al. (2011) showed auditory facilitation effects in a visual search task modulating activity within parieto-occipital cortices. Following up on the latter results, Van den Brink et al. (2013) found that this facilitation was predicted by the strength of anatomical connections between sub-cortical and cortical auditory structures.

While these and similar data reveal much about the instantaneous interactions between the senses, other studies have focused on how multisensory interactions taking place at one point in time have an impact on subsequent unisensory processing. For example, a large number of studies have investigated how unisensory stimulus discrimination and perceptual learning are affected by prior multisensory experiences (Gottfried et al., 2004, Nyberg et al., 2000, Shams and Seitz, 2008, Shams et al., 2011, von Kriegstein and Giraud, 2006, Wheeler et al., 2000). Likewise, Meylan and Murray (2007) showed that occipital cortical activation, due to the processing of visual stimuli was significantly attenuated when these stimuli were preceded by a multisensory stimulus. Our group has therefore specifically focused on how multisensory contexts may exert their influences in a more implicit manner and via single-trial exposures (Lehmann and Murray, 2005, Murray et al., 2005, Murray and Sperdin, 2010, Murray et al., 2004, Thelen et al., 2012, Thelen and Murray, 2013, Thelen et al., 2014). These studies show that visual object recognition is improved when the initial multisensory context had been semantically congruent and can be impaired if this context was either semantically incongruent or meaningless, when compared to recognition of visual stimuli only encountered in a unisensory visual context. More generally, these “single-trial” memories (i.e. memories that form after a single, initial pairing of a semantically congruent image and sound) of multisensory object associations are formed incidentally (i.e. parenthetically) and despite many intervening stimuli, are distinguishable from encoding processes, and promote distinct object representations that manifest as differentiable brain networks whose activity is correlated with recognition performance (Thelen & Murray, 2013).

Despite these advances in our understanding of multisensory memory and its impact on visual recognition, it is still not clear whether or not auditory object discrimination also benefits from (single-trial) multisensory memories. Some research would emphatically contend that auditory memory is grossly inferior to visual memory (Cohen, Horowitz, & Wolfe, 2009). Memory performance in a recognition task was impaired for sounds that had been paired with a corresponding image during the preceding study phase, as well as when the stimuli for the task were either speech stimuli or clips of music, which were considered to be richer in their content. The only situation wherein recognition memory for sounds was better than that for images was when the images were highly degraded. In terms of a putative explanation, Cohen et al. went so far as to suggest the following: “…auditory memory might be fundamentally different/smaller than visual memory. We might simply lack the capacity to remember more than a few auditory objects, however memorable, when they are presented one after another in rapid succession.” (p. 6010 of Cohen et al., 2009).

By this account, benefits of multisensory contexts on subsequent unisensory auditory discrimination may not be expected. If true, this would dramatically curtail potential applications of this paradigm to remediation or training situations; a central issue for the development of multisensory rehabilitation strategies across the lifespan (Johansson, 2012, White-Traut et al., 2013). By contrast, an alternative interpretation of the results of Cohen et al. (2009) may be warranted. This is based on an extension of the principle of inverse effectiveness (Altieri et al., 2013, Stein and Meredith, 1993, Stevenson et al., 2014). This interpretation would instead suggest that greater benefits would be observed in the sensory modality wherein information is less effective in eliciting a given behavior. If memory is generally less efficient in the auditory modality, then relatively greater gains from multisensory contexts would be expected. In accordance, Yuval-Greenberg and Deouell (2009) observed that visual information has a greater impact on auditory object identification than vice-versa. Likewise, selective delay-period activity on a delayed match-to-sample task was observed in intracellular recordings from monkey infero-temporal neurons not only when the animal performed a visual-to-visual task, but also when it performed either a visual-to-auditory or auditory-to-visual task (Gibson & Maunsell, 1997). This kind of neural response provides an indication that memory representations can be formed across the senses, and can also be activated by input from either sense alone. Likewise, functional imaging in humans is increasingly documenting the involvement of visual cortices in the categorical processing of sounds either via predictive coding (Vetter, Smith, & Muckli, 2014) or multisensory learning (von Kriegstein & Giraud, 2006; see also Schall et al., 2013, Sheffert and Olson, 2004).

It thus remains to be established (1) if auditory object discrimination is affected by single-trial multisensory memories and if so whether this is to the same degree as that observed in the visual modality, and (2) if there is a systematic relationship between memory performance in the visual and auditory modalities. Given these outstanding issues, the present study assessed the efficacy of multisensory exposures on auditory object discrimination during the completion of a continuous recognition task requiring the discrimination of initial from repeated sound object presentations. On the one hand, establishing such an effect will reveal whether or not auditory object processing has access to (and potentially benefits from) visual object representations, even when such information is task-irrelevant and occurred during initial object encoding. On the other hand and given the preponderance of auditory functional deficits following stroke (e.g. Griffiths, 2002), determining the ability of multisensory learning contexts to improve auditory memory functions in an incidental manner confers potential clinical applicability. By having the same set of participants also perform the task in the visual modality, we were able to compare the relative impact of single-trial and task-irrelevant multisensory contexts on subsequent unisensory memory functions (see also Cohen et al., 2009). This would reveal potential coupling and/or independence between the senses in terms of memory functions and by extension potential common resources.

Section snippets

Participants

The experiment included 26 adults (6 men) aged 17–41 years (mean age ± SD = 26 ± 6.16 years). 24 subjects were right-handed, according to the Edinburgh Inventory (Oldfield, 1971). No subject had a history of neurological or psychiatric illness, and all subjects had normal or corrected-to-normal vision and reported normal hearing. Subjects were either undergraduate students enrolled in psychology at the University of Lausanne (N = 13), who received course credit in exchange or were unpaid volunteers (N = 

Gain/cost indices

The gain/cost index describes the relative percentage of accuracy enhancement or impairment for objects initially encountered in a multisensory vs. unisensory context, independently of general sensory modality related differences. These values were entered into a 2 × 3 repeated-measures ANOVA. There was no main effect of Task Modality (overall gain/cost ± s.e.m.: visual blocks = −1.44 ± 1.01%; vs. auditory blocks = −1.63 ± 1.37%; F(1,25) = 0.021; p = 0.885; ηp2 = 0.001), indicative of similar magnitudes of

Discussion

The present study demonstrated that the discrimination of objects presented in an auditory manner is affected by prior, single-trial multisensory experiences. In what follows we discuss results of the auditory recognition task in light of our prior and present findings in the visual modality with a particular focus on the potential inter-independence of multisensory influences on visual and auditory object discrimination. Further, since similar patterns of performance were observed for

Conclusions

Taken together, the present study shows that memory traces formed after single-trial multisensory encounters impact subsequent auditory object discrimination. To our knowledge this is the first demonstration of such effects. Moreover, we demonstrate there to be generally similar effects of prior multisensory contexts on both auditory and visual object discrimination in the same group of participants. This was the case even though raw performance was generally poorer in the auditory than visual

Acknowledgements

We thank Dr. Pawel Matusz for comments and discussion during the revision of this manuscript. Financial support was provided by the Swiss National Science Foundation (Grants 151771 to AT and 149982 to MMM). Additional support was provided by the Swiss Brain League (2014 Research Prize to MMM).

References (75)

  • S. Schall et al.

    Early auditory sensory processing of voices is facilitated by visual mechanisms

    NeuroImage

    (2013)
  • T.R. Schneider et al.

    Enhanced EEG gamma-band activity reflects multisensory semantic matching in visual-to-auditory object priming

    NeuroImage

    (2008)
  • L. Shams et al.

    Benefits of multisensory learning [Review]

    Trends in Cognitive Sciences

    (2008)
  • L. Spierer et al.

    Learning-induced plasticity in human audition: Objects, time, and space

    Hearing Research

    (2011)
  • A. Thelen et al.

    Electrical neuroimaging of memory discrimination based on single-trial multisensory learning

    NeuroImage

    (2012)
  • A. Thelen et al.

    Multisensory context portends object memory

    Current Biology

    (2014)
  • N. Van Atteveldt et al.

    Multisensory integration: Flexible use of general operations

    Neuron

    (2014)
  • E. Van der Burg et al.

    Early multisensory interactions affect the competition among multiple visual objects

    NeuroImage

    (2011)
  • P. Vetter et al.

    Decoding sound and imagery content in early visual cortex

    Current Biology: CB

    (2014)
  • R. White-Traut et al.

    Mother–infant interaction improves with a developmental intervention for mother-preterm infant dyads

    Infant Behavior & Development

    (2013)
  • U. Zimmer et al.

    Multisensory conflict modulates the spread of visual attention across a multisensory object

    NeuroImage

    (2010)
  • N. Altieri et al.

    Learning to associate auditory and visual stimuli: Behavioral and neural mechanisms

    Brain Topography

    (2013)
  • B.G. Breitmeyer et al.

    Recent models and findings in visual backward masking: A comparison, review, and update

    Perception & Psychophysics

    (2000)
  • L. Brunel et al.

    When seeing a dog activates the bark: Multisensory generalization and distinctiveness effects

    Experimental Psychology

    (2013)
  • L. Brunel et al.

    The sensory nature of episodic memory: Sensory priming effects due to memory trace activation

    Journal of Experimental Psychology: Learning, Memory, and Cognition

    (2009)
  • C. Cappe et al.

    Looming signals reveal synergistic principles of multisensory interactions

    Journal of Neuroscience

    (2012)
  • Y.C. Chen et al.

    Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity

    Journal of Experimental Psychology: Human Perception and Performance

    (2011)
  • M.A. Cohen et al.

    Auditory recognition memory is inferior to visual recognition memory

    Proceedings of the National Academy of Sciences of the United States of America

    (2009)
  • S.M. Crouzet et al.

    The fastest saccadic responses escape visual masking

    PLoS One

    (2014)
  • M. De Lucia et al.

    A temporal hierarchy for conspecific vocalization discrimination in humans

    The Journal of Neuroscience: The Official Journal of the Society for Neuroscience

    (2010)
  • M. De Lucia et al.

    Perceptual and semantic contributions to repetition priming of environmental sounds

    Cerebral Cortex (New York, N.Y.: 1991)

    (2010)
  • S. Diamantopoulou et al.

    Visual working memory capacity and stimulus categories: A behavioral and electrophysiological investigation

    Experimental Brain Research

    (2011)
  • M. Donk et al.

    Effects of salience are short-lived

    Psychological Science

    (2008)
  • S.E. Donohue et al.

    The rapid distraction of attentional resources toward the source of incongruent stimulus input during multisensory conflict

    Journal of Cognitive Neuroscience

    (2013)
  • T. Egner

    Congruency sequence effects and cognitive control

    Cognitive, Affective, & Behavioral Neuroscience

    (2007)
  • J.R. Gibson et al.

    Sensory modality specificity of neural activity related to memory in visual cortex

    Journal of Neurophysiology

    (1997)
  • J.C. Goll et al.

    Central auditory disorders: Toward a neuropsychology of auditory objects

    Current Opinion in Neurology

    (2010)
  • Cited by (72)

    • Enriched learning: behavior, brain, and computation

      2023, Trends in Cognitive Sciences
      Citation Excerpt :

      This early FFA response is followed by an increase in auditory cortex activity that peaks ~200 ms post-auditory onset and correlates with voice recognition performance, indicating that crossmodal responses are evoked during early stages of sensory processing. In addition, sounds that have been paired with pictures during prior learning elicit a distinct EEG time course as early as 35 ms post-auditory onset relative to sounds learned without pictures [72]. Source localization suggests that these differences are associated with increased responses in visual brain regions.

    • Does multisensory study benefit memory for pictures and sounds?

      2022, Cognition
      Citation Excerpt :

      The full set of conditions with number of items in each condition is presented in Table 1. The experimental procedure was closely based on that of Thelen et al. (2015). Participants were tested individually.

    • Distinct multisensory perceptual processes guide enhanced auditory recognition memory in older cochlear implant users

      2022, NeuroImage: Clinical
      Citation Excerpt :

      This indicates that a multisensory stimulus context allows more accurate access to memory representations which can be utilized for subsequent top-down unisensory stimulus recognition. Consistent with this view, it has been previously shown that an audio-visual context – when compared to a unisensory context – facilitates the later recognition of visual (Murray et al., 2005, 2004; Thelen et al., 2012) and auditory objects in normal-hearing (NH) listeners (Matusz et al., 2015; Thelen et al., 2015). It is currently not well understood how multisensory facilitation in the context of auditory object recognition is affected by auditory deprivation and the use of hearing devices.

    • Audiovisual working memory and association with resting-state regional homogeneity

      2021, Behavioural Brain Research
      Citation Excerpt :

      The results showed that subsequent recognition for sound stimuli was better during congruent audiovisual stimuli than neutral audiovisual stimuli or sounds presented in isolation. Similarly, other studies have found that recognition for visual stimuli is improved when they are initially presented with semantically congruent sounds compared to when the stimuli are presented with semantically incongruent sounds [6,7]. These studies mentioned above may suggest that audiovisual object encoding, particularly congruent audiovisual object encoding in working memory, can facilitate subsequent recognition of objects from a single sensory modality.

    View all citing articles on Scopus
    View full text