Skip to main content

Advertisement

Log in

The dog’s meow: asymmetrical interaction in cross-modal object recognition

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Little is known on cross-modal interaction in complex object recognition. The factors influencing this interaction were investigated using simultaneous presentation of pictures and vocalizations of animals. In separate blocks, the task was to identify either the visual or the auditory stimulus, ignoring the other modality. The pictures and the sounds were congruent (same animal), incongruent (different animals) or neutral (animal with meaningless stimulus). Performance in congruent trials was better than in incongruent trials, regardless of whether subjects attended the visual or the auditory stimuli, but the effect was larger in the latter case. This asymmetry persisted with addition of a long delay after the stimulus and before the response. Thus, the asymmetry cannot be explained by a lack of processing time for the auditory stimulus. However, the asymmetry was eliminated when low-contrast visual stimuli were used. These findings suggest that when visual stimulation is highly informative, it affects auditory recognition more than auditory stimulation affects visual recognition. Nevertheless, this modality dominance is not rigid; it is highly influenced by the quality of the presented information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. In “semantic representation”, we include here also the access to the object’s name.

References

  • Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14(3):257–262

    PubMed  CAS  Google Scholar 

  • Albrecht DG, Geisler WS, Frazor RA, Crane AM (2002) Visual cortex neurons of monkeys and cats: temporal dynamics of the contrast response function. J Neurophysiol 88(2):888–913

    PubMed  Google Scholar 

  • Beauchamp MS, Lee KE, Argall BD, Martin A (2004) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41(5):809–823

    Article  PubMed  CAS  Google Scholar 

  • Ben-Artzi E, Marks LE (1995) Visual–auditory interaction in speeded classification: role of stimulus difference. Percept Psychophys 57(8):1151–1162

    PubMed  CAS  Google Scholar 

  • Bermant RI, Welch RB (1976) Effect of degree of separation of visual–auditory stimulus and eye position upon spatial interaction of vision and audition. Percept Mot Skills 42(43):487–493

    PubMed  CAS  Google Scholar 

  • Bertelson P (1999) Ventriloquism: a case of crossmodal perceptual grouping. In: Aschersleben G (ed) Cognitive contributions to the perception of spatial and temporal events. North-Holland/Elsevier, Amsterdam, pp 347–362

    Chapter  Google Scholar 

  • Bertelson P, Aschersleben G (2003) Temporal ventriloquism: crossmodal interaction on the time dimension. 1. Evidence from auditory–visual temporal order judgment. Int J Psychophysiol 50(1–2):147–155

    Article  PubMed  Google Scholar 

  • Bertelson P, Radeau M (1981) Cross-modal bias and perceptual fusion with auditory–visual spatial discordance. Percept Psychophys 29(6):578–584

    PubMed  CAS  Google Scholar 

  • Bullmore E, Horwitz B, Honey G, Brammer M, Williams S, Sharma T (2000) How good is good enough in path analysis of fMRI data? Neuroimage 11(4):289–301

    Article  PubMed  CAS  Google Scholar 

  • Calvert J, Manahilov V, Simpson WA, Parker DM (2005) Human cortical responses to contrast modulations of visual noise. Vision Res 45(17):2218–2230

    Article  PubMed  Google Scholar 

  • Driver J, Baylis GC (1993) Cross-modal negative priming and interference in selective attention. Bull Psychon Soc 31(1):45–48

    Google Scholar 

  • Elliott EM, Cowan N, Valle-Inclan F (1998) The nature of cross-modal color-word interference effects. Percept Psychophys 60(5):761–767

    PubMed  CAS  Google Scholar 

  • Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433

    Article  PubMed  CAS  Google Scholar 

  • Ernst MO, Banks MS, Bulthoff HH (2000) Touch can change visual slant perception. Nat Neurosci 3(1):69–73

    Article  PubMed  CAS  Google Scholar 

  • Greene AJ, Easton RD, LaShell LS (2001) Visual–auditory events: cross-modal perceptual priming and recognition memory. Conscious Cogn 10(3):425–435

    Article  PubMed  CAS  Google Scholar 

  • Hairston WD, Laurienti PJ, Mishra G, Burdette JH, Wallace MT (2003) Multisensory enhancement of localization under conditions of induced myopia. Exp Brain Res

  • Hairston WD, Wallace MT, Vaughan JW, Stein BE, Norris JL, Schirillo JA (2003b) Visual localization ability influences cross-modal bias. J Cogn Neurosci 15(1):20–29

    Article  PubMed  CAS  Google Scholar 

  • Hall SD, Holliday IE, Hillebrand A, Furlong PL, Singh KD, Barnes GR (2005) Distinct contrast response functions in striate and extra-striate regions of visual cortex revealed with magnetoencephalography (MEG). Clin Neurophysiol 116(7):1716–1722

    Article  PubMed  Google Scholar 

  • Heron J, Whitaker D, McGraw F (2004) Sensory uncertainty governs the extent of audio–visual interaction. Vision Res 44(25):2875–2884

    Article  PubMed  CAS  Google Scholar 

  • Huang J, Francis AP, Carr TH (2008) Studying overt word reading and speech production with event-related fMRI: a method for detecting, assessing, and correcting articulation-induced signal changes and for measuring onset time and duration of articulation. Brain Lang 104(1):10–23

    Article  PubMed  Google Scholar 

  • Larsen A, McIlhagga W, Baert J, Bundesen C (2003) Seeing or hearing? Perceptual independence, modality confusions, and crossmodal congruity effects with focused and divided attention. Percept Psychophys 65(4):568–574

    PubMed  Google Scholar 

  • Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19(4):213–223

    Article  PubMed  Google Scholar 

  • Lehmann S, Murray MM (2005) The role of multisensory memories in unisensory object discrimination. Brain Res Cogn Brain Res 24(2):326–334

    Article  PubMed  Google Scholar 

  • Lewis JL (1972) Semantic processing with bisensory stimulation. J Exp Psychol 96(2):455–457

    Article  PubMed  CAS  Google Scholar 

  • Marks LE (2004) Cross-modal interactions in speeded classification. In: Calvert GA, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT, Cambridge, pp 85–105

    Google Scholar 

  • Marks LE, Ben-Artzi E, Lakatos S (2003) Cross-modal interactions in auditory and visual discrimination. Int J Psychophysiol 50(1–2):125–145

    Article  PubMed  Google Scholar 

  • McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264(5588):746–748

    Article  PubMed  CAS  Google Scholar 

  • Michelson A (1927) Studies in optics. University of Chicago Press, Chicago

    Google Scholar 

  • Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual–auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14(4):452–465

    Article  PubMed  Google Scholar 

  • Morein-Zamir S, Soto-Faraco S, Kingstone A (2003) Auditory capture of vision: examining temporal ventriloquism. Brain Res Cogn Brain Res 17(1):154–163

    Article  PubMed  Google Scholar 

  • Mynatt BT (1977) Reaction times in a bisensory task: implications for attention and speech perception. J Exp Psychol Hum Percept Perform 3(2):316–324

    Article  PubMed  CAS  Google Scholar 

  • Näsänen R, Ojanpää H, Tanskanen T, Päällysaho J (2006) Estimation of temporal resolution of object identification in human vision. Exp Brain Res 172(4):464–471

    Article  PubMed  Google Scholar 

  • Porciatti V, Bonanni P, Fiorentini A, Guerrini R (2000) Lack of cortical contrast gain control in human photosensitive epilepsy. Nat Neurosci 3(3):259–263

    Article  PubMed  CAS  Google Scholar 

  • Recanzone GH (2003) Auditory influences on visual temporal rate perception. J Neurophysiol 89(2):1078–1093

    Article  PubMed  Google Scholar 

  • Reich DS, Mechler F, Victor JD (2001) Temporal coding of contrast in primary visual cortex: when, what, and why. J Neurophysiol 85(3):1039–1050

    PubMed  CAS  Google Scholar 

  • Rock I, Victor J (1964) Vision and touch: an experimentally created conflict between the two senses. Science 143:594–596

    Article  PubMed  CAS  Google Scholar 

  • Roelofs A (2005) The visual–auditory color-word stroop asymmetry and its time course. Memory Cogn 33(8):1325–1336

    Google Scholar 

  • Sen A, Posner MI (1979) The effect of unattended visual and auditory words on cross-modal naming. Bull Psychon Soc 13(6):405–408

    Google Scholar 

  • Shimada H (1990) Effect of auditory presentation of words on color naming: the intermodal Stroop effect. Percept Mot Skills 70(3 Pt 2):1155–1161

    PubMed  CAS  Google Scholar 

  • Stroop JR (1935) Studies of interference in serial verbal reactions. J Exp Psychol 18:643–662

    Article  Google Scholar 

  • Stuart DM, Carrasco M (1993) Semantic component of a cross-modal Stroop-like task. Am J Psychol 106(3):383–405

    Article  Google Scholar 

  • Tellinghuisen DJ, Nowak EJ (2003) The inability to ignore auditory distractors as a function of visual task perceptual load. Percept Psychophys 65(5):817–828

    PubMed  Google Scholar 

  • von Kriegstein K, Giraud AL (2006) Implicit multisensory associations influence voice recognition. PLoS Biol 4(10):1809–1820

    Google Scholar 

  • Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88(3):638–667

    Article  PubMed  CAS  Google Scholar 

  • Witten IB, Knudsen EI (2005) Why seeing is believing: merging auditory and visual worlds. Neuron 48(3):489–496

    Article  PubMed  CAS  Google Scholar 

  • Yuval-Greenberg S, Deouell LY (2007) What you see is not (always) what you hear: induced gamma band responses reflect cross-modal interactions in familiar object recognition. J Neurosci 27(5):1090–1096

    Article  PubMed  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Shlomit Yuval-Greenberg.

Electronic supplementary material

Below is the link to the electronic supplementary material

221_2008_1664_MOESM1_ESM.pdf

Supplemental Fig. 1: Facilitation and interference compared to neutral. For reaction time, the congruent (C, crossed bars) or incongruent (IC, dotted bars) RTs were subtracted from the neutral RT. For accuracy, the neutral hit rate was subtracted from the congruent or incongruent hit rate. Thus, facilitation is upwards, interference downwards for both measures. Dark bars attend-visual, light bars attend-auditory. Error bars reflect the standard error. a Performance on the short-delay condition of Experiment 1. b Performance on the long delay condition of Experiment 1. c Performance on Experiment 2 (PDF 108 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Yuval-Greenberg, S., Deouell, L.Y. The dog’s meow: asymmetrical interaction in cross-modal object recognition. Exp Brain Res 193, 603–614 (2009). https://doi.org/10.1007/s00221-008-1664-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-008-1664-6

Keywords

Navigation