Pre-training performances

Three days before the training session, all the subjects were tested in bi- and monaural conditions for auditory localisation of sounds. When combining all the subjects, the binaural performances were relatively precise (see Figure 2) with mean unsigned errors of 7.7±0.3°. Such performance level is similar to that reported in previous studies using, for example, a head orientation response (see Middlebrooks and Green 1991 for a review) but which is less than reported with a similar laser beam pointer apparatus [19]. When the subjects' performance is compared across the groups (A, AV and FB), there were no statistical differences between the three groups in the pre-training sound localization abilities (bootstrap analysis). In monaural condition, all subjects showed a dramatic alteration ins their abilities to localize a sound source in azimuth. As previously reported [3], plugging one ear induced a shift of responses toward the unplugged side (see Figure 2). In terms of accuracy of localization, we observed a strong increase in the mean unsigned error of the subjects compared to binaural conditions (all subjects combined, mean error 30.4±1.4°, p<0.05, Figure 2 right). A bootstrap statistical analysis revealed the same amount of deficit when comparing the three groups (groups A, AV and FB, Figure 2) in this pre-training session.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Performances of the subjects in the binaural and monaural conditions.

On the left, the horizontal axis represents the azimuthal positions of loudspeakers, the vertical axis represents the azimuthal response of the subjects. On the right, the mean pre-training responses collapsed across the positions are shown for each group of subjects. The dashed line is the ideal performance curve in this case (e.g., the sound source at 60° corresponds to the response at 60°).

https://doi.org/10.1371/journal.pone.0018344.g002

Post-training performances

In all groups, the subjects went through a daily session of monaural sound localization during 5 consecutive days. A direct comparison between pre- and post-training performance when the sound was presented alone, showed that the accuracy of the subjects increased after 5 days of practice. However, in spite of a daily training, the subjects never reach the performance level observed in binaural conditions during the pre-training test and in all groups the unsigned errors (in degree) remained statistically higher than that observed in the normal situation (bootstrap).

The improvement in sound localization varies according to the conditions of practice (auditory only, audiovisual, with feedback) and according to the side of stimulation with respect to the plugged ear. When comparing the unsigned error before and after training (Figure 3), one can see that this improvement is the smallest for the auditory training group, much higher for the group with feedback, and even higher for the audiovisual training group. For example, in the A-only group, the subjects present a global reduction of about 0.9° (±0.1°) in their errors localizing a sound source in azimuth. Such decrease in unsigned errors is small but significant (bootstrap) when comparing the pre- and post-training values.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Improvement after training in total and per side.

Improvement is presented as the difference in error (deviation from the sound source) before and after training. Error bars represent bootstrap bias-corrected and accelerated confidence intervals (p<0.05).

https://doi.org/10.1371/journal.pone.0018344.g003

In the group which received a visual indication of their performance accuracy (group FB), subjects also present an overall improvement in their sound localization performances expressed as a 10.8° (±0.1°) reduction in their mean unsigned error of sound localization.

In the audiovisual group (group AV), a spatially congruent visual stimulation was presented simultaneously to the sound. After the 5 training sessions, when tested in auditory alone conditions, this group showed the highest improvement in monaural sound localization with a global decrease in unsigned errors of 13.6° (±0.1°). Such increase in accuracy is statistically greater than that observed in the FB and A-only groups (respectively 10.8 and 0.9°, p<0.05 bootstrap). Of importance is that such an improvement by audiovisual training can be observed for all the spatial fields in the azimuth. The highest reduction in spatial errors was observed when the sound was presented in the 30–70° ispilateral to the plugged ear (20.6°±0.4° reduction). The improvement was the smallest in the −30°/+30° central region (6.3°±0.2° decrease) and intermediate when the sound appeared in the side ipsilateral to the unplugged ear (12.2°±0.4°). In the A-only and the FB groups, the amelioration of the performances of the subjects is also the strongest for sound location in the azimuth ipsilateral to the plugged ear. In the A-only and FB group the subjects tend to be worse in localizing the auditory stimuli when it appears in the central 30 degrees from the central fixation point on each side. On the opposite, in the AV group, the subjects showed a fairly significant improvement in sound localization for sound located in the central region. This can be also seen in Figure 4 where the pre-post-training difference in error is presented per loudspeaker. To clarify these differences in localizing sounds at the central positions, we analyzed the performance of the subjects with respect to the correct left/right discrimination. In the AV group, we observed a significant reduction (13±7%, p<0.05) in the lateralization errors when comparing pre- and post-training performance in auditory conditions for the central position (10–30° on both sides). On the opposite, both A-only and FB groups did not improve their performance in localizing the correct side of the sound (p>0.05). Such results can explain, at least partly, the absence of amelioration of sound localization when expressed in degrees.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Improvement after training per side per loudspeaker.

Improvement is presented as the difference in error (deviation from the sound source) before and after training. Error bars represent bootstrap bias-corrected and accelerated confidence intervals (p<0.05).

https://doi.org/10.1371/journal.pone.0018344.g004

Finally, we compared the performance of the audiovisually trained group for the audiovisual trials in the beginning of the training and in the end. This analysis was performed to check if the subjects were guided by the visual cues to localize the sound source. The errors during the audiovisual stimulation were minimal (4.5±0.3°, p<0.05) and were much lower than that observed for the same group in the binaural pre-training session (p>0.5). Further, no significant difference was found for any loudspeaker in the comparison of the training days (p>0.7). Such analysis suggests that the subjects were highly influenced by the visual stimuli to localize the sound but failed to reveal an improvement due to the training in the visual modality.

To conclude, we observed the highest significant improvement in monaural sound localization when the sound is simultaneously accompanied by the visual cue; such improvement can be obtained for all azimuth location of the sound source.

Changes in perceptual sensitivity

We used signal detection measures to separate perceptual (d′) and decision-level (ß) effects. On the basis of the similar performances between groups during the pre-training test, we searched for an effect of the training protocols (A, AV, or FB) on the evolution of the d′ values with the hypothesis of a larger increase of the perceptual sensitivity following the bimodal training. As explained in the method section, we considered as “hits” the responses located within 5° from the centre of the sound source. This method for the hits corresponds to the one applied in the feedback group to indicate the correct response during the experiment, thus we can compare directly the d′ values of all our groups. The direct comparison of pre- and post-training values of d′ (Figure 5) did not reveal a significant improvement for the A-only and feedback groups. On the opposite, when considering the audiovisual training group we observed a statistically increase in d′ values (bootstrap analysis) which can be interpreted as a facilitatory perceptual effect of the bimodal training on monaural sound localization.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. Changes of d′ and ß due to training.

Differential values after and before training are presented. Error bars represent bootstrap bias-corrected and accelerated confidence intervals (p<0.05).

https://doi.org/10.1371/journal.pone.0018344.g005

It is worth mentioning that the different training protocols are also expressed by different variations in the decision criteria of the subjects when pre- and post-training data are compared (Figure 5). Subjects in both the A-only and AV groups present a significant increase in the decision criteria, while we did not observe a variation in the Feedback group. Altogether, this suggests that the training procedures induced different strategies to localize accurately the sound by the subjects.

Monaural sound localization and practice

In normal binaural conditions, sound localization in the horizontal plane is performed by computing the differences in intensity level or time of arrival of the sound (ILD and ITD respectively) that are present between the two ears [1]. In case of monaural conditions, sound localization can be performed only by using the spectral cues provided by pinna filtering which amplifies or attenuates differently the frequencies according to the azimuthal sound sources [3]. In this case, the performances are poor in term of precision and the perception of sounds presented from the plugged ear is displaced toward the unplugged functional ear. Using an active laser pointing, we have replicated such results and we observed a large error in horizontal sound localization, over 30° of error, which correspond to that previously reported [20]. After a daily practice of sound localization with an ear plugged, we observed some reduction in the spatial errors of the subjects which is highly dependent on the protocol showing a significant advantage for a bimodal visuo-auditory training.

There are numerous evidences that normal hearing subjects can learn to localize a sound source when the spatial cues are experimentally modified [21], [22], either following plugging one ear [3], [21], [23] or after altering the spectral cues [24], [25]. In our experimental design, a moderate daily training of monaural sound localization is not sufficient to restore the same level of performance observed during natural binaural stimulation. We report only a weak improvement as a reduction of a few degrees in spatial error. Other studies using a chronic ear plug during one or several days have reported a higher rate of recovery of sound localization [3], [21]. Probably, during a continuous earplug, subjects are able to interact with the environment and therefore can use the visual information to recalibrate the altered spatial cues with the sound source location. The role of visual inputs in spatial auditory adaptation has been clearly demonstrated using modified vision [26] and is also present in our study as the simultaneous presentation of a spatially congruent visual stimulus leads to the greater amount of improvement.

Facilitation of auditory adaptation

Several hypotheses have been proposed to explain how subjects can adapt to the altered interaural cues induced by the earplug including internal representation [23]. During the unisensory A-only training, the only criterion available for the subjects to differentiate between the sounds coming from the unplugged and plugged sides was the possible effect of head shadow and/or pinna cues [5], [27]. The Head Shadow effect leads to the difference both in the intensity and spectral characteristics of the perceived sound because different frequencies are absorbed differently by the head [28], [29]. Such effect can be efficiently used only when the sounds to localized are of the same intensity such as in the present experiment. Further, a strategy based on such effect could also explain the higher improvement in all groups for auditory stimuli presented in peripheral location compared to central presentations.

However, in the AV and FB groups, additional mechanisms have to be considered because these subjects present a much higher level of adaptation to the altered binaural cues when compared to the A-only group. Perceptual learning [30] and associated brain plasticity mechanisms are probably participating to the amelioration of sound localization of these subjects. Perceptual learning corresponds to the improvement in perceptive performance induced by repeated sensory practice. The implication of multimodal perceptual learning in our protocol is reinforced by the observation of an increase in d′ values that reflect strict perceptual enhancement induced by the training. Such an increase in d′ values excludes the possibility that the visual stimulus is affecting the performance at a cognitive level while it does induce decisional changes as shown independently by the higher ß values. The role of feedback and top-down mechanisms have been shown to have a strong impact on perceptual learning [31]. In the FB group, in half of trials the subjects received a visual indication of the accuracy of their response. While the feedback signal in case of incorrect responses (a spatial error greater than 5°) does not provide to the subjects a magnitude of their mislocalization, subjects were able to use this signal to recalibrate the altered spatial cues.

One conceptual question which emerges from our results concerns the theoretical differences between the “feedback” inputs and multisensory interactions in the present conditions. In the group that received a bimodal visuo-auditory stimulation in half of the trials (AV group), the improvement can be also interpreted as resulting from the feedback mechanisms provided by the azimuth and time congruent LED. The temporal order of feedback inputs is important in the efficiency of perceptual learning [32]; the temporal and spatial congruencies serve as key features to obtain the maximal gain from multisensory interaction [11]. However, there are some important peculiarities that suggest that multisensory interaction might constitute different mechanisms supported by different neuronal processes and/or structures from the feedback part of our study. Firstly, the multisensory paradigm has induced an improvement in perceptual sensitivity (d′) and a modification of the decision criteria (ß). On the opposite, the feedback training did not influence neither the perceptual sensitivity nor the decisional criteria of the subjects. Further, the feedback training did not improve sound lateralization for the more misleading positions (10–30° from the centre), while the audiovisual training did reduce significantly the errors in allocating the sound to the correct hemi-field. Such result might account for the greater absolute errors observed in the feedback group when considering these positions. It should be noted that when the changes in both d′ and ß are present, their attribution is rather difficult as they could be related to increased sensitivity, cognitive bias, or both. Given that sound localization and auditory left/right lateralization involve probably different auditory structures (see [33], [34], [35]) we can suspect that the improvement obtained in our study by the AV and FB groups results from different mechanisms. Moreover, additional studies using variable temporal and spatial mismatch in the visuo-auditory stimuli need to be performed to dissociate clearly the benefits due to Feedback mechanisms from those multisensory integration.

Role of multisensory training

There is a large body of evidence for the importance of synergy between sensory modalities in our global perception and the associated behaviour [11]. Indeed, simultaneous polysensory stimulation results in qualitative percept distinct from those derived from a single uni-sensory stimulus [36]. Under specific context of congruency, multisensory integration results in perceptual improvements in various tasks, from simple detections to complex discriminations and memory [37], [38]. In addition, the role of multisensory interactions had been extended to visual learning and adaptation (reviewed in [39], [40]). In such cases, when comparing uni- and multisensory training, it has been shown that a multimodal practice induced a significantly better learning both in term of performance and of speed rate [14]. In addition, multisensory learning improves various types of sensory processing, such as visual motion detection [41] or visual temporal order judgment [42] and even auditory speech comprehension [43]. Our results in the present study show that benefits of crossmodal perceptual learning can be extended to auditory perception such as to sound localization. By presenting a temporally and spatially (in azimuth) congruent visual cue, subjects present a significant larger improvement in monaural localization, a result in line with the rule that multisensory perceptual learning depends on the congruency of the two sensory stimuli [41], [44].

Neuronal mechanisms of multisensory training

Our results showed that the repetitive presentation of the visual stimulus in the temporal and spatial congruence with the auditory stimulus can improve the performance for the auditory stimulus presented in isolation. Such results imply a convergence of the visual and auditory spatial representations in the brain. Several studies have pointed out the role of the tectum in merging auditory and visual spatial maps (see [45] for a review). Further, these modality specific maps are highly interdependent and any alterations of the visual or auditory modality during the development have a great impact on the spatial representation of the spared modality in the superior colliculus [46], [47], [48]; the mechanisms of sound localization are supported by a large network of subcortical and cortical regions [49], [50], [51]. Among these set of cortical areas, several studies have shown the role of the primary auditory cortex A1 in spatial hearing [33], [35], [52], [53]. Of interest for the present study, it has been shown that A1 contains strictly monaural cells that derived azimuth sensitivity for sound source from spectral cues [54], which are probably important for monaural sound localization [55]. Thus, a crucial question concerns the implication of the early stages of auditory processing in the improvement of monaural sound localization during visuo-auditory training. Recent studies in the ferret suggest an implication of A1 because after alteration of binaural cues, ferrets can recalibrate a sound source location by using visual cues [56], a mechanism that involves the primary auditory cortex [53].

In case of uni-sensory protocols, it has been proposed that perceptual learning is expressed by plastic changes that can occur at early cortical stages of sensory processing [57]. In the visual domain, in both animals [58], [59], [60] and human [61], perceptual learning induces modification of neuronal properties at the level of V1, the primary visual cortex. Similarly, both anatomical [17], [62], [63], [64] and electrophysiological animal studies [65], [66] have shown that the early stages of sensory processing, including V1 [67], are involved in multisensory processing [68]. In humans the implication of early unimodal sensory areas has been similarly shown during multisensory processing [69], [70], [71], [72]. Furthermore, in chronically blindfold subjects, intense Braille reading training induces crossmodal modifications at the level of the primary visual cortex [73] suggesting that crossmodal perceptual learning and multisensory interactions could share some common cortical network [74]. All together, it suggests that the improvement of monaural sound localization performance during the visuo-auditory training could be supported by the direct heteromodal connections that directly link visual and auditory areas [17]. The auditory cortex, in particular the caudal auditory areas involved in spatial processing [75], [76], is receiving direct inputs from the pre-striate cortex [63] originating specifically from the representation of the peripheral visual field. Such specificity in this visuo-auditory connection could account for our observation of a higher post-training improvement in localizing sound sources located over 30° of eccentricity. Thus, one could suggest that the visual presentation concomitant to the sound will reinforce the role of monaural spectral processing in A1 through Hebbian mechanisms [77] via the direct visual projections to the auditory cortex. However, we cannot exclude that the influence of multisensory training on monaural sound localization can be mediated in addition by a top-down influence originating from multisensory high-order areas. The caudal auditory cortex is receiving non-auditory inputs including visual, from the temporal, parietal and frontal lobes [78], [79], [80], which can participate in the recalibration of the sound source throughout the training sessions.

Implication for rehabilitation of patients with sensory loss

There is some data, though not related to spatial hearing, that ecological visual cues play a very important role in patients with unilateral cochlear implants (CI) helping them to restore the auditory modality. We have shown that in post-lingual CI recipient, patients maintain the high skill in lip-reading acquired during the prolonged period of deafness, even after several years of auditory recovery [81], [82]. Our previous observations suggest a synergetic perceptual facilitation involving the visual and the recovering auditory modalities, which can be observed both at the behavioural [81] and brain levels [83] in the speech domain. Furthermore, multisensory perceptual learning is improving speech comprehension in normal hearing subject tested with a degraded auditory information using a simulation of a cochlear implant [43]. Based on the present results, we can propose that the sound localization deficit observed in unilaterally CI deaf patients [6], [84] (Grantham et al 2004) could be reduced by intense visuo-auditory training. Such strategy of multisensory stimulation has been shown to be efficient in patients suffering of visual hemineglect and hemianopsia [85], [86].

Subjects

Eighteen normally-hearing subjects (mean age 25, range 20–40) participated in the protocol. They were divided into 3 groups with no distinction of age and gender between the groups (3 men and 3 women per group). All subjects reported no auditory or neurological disease and had normal or corrected to normal vision. All participants gave their full-informed consent prior to their participation in this study in accordance with the Declaration of Helsinki (1968). The study was approved by the local research ethics committee (Comite Consultatif de Protection des Personnes dans la Recherche Biomédicale Toulouse II Avis N°2-03-34/Avis N°2). The subjects were financially compensated for their participation.

Experimental set-up

The experiment was conducted in a dark soundproof anechoic room. A subject sat on a chair with his chin stabilized on a special framework (UHCOTech HeadSpot). During the experiment, the subjects were asked to fixate upon a green light-emitting diode in front of them which corresponded to the central loudspeaker. The study was realized in monaural conditions; one ear of the subject was plugged with an ear plug (average noise reduction 30 dB) and covered with an ear muff (average noise reduction 20 dB). The opposite muff was taken off, a sponge was glued to the resulting free end of the muffing device and it was comfortably placed behind the subject's open ear during the experiment.

The apparatus consisted of 15 piezoelectric loudspeakers arranged horizontally in semicircle with a radius of 0.5 m in front of the subject, the subject being in the centre of the semicircle (Figure 1). Loudspeakers in the semicircle were masked by a black acoustically transparent fabric so that the subject could not visually distinguish them. They were mounted on a plastic support that was held in place by 4 wooden stands fixed to the table. The angular positions of the loudspeakers were 70°, 60°, 50°, 40°, 30°, 20°, 10° with respect to the central loudspeaker (0°). Right above the centre of each loudspeaker, a red light-emitting diode (LED) was fixed.

Having perceived the sound from a loudspeaker (accompanied sometimes by the corresponding LED, see below), the subject had to indicate the source of the stimulation with a laser beam. This beam was projected from a rotating emitter in the centre of the semicircle of loudspeakers which can be manipulated by the subject through a manual knob. A home-made device, using a numerical potentiometer was allowed to record the position of the laser beam on the semicircle with a precision of 0.3°. A knob for rotating the laser emitter was on both the right and the left sides of the support so that the subjects could use either hand. Near each knob, there was a button to confirm the response. The subjects turned the knob with their preferred hand and pressed the button on the other side with the other hand. At time the response button was pressed, the laser position in degrees was registered. Having confirmed the response with the button, the subject repositioned the laser beam at the centre of the semicircle and waited for the next trail to start. The inter-trials time interval was random in the interval of 0.5–1.5 sec.

Auditory stimuli were the rectangular white noise (0,1–22 kHz) generated by Adobe Audition 3.0 lasting 50 ms and presented at the intensity of 60 dB SPL (measured at the centre of the semicircle of loudspeakers). Visual stimuli (red LEDs) of same duration (50 ms) were delivered simultaneously to the sound in cases of visuo-auditory conditions (see below). The LEDs were located above each loudspeaker (and above the black fabric covering the loudspeakers).

Experimental protocols

Three groups of 6 subjects participated to this protocol during which they underwent a testing session once a day during five consecutive days. Three days before (pre-training session), subjects had to perform the task in binaural and in monaural conditions to get familiar with the apparatus and to asses their pre-training sound localization abilities. The subjects were firstly engaged in 15 trials to familiarize them with the device, then 5 trials per loudspeaker were presented binaurally (75 random trials). Then the presentation of the monaural (to the left ear) auditory stimulation followed with 10 trials per loudspeaker resulting in 150 random trials. Having perceived the sound from a loudspeaker, the subject had to indicate the source of the stimulation with a laser beam as explained above. The pre-training session as well as the training sessions lasted about 1 hour including 2 pauses of 5 minutes each.

During the training sessions, the subjects were divided into three comparable groups. In one group, the conditions of stimulation were auditory only (Group A). In the second group, auditory stimuli were accompanied in half of the cases by an azimuth spatially congruent visual LED (Group AV). Audiovisual trials in this second group were presented randomly among auditory stimuli. In a third group of 6 subjects, stimulation was only auditory but in half of the cases a feedback was given to the subjects on the accuracy of their performance (Group FB). After pressing the button a small screen fixed above the central loudspeaker indicated “correct” or “incorrect”. A “Correct” message was given when the response was ±5° from the centre of the correct loudspeaker. If the response was outside this 5° range, the “incorrect” indication was presented. All the experimental conditions were similar to the ones of the auditory group (Group A).

Each day during 5 consecutive days, subjects were presented a session of 20 trials per loudspeaker (300 random trials). The performance of the subjects of all groups were analyzed and compared before (pre-training) and after this 5 days practice (post-training) during an auditory-alone presentation. Thus, by comparing the pre- and post-training performance, in Group A we assessed the effect of auditory practice, while in group AV we could observe the effect of audiovisual training on spatial monaural hearing in the horizontal plane.

Data analysis

Direct comparisons of the post- and pre-training performance, as well as between groups, were performed using the bootstrap method with bias-corrected and accelerated confidence intervals [87]. The effect was considered to be significant if there was no overlapping of confidence intervals at p<0.05.

First, we considered the difference in unsigned error (deviance of the response from the source of the sound in degrees) before and after training for each group. For each loudspeaker, the errors were re-sampled 60 times, then we calculated the mean post- pre-training difference for each sample and re-sampled the difference 10000 times to obtain confidence intervals per loudspeaker.

We also calculated the post- pre-training difference after dividing the semicircle into three sectors: the “plugged sector” corresponding to the responses to sound locations at 70°, 60°, 50°, 40°, and 30° on the side ipsilateral to the plugged ear, the central sector corresponding to the positions at 10° and 20° on both sides, the “unplugged” sector that encompass the locations at 70°, 60°, 50°, 40°, and 30° ipsilateral to the unplugged ear. For each sector, we re-sampled the error 300 times, calculated the mean post- pre-training difference for each sample and re-sampled the difference 10000 times to obtain confidence intervals per sector.

Finally, we applied SDT to analyse the performance of the subjects [88] to separate decisional bias from perceptual mechanisms. In this case, we considered as hits the responses located ±5° from the centre of the loudspeaker. We have chosen this value because it corresponded to the values applied in the feedback group (Group FB) to indicate a correct response of the subject. Then we calculated the post- pre-training differences in d′ and ß per sector. The differences between d′ and ß were calculated for each subject per loudspeaker and then re-sampled 10000 times to obtain confidence intervals per sector. The d′ and ß values were calculated according to the Matlab formulas:where HR is Hit Rate, FAR - False Alarm Rate and b is the input. Here, z_HR = −sqrt(2) * erfcinv(2*HR), where erfcinv is the inverse complementary error function. False alarm rate was determined as the response to the given loudspeaker when the sound was emitted elsewhere.