Differential Adaptation in Azimuth and Elevation to Acute Monaural Spatial Hearing after Training with Visual Feedback

Major findings

Our experiments demonstrate short-term adaptation of sound-localization performance for all subjects in response to a monaural plug and a short training session with explicit visuomotor feedback, for a fixed-intensity HP sound source, presented at a limited number of locations in the horizontal plane. We showed that the adaptation generalized to target locations across the two-dimensional frontal hemifield, and to sounds with different intensities, indicating that the adaptation involved a remapping of available acoustic cues, rather than a mere cognitive trick imposed by the particular set of trained stimuli. The plug and training session did not invoke an aftereffect in the azimuth responses (Fig. 8A), or in the elevation data (Fig. 13A).

Interestingly, although the plug perturbed the binaural intensity differences required for azimuth localization of higher sound frequencies, the adaptation affected not only the azimuth response components, but also the elevation components. Azimuth responses became more accurate and precise after the training (Fig. 9; Table 1), but, quite surprisingly, accuracy and precision of the elevation response components deteriorated (Fig. 11), even on the unaffected free-hearing side (Figs. 12, 15).

Ill-posed problems

To our knowledge, such differential effects of short-term training on sound-localization performance in the azimuth and elevation directions have not been reported before (but see also the section Azimuth versus elevation in acutely plugged early blind and chronic single-sided deaf). It indicates that the human sound-localization system is highly plastic, and continuously evaluates the current acoustic evidence against its internal representations. This fits well with the notion that to localize a sound, the human auditory system is in fact faced with a fundamental ill-posed problem (Middlebrooks and Green, 1991; Hofman and Van Opstal, 1998; Van Opstal, 2016): first, the ITD and ILD cues alone cannot uniquely encode sound-source direction, as all points on the so-called ‘cone of confusion’ yield identical ILD and ITD values (Blauert, 1997). Second, to disambiguate the cone of confusion, the system needs to estimate the source-elevation angle from the spectral pinna cues. However, because the sensory spectrum at the eardrum is always a convolution of the actual source spectrum and the direction-specific HRTF, both of which are a-priori unknown to the system, the extraction of elevation is ill-posed, even for a single source: infinitely many combinations of sound spectra and pinna filters (i.e., elevation angles) can generate the same sensory spectrum (Hofman and Van Opstal, 1998; Van Opstal, 2016). Third, the system should decide whether the acoustic input arose from a single source, or from multiple sources, which again poses an ill-posed problem that lacks a unique solution. Thus, on the basis of the acoustics alone, the auditory system cannot localize a sound source with absolute certainty.

To deal with this problem, the brain has to rely on additional (non-acoustic) sources of information, like visual input, priors regarding potential source locations, on the number of sources in the environment, and implicit assumptions about real-world source spectra and properties of its own pinna filters. It has been shown that the auditory system may indeed use such prior information to update its localization estimates (Hofman et al., 1998; Parise et al., 2014; Ege et al., 2018, 2019; Zonooz et al., 2018, 2019), and that it can rapidly learn to reweight its spectral contributions to the elevation percept. Experiments have also demonstrated strong plasticity to long-term changes in the spectral pinna cues (Hofman et al., 1998; Van Wanrooij and Van Opstal, 2005; Carlile et al., 2014), and in response to a visual manipulation with minifying eye-glasses (Zwiers et al., 2003). The latter indicates that visual feedback may be important in calibrating the auditory system (Zwiers et al., 2001).

Rapid adaptation

Recently, we reported that the auditory system can demonstrate rapid short-term adaptation of localization in the midsagittal plane to repeatedly presented LP filtered noises at only six possible target locations (Zonooz et al., 2018). The results showed that listeners improved elevation response accuracy to sounds across the two-dimensional frontal hemifield, after a similarly short training session with visual feedback as in the present study. Interestingly, responses even improved without providing the visual feedback, albeit to a lesser extent. Moreover, response changes were confined to the elevation response components, and did not affect the azimuth responses. We explained these data by assuming an increased weighting of the low-frequency spectrum in HRTFs that would be associated with an increased gain (i.e., accuracy) of the localization responses, without affecting the robust binaural difference cues.

Here, we observed adjustments of localization performance to visual feedback training after monauralization. Comparison of the pre-adaptation and post-adaptation results of the multiple regression analyses indicated that the azimuth responses had an increased contribution of both the proximal sound level cue (i.e., the ambiguous HSE), and the true target azimuth (Fig. 10). The latter could be mediated by different contributions from the spectral head- and pinna cues at the hearing ear, consistent with earlier studies (Kumpik et al., 2010; Keating and King, 2013; Keating et al., 2016), and the weak, but strongly perturbed level difference cues that may have survived the strong attenuation of the plug and muff (Keating et al., 2016). If spectral pinna cues would underlie the improved performance in azimuth, also the elevation responses might have benefited from the training. However, our elevation results (Figs. 13, 14) seem to suggest that an increased use of pinna cues from the hearing ear to azimuth localization is either unlikely, or somehow interferes with the estimation process for elevation (discussed below). It is not trivial as to why the major cue for elevation (spectral pinna cues) became in fact less effective after the training, even at the normal-hearing side. We hypothesize that the improvements in azimuth were due to an increased weighting of monaural head-shadow cues (proximal sound level and LP filtered spectral cues), and of a remapping of the weak, highly perturbed, binaural level-difference cues.

Although the HSE provides ambiguous localization cues, nearly all subjects increased its contribution during training (Fig. 10B). This strategy may have made perfect sense, as the training was provided for a single sound level only. Although listeners were not aware of this, they learned very quickly, through the visual feedback, that the perceived sound level actually provided them with a valid cue to localize the stimulus. In the same realm, the very weak binaural difference cues that survived the plug and muff for especially the higher sound levels, could have been remapped to reduce their strong leftward localization bias, and to increase the localization gain, as observed in the data of Figure 8.

Why would the perceived elevation suffer from this brief training session? In principle, there should be no need to change the contribution of the spectral pinna cues: for azimuth, their weight is low anyway (Van Wanrooij and Van Opstal, 2004, 2007), whereas for elevation, these cues are absolutely crucial (Carlile, 2014; Blauert, 1997; Hofman et al., 1998). Our data, however, show that the elevation response gains changed by reducing the spectral elevation cues (their partial correlation was reduced by ∼23% from s = 0.49 to s = 0.38), without changing the contribution of the azimuth cues (which stayed at about q = –0.14) and proximal level cues (stable at about p = –0.10). This unexpected change in elevation behavior (Figs. 11, 12) indicates that the sound-localization system flexibly and rapidly re-weighted the different localization cues (binaural differences, spectral cues, HSE cues) and updated its internal priors, consistent with the actual acoustic situation, even if these changes would hamper daily-life hearing situations.

Indeed, during the training, the auditory system was repeatedly exposed to stimuli that provided consistent head-shadow cues (target at fixed intensity and spectrum) and binaural level differences (albeit distorted), and at the same time, source elevation never changed (i.e., remained consistently at 0°). Therefore, the adopted strategy by the listeners could have been to use the (valid) head-shadow cue, to remap the weak, but consistent azimuth cue, and to drag the mean elevation estimate toward the horizon. The latter, however, resulted in further ignoring the actual spectral cues. By emphasizing a prior assumption toward the horizon (Parise et al., 2014; next section) induced a lower gain and correlation with the actual stimulus elevation (Fig. 11C). The system crudely remapped sources coming from the impaired side to more downward locations, and sources from the hearing side to more upward locations (Fig. 14C), despite the fact that these latter stimuli contained perfectly valid pinna-related elevation cues.

Azimuth versus elevation in acutely plugged early blind and chronic single-sided deaf

A recent study by Voss et al. (2015) on the monaural localization performance of acutely-plugged early-blind listeners demonstrated a similar negative coupling between azimuth and elevation performance than reported here for rapid adaptation in acutely-plugged sighted individuals. They grouped early blind listeners in two categories, according to their monaural azimuth performance: it either remained poor after plugging, just like in the pre-adaptation case of our sighted participants (Figs. 4, 5), or they immediately localized quite well with the plug, in which case they were shown to rely on spectral cues. Interestingly, this latter group (∼50% of their subjects) had poorer elevation performance than the former. Apparently, using spectral cues for azimuth localization (also under binaural hearing conditions in their daily lives) seemed incompatible with the use of spectral-shape cues for elevation.

In contrast, Van Wanrooij and Van Opstal (2004) described the azimuth and elevation results for chronic single-sided deaf (but normal-sighted) listeners, and showed that the more these listeners employed spectral cues for azimuth localization, the better they also localized in elevation. This suggested that spectral cues may in principle subserve both coordinates, given sufficient time (and perhaps, visual feedback). It also suggests that azimuth and elevation could rely on different, independent, but probably subtle, aspects of the HRTFs. The latter was also suggested by Voss et al. (2015).

Mechanisms

Figure 15 extends a conceptual model (after Van Wanrooij and Van Opstal, 2007) that summarizes how the different cues are weighted to generate the azimuth and elevation percepts for the three different hearing conditions. Under normal binaural hearing (Fig. 16A), source azimuth is fully determined by the ILDs and ITDs, as these are the most robust and reliable cues. Elevation is specified by the monaural HRTFs of the ipsilateral and contralateral ear, whereby perceived azimuth acts as a binaural weighting factor (Humanski and Butler, 1988; Morimoto, 2001; Hofman and Van Opstal, 2003). Under acute plugging (here: contralateral ear, c), the azimuth percept loses the ILDs, as they become highly distorted and uninformative, although the ITDs may still survive for the lower frequencies. For the higher frequencies, three ipsilateral cues have increased their contribution: the overall proximal sound-level (LEV_i), a (potential) spectral component from the LP filter of the head (LPF_i), as well as (information derived from) the ipsilateral HRTF (Van Wanrooij and Van Opstal, 2005, 2007). After training, we observed a considerable change in the weightings for high frequencies, and a concomitant decrease of the elevation gain. The latter is not explained by a further increase of the azimuth-related cues, as a windowed analysis on the azimuth gain and bias did not show such an effect (data not shown). Our results indicate that the azimuth percept became more reliant on the (weak) ILDs and on the spectral and level HSE, whereas the HRTF cues started to contribute less to elevation. The latter percept thus fell under a stronger influence from the trained prior that the target was always near the horizon.

• Download figure
• Open in new tab
• Download powerpoint

Figure 16.

Six acoustic cues can contribute to the perceived azimuth, whereas the left- and right HRTFs determine the elevation percept, modulated by an azimuth-dependent binaural weighting and internal priors. The strength (reliability) of a cue is indicated by line thickness. p,q,s are the partial correlations, obtained from Equations 2, 4. A, Under normal-hearing, the azimuth percept is fully determined by robust ITD and ILD cues, and the elevation percept mainly by the veridical HRTFs and azimuth. B, After acute monaural plugging (in the contra ear, c), azimuth is determined by low-frequency ITDs, and the monaural intensity and filter cues from the HSE. The resulting azimuth percept modulates the elevation percept, thereby decreasing the weight of the plugged ear. C, After training, the elevation percept is more strongly influenced by the prior (at the horizon), and less by the sensory spectral and azimuth cues.

Updating azimuth and elevation priors

Could a Bayesian model, in which the weights for the prior and spectral sensory cues are gradually updated, account for our results? Here, the idea would be that the auditory system assumed different, independent priors for azimuth and elevation during the different epochs of the experiment, relying on the current incoming target information (either acoustic, or otherwise). We recently suggested (Ege et al., 2018) that the normal-hearing auditory system adopts a bivariate prior for azimuth and elevation: a nearly uniform prior for azimuth (which therefore would be governed by maximum-likelihood estimation), and a Gaussian prior for elevation, centered around some default mean (Parise et al., 2014). The normal-hearing spatial prior could thus be described by (5)in which the width of the default elevation prior may be around 10–15° (Ege et al., 2018), (upward). Under acute plugged hearing, however, the azimuth percept strongly shifts to the hearing ear, prompting a new, and narrower azimuth prior: (6)where is the mean of the new azimuth prior, corresponding to the perceived (plug-induced) ILD, and its new width, .

During the training, visual feedback provides explicit information about the “true” target distribution, leading the listener to gradually assume that (7)with 10°, and The consequence of the changes in these different priors is that the azimuth and elevation gains both vary with the imposed experimental conditions: the broader the prior with respect to the sensory encodings, the more the percept relies on the sensory input. Conversely, the narrower the prior, the more the percept (response) is dominated by the prior (and less by the sensory stimulus). The optimal Bayesian model [relying on the maximum-a-posteriori (MAP) response decision] is then quantified (separately for azimuth and elevation) by (8)with the uncertainty in the sensory input (likelihood), and the width of the adopted prior. We recently provided evidence that the auditory system may in fact be suboptimal by following a different decision rule than the optimal MAP decision. In this strategy, the system aims to approximate, or match, the posterior distribution on a trial-by-trial basis by taking a random sample from the posterior (Ege et al., 2018). In this case, the response variance of Equation 8 will increase to (9)

According to either model, however, the azimuth percept under normal binaural hearing can depend entirely on the acoustic input, as the binaural difference cues are highly reliable ( ), so that from Equation 8: . The slightly more uncertain elevation percept, on the other hand, is mildly influenced by its prior, leading to a lower stimulus-response gain (i.e., around 0.8–0.9) than for azimuth (its gain is close to 1.0), and a small, often upward bias of a few degrees (Parise et al., 2014).

In the acute plugged condition, before feedback training, the azimuth percept becomes dominated by a new, much narrower azimuth prior , leading to a low azimuth gain, and a large bias toward the hearing ear: . The elevation percept will strongly follow the influence of its prior on the impaired side (because of the low confidence for the elevation cues: low gain), but is dominated by the spectral cues on the hearing side (high gain; Fig. 12, red symbols).

During training, however, the new elevation prior (horizon, i.e., small) starts to dominate, as more evidence accumulates across trials, leading to a gradually lower response gain across the entire frontal hemifield, including at the hearing side. At the same time, the azimuth gain will increase, as it can again rely more on the (updated) sensory (spectral, and/or distorted binaural) inputs, than on the increased variance of its prior. To assess which model may better account for the data in a quantitative way, their parameters should be fitted for the different experimental conditions and results. This effort, however, falls beyond the scope of the present study.

Visual feedback

One may wonder whether visual feedback would have been essential to induce the observed changes in localization behavior for the azimuth and elevation components. Although we have not tested this aspect in the present experiments, we conjecture that, like in a recent report on sound-source elevation (Zonooz et al., 2018), the auditory system might be able to construct a better estimate for source azimuth, merely from the repeated exposure to variations in perceived sound level, weak interaural difference cues (providing a strong bias toward the free ear), and the systematic spectral attenuation of high frequencies by the head, in combination with feedback about its own orienting movements (Hofman et al., 1998; Zwiers et al., 2001, 2003; Carlile et al., 2014). Especially the spectral attenuation by the head could provide a relatively simple and invariant monaural BB cue for source azimuth under natural hearing conditions as well, and as such serves as a valid reinforcement cue to reduce the large bias in perceived azimuth due to the plug. Note, however, that also this spectral head-shadow cue is ambiguous without prior assumptions regarding actual source spectra. However, the auditory system might infer a reasonable spectral estimate of the source from the repeated exposure to the same sound during training. It remains to be tested, however, whether the auditory system can indeed extract and combine these endogenous sources of information, and whether this would also lead to a degradation of elevation performance.