Analysis of the spectral envelope of sounds by the human brain

doi:10.1016/j.neuroimage.2004.10.031

NeuroImage

Volume 24, Issue 4, 15 February 2005, Pages 1052-1057

https://doi.org/10.1016/j.neuroimage.2004.10.031 Get rights and content

Abstract

Spectral envelope is the shape of the power spectrum of sound. It is an important cue for the identification of sound sources such as voices or instruments, and particular classes of sounds such as vowels. In everyday life, sounds with similar spectral envelopes are perceived as similar: we recognize a voice or a vowel regardless of pitch and intensity variations, and we recognize the same vowel regardless of whether it is voiced (a spectral envelope applied to a harmonic series) or whispered (a spectral envelope applied to noise). In this functional magnetic resonance imaging (fMRI) experiment, we investigated the basis for analysis of spectral envelope by the human brain. Changing either the pitch or the spectral envelope of harmonic sounds produced similar activation within a bilateral network including Heschl's gyrus and adjacent cortical areas in the superior temporal lobe. Changing the spectral envelope of continuously alternating noise and harmonic sounds produced additional right-lateralized activation in superior temporal sulcus (STS). Our findings show that spectral shape is abstracted in superior temporal sulcus, suggesting that this region may have a generic role in the spectral analysis of sounds. These distinct levels of spectral analysis may represent early computational stages in a putative anteriorly directed stream for the categorization of sound.

Introduction

Spectral envelope defines the shape of the power spectrum of a complex sound. It is one means by which sound sources such as voices or sound classes such as vowels can be characterized. The spectral envelope of a sound is related to a dimension of timbre, the spectral centroid (Grey, 1977, Krumhansl and Iverson, 1992, McAdams and Cunible, 1992). Timbre is defined operationally as the property that distinguishes two sounds of identical pitch, duration, and intensity (American Standards Association, 1960). The spectral envelope is not the sole determinant of timbre, a property with other dimensions including temporal envelope (the attack and decay of a sound) and a third dimension that is not consistent between studies (Grey, 1977, Krumhansl and Iverson, 1992, McAdams and Cunible, 1992). Everyday experience suggests that we possess mechanisms for the abstraction of spectral envelope from the detailed spectrotemporal structure of the sound with which it is associated. We identify individual voices despite variations in pitch and intensity, and we perceive the same vowel sound whether the vowel is voiced or whispered. In the latter case, the same spectral envelope is applied to a harmonic series or to noise. Fig. 1 demonstrates the common spectral envelope of voiced and whispered vowels.

In this functional magnetic resonance imaging (fMRI) experiment, we investigated the analysis of spectral envelope by the human brain. We used stimuli in which ‘generic’ spectral envelopes were applied to harmonic series or to noise. These stimuli are likely to be processed by brain mechanisms that analyze the spectral envelopes of natural sounds such as vowels (Fig. 1), while avoiding the semantic associations of real vowels or musical instrumental timbres.

The experimental design (Fig. 2) employed sequences of sounds composed either entirely of harmonic sounds or from alternating harmonic sounds and noise. In stimulus sequences consisting entirely of harmonic sounds, spectral envelope or pitch was manipulated (Fig. 2: all-harmonic conditions). This manipulation allowed us to examine a level of analysis that could be based on the detailed spectrotemporal structure of sound. Changing the spectral envelope alters the detailed spectral structure of the sound while changing pitch (in stimuli such as these with unresolved harmonics) is associated with changes in the repetition rate and temporal structure of the sound. In both cases, there are changes to detailed spectrotemporal structure that are likely to be processed by early auditory areas in the superior temporal lobe. Based on previous studies of pitch change alone (Patterson et al., 2002, Warren et al., 2003), we predicted that the analysis of detailed spectrotemporal structure would engage a cortical network including nonprimary auditory cortex in Heschl's gyrus (HG) and planum temporale (PT).

In stimulus sequences consisting of alternating noise and harmonic sounds, spectral envelope was manipulated while the detailed spectrotemporal structure constantly varied (Fig. 2: alternating conditions). This manipulation allowed us to examine a level of analysis in which spectral envelope is abstracted independently of changes in the detailed spectrotemporal structure. We predicted that this level of spectral analysis would engage additional brain areas to those involved in the analysis of detailed spectrotemporal structure: these additional areas are required for the abstraction of spectral shape independently of changes in fine structure. Based on evidence for the involvement of the superior temporal sulcus (STS) in the identification of a variety of natural sounds (Adams and Janata, 2002, Beauchamp et al., 2004, Belin et al., 2000, Binder et al., 2004, Engelien et al., 1995, Lewis et al., 2004, Maeder et al., 2001, Menon et al., 2002, Nakamura et al., 2001, Zatorre et al., 2004), we predicted the specific involvement of STS in this more abstract level of spectral analysis. We hypothesized that the abstraction of the spectral shapes of sounds by STS is a general mechanism of auditory cognition, in addition to any more specific role of STS in the processing of particular sound categories.

Section snippets

Materials and methods

Stimuli were synthesized digitally in the frequency domain from harmonic series or fixed-amplitude, random-phase noise with equivalent passband and intensity (sampling rate 44.1 kHz and 16 bit resolution). Harmonic sounds were in positive Schroeder phase (Schroeder and Strube, 1986) to reduce peak factor. Spectral envelope was specified in the frequency domain for both noise and harmonic stimuli (Fig. 2). The duration of each sound was 500 ms (with 20 ms gating windows). Sounds were combined

Results

This experiment was designed to examine two levels of auditory analysis using contrasts based on two types of stimuli. In the all-harmonic contrasts, the baseline condition has constant spectrotemporal fine structure, whereas in the contrast between the alternating conditions, the baseline condition has constantly changing spectrotemporal fine structure (harmonic series or noise). Spectral envelope or pitch changes in the all-harmonic sequences can be assessed from changes in spectrotemporal

Discussion

This experiment has demonstrated distinct levels of analysis of spectral envelope that map onto distinct cortical regions in the human auditory brain. Changing the spectral envelope or the pitch of a harmonic sound engages a brain network that includes nonprimary auditory cortex in lateral HG and anterolateral PT. In these harmonic stimuli, both pitch and spectral envelope changes might be analyzed on the basis of the detailed spectrotemporal structure of the stimulus, and it is likely that

Acknowledgment

This work is supported entirely by the Wellcome Trust.

References (38)

R.B. Adams et al.
Comparison of neural circuits underlying auditory and visual object categorization
NeuroImage
(2002)
M.S. Beauchamp et al.
Integration of auditory and visual information about objects in superior temporal sulcus
Neuron
(2004)
F.T. Husain et al.
Relating neuronal dynamics for auditory object processing to neuroimaging activity: a computational modelling and an fMRI study
NeuroImage
(2004)
P.P. Maeder et al.
Distinct pathways involved in sound recognition and localization: a human fMRI study
NeuroImage
(2001)
V. Menon et al.
Neural correlates of timbre change in harmonic sounds
NeuroImage
(2002)
K. Nakamura et al.
Neural substrates for recognition of familiar voices: a PET study
Neuropsychologia
(2001)
R.D. Patterson et al.
The processing of temporal pitch and melody information in auditory cortex
Neuron
(2002)
J. Rademacher et al.
Probabilistic mapping and volume measurement of human primary auditory cortex
NeuroImage
(2001)
J.P. Rauschecker
Cortical processing of complex sounds
Curr. Opin. Neurobiol.
(1998)
G. Thierry et al.
Hemispheric dissociation in access to the human semantic system
Neuron
(2003)

C. Alain et al.

‘What’ and ‘where’ in the human auditory system

Proc. Natl. Acad. Sci. U. S. A.

(2001)

American Standards Association

Acoustical Terminology SI. 1-1960

(1960)

C.L. Barnes et al.

Efferent cortical connections of multimodal cortex of the superior temporal sulcus in the rhesus monkey

J. Comp. Neurol.

(1992)

G.C. Baylis et al.

Functional subdivisions of the temporal lobe neocortex

J. Neurosci.

(1987)

P. Belin et al.

Adaptation to speaker's voice in right anterior temporal lobe

NeuroReport

(2003)

P. Belin et al.

Voice-selective areas in human auditory cortex

Nature

(2000)

J.R. Binder et al.

Neural correlates of sensory and decision processes in auditory object identification

Nat. Neurosci.

(2004)

A. Engelien et al.

The functional anatomy of recovery from auditory agnosia. A PET study of sound categorization in a neurological patient and normal controls

Brain

(1995)

A.C. Evans et al.

3D statistical neuroanatomical models from 305 MRI volumes

Proc. IEEE Nucl. Sci. Symp. Med. Imag. Conf.

(1993)

Cited by (107)

The hearing hippocampus
2022, Progress in Neurobiology
The hippocampus has a well-established role in spatial and episodic memory but a broader function has been proposed including aspects of perception and relational processing. Neural bases of sound analysis have been described in the pathway to auditory cortex, but wider networks supporting auditory cognition are still being established. We review what is known about the role of the hippocampus in processing auditory information, and how the hippocampus itself is shaped by sound. In examining imaging, recording, and lesion studies in species from rodents to humans, we uncover a hierarchy of hippocampal responses to sound including during passive exposure, active listening, and the learning of associations between sounds and other stimuli. We describe how the hippocampus' connectivity and computational architecture allow it to track and manipulate auditory information – whether in the form of speech, music, or environmental, emotional, or phantom sounds. Functional and structural correlates of auditory experience are also identified. The extent of auditory-hippocampal interactions is consistent with the view that the hippocampus makes broad contributions to perception and cognition, beyond spatial and episodic memory. More deeply understanding these interactions may unlock applications including entraining hippocampal rhythms to support cognition, and intervening in links between hearing loss and dementia.
MASS: Multi-task anthropomorphic speech synthesis framework
2021, Computer Speech and Language
Citation Excerpt :
Therefore, if the high frequency is increased in the mel-spectrum, the emotion of the speech is likely to be transformed into an emotional emotion. As for spectrum, according to Warren’s (Warren et al., 2005) research, speech with similar spectral envelopes are considered similar. We obtain the spectrum envelope from the spectrum, and compare the spectrum envelope of the speech synthesized by MASS framework with the spectrum envelope of the target speaker’s speech to explore whether the synthesized speech contains speaker’s feature of the target speaker.
Text-to-Speech (TTS) synthesis plays an important role in human-computer interaction. Currently, most TTS technologies focus on the naturalness of speech, namely, making the speeches sound like humans. However, the key tasks of the expression of emotion and the speaker identity are ignored, which limits the application scenarios of TTS synthesis technology. To make the synthesized speech more realistic and expand the application scenarios, we propose a multi-task anthropomorphic speech synthesis framework (MASS), which can synthesize speeches from text with specified emotion and speaker identity. The MASS framework consists of a base TTS module and two novel voice conversion modules: the emotional voice conversion module and the speaker voice conversion module. We propose deep emotion voice conversion model (DEVC) and deep speaker voice conversion model (DSVC) based on convolution residual networks. It solves the problem of feature loss during voice conversion. The model trainings are independent of parallel datasets, and are capable of many-to-many voice conversion. In the emotional voice conversion, speaker voice conversion experiments, as well as the multi-task speech synthesis experiments, experimental results show DEVC and DSVC convert speech effectively. The quantitative and qualitative evaluation results of multi-task speech synthesis experiments show MASS can effectively synthesis speech with specified text, emotion and speaker identity.
Bilateral age-related atrophy in the planum temporale is associated with vowel discrimination difficulty in healthy older adults
2021, Hearing Research
Citation Excerpt :
Based on the dorsal and ventral stream framework by Hickok and Poeppel, a dorsal stream critical for auditory-motor integration, located within the posterior portion of the PT (area Spt i.e., Sylvian-parietal-temporal) (Hickok et al., 2003; 2009) gives reason to support the view for a separation between an anterior spatial hearing region of the PT (Buchsbaum et al., 2005; Isenberg et al., 2012). A recent study showed recruitment of dorsal stream areas in the PT when performing speech discrimination tasks (Rogalsky et al., 2020), while Warren et al. showed activation of anterolateral PT while changing spectral envelope or the pitch of a sound. In previous studies, the PT showed to be of importance in early auditory processing such as pitch perception (Binder et al., 1996).
In this study we investigated the association between age-related brain atrophy and behavioural as well as electrophysiological markers of vowel perception in a sample of healthy younger and older adults with normal pure-tone hearing. Twenty-three older adults and 27 younger controls discriminated a set of vowels with altered second formants embedded in consonant-vowel syllables. Additionally, mismatch negativity (MMN) responses were recorded in a separate oddball paradigm with the same set of stimuli. A structural magnet resonance scan was obtained for each participant to determine cortical architecture of the left and right planum temporale (PT). The PT was chosen for its function as a major processor of auditory cues and speech. Results suggested that older adults performed worse in vowel discrimination despite normal-for-age pure-tone hearing. In the older group, we found evidence that those with greater age-related cortical atrophy (i.e., lower cortical surface area and cortical volume) in the left and right PT also showed weaker vowel discrimination. In comparison, we found a lateralized correlation in the younger group suggesting that those with greater cortical thickness in only the left PT performed weaker in the vowel discrimination task. We did not find any associations between macroanatomical traits of the PT and MMN responses. We conclude that deficient vowel processing is not only caused by pure-tone hearing loss but is also influenced by atrophy-related changes in the ageing auditory-related cortices. Furthermore, our results suggest that auditory processing might become more bilateral across the lifespan.
Stroke and acquired amusia
2020, Music and the Aging Brain
Owing to the increasing prevalence of stroke in the aging population, more and more elderly persons suffer from acquired amusia, a music deficit caused by the stroke lesion. Similar to aphasia, acquired amusia has been recognized in clinical neurology for more than 100 years. Unlike aphasia, only very recently has its neural basis been systematically explored using modern neuroimaging techniques. This Chapter presents an overview of recent advances in mapping the structural and functional neural correlates of acquired amusia and its longitudinal recovery after stroke using multimodal magnetic resonance imaging (MRI) techniques, including voxel-based lesion–symptom mapping, voxel-based morphometry, diffusion tensor imaging, and functional MRI. The findings are discussed both in terms of what they reveal about the crucial brain regions and pathways underlying music perception and what their clinical relevance is for stroke recovery and rehabilitation.
Disorders of music processing in dementia
2020, Music and the Aging Brain
The musical brain is both highly susceptible and remarkably resilient to the effects of dementia. This apparent paradox reflects the complex and modular architecture of music cognition, the diversity of music processing disorders, and the characteristics of the diseases that cause dementia. The hierarchically organized, distributed brain networks that mediate different aspects of music processing are selectively targeted by neurodegenerative pathologies; accordingly, the major dementias have distinctive musical phenotypes and music is an informative probe of brain function in these diseases. In this chapter, we consider the rationale for studying music processing in the dementias, and the challenges this poses. We outline a taxonomy of musical disorders, neuroanatomical substrates, and disease associations. We highlight future avenues toward improved understanding of the musical brain in dementia, and how this may inform the development of musical therapies.
Neural architectures of music – Insights from acquired amusia
2019, Neuroscience and Biobehavioral Reviews
The ability to perceive and produce music is a quintessential element of human life, present in all known cultures. Modern functional neuroimaging has revealed that music listening activates a large-scale bilateral network of cortical and subcortical regions in the healthy brain. Even the most accurate structural studies do not reveal which brain areas are critical and causally linked to music processing. Such questions may be answered by analysing the effects of focal brain lesions in patients´ ability to perceive music. In this sense, acquired amusia after stroke provides a unique opportunity to investigate the neural architectures crucial for normal music processing. Based on the first large-scale longitudinal studies on stroke-induced amusia using modern multi-modal magnetic resonance imaging (MRI) techniques, such as advanced lesion-symptom mapping, grey and white matter morphometry, tractography and functional connectivity, we discuss neural structures critical for music processing, consider music processing in light of the dual-stream model in the right hemisphere, and propose a neural model for acquired amusia.

View all citing articles on Scopus

View full text

Analysis of the spectral envelope of sounds by the human brain

Abstract

Introduction

Section snippets

Materials and methods

Results

Discussion

Acknowledgment

NeuroImage

Neuron

NeuroImage

NeuroImage

NeuroImage

Neuropsychologia

Neuron

NeuroImage

Curr. Opin. Neurobiol.

Neuron

‘What’ and ‘where’ in the human auditory system

Proc. Natl. Acad. Sci. U. S. A.

Acoustical Terminology SI. 1-1960

Efferent cortical connections of multimodal cortex of the superior temporal sulcus in the rhesus monkey

J. Comp. Neurol.

Functional subdivisions of the temporal lobe neocortex

J. Neurosci.

Adaptation to speaker's voice in right anterior temporal lobe

NeuroReport

Voice-selective areas in human auditory cortex

Nature

Neural correlates of sensory and decision processes in auditory object identification

Nat. Neurosci.

The functional anatomy of recovery from auditory agnosia. A PET study of sound categorization in a neurological patient and normal controls

Brain

3D statistical neuroanatomical models from 305 MRI volumes

Proc. IEEE Nucl. Sci. Symp. Med. Imag. Conf.