Brain-based decoding of mentally imagined film clips and sounds reveals experience-based information patterns in film professionals
Introduction
With awe one can watch breath-taking thrillers, or laugh out loud at a silly comedy film. While an increasing body of functional MRI (fMRI) studies have looked at neural responses of film viewers (Bartels and Zeki, 2004, Hasson et al., 2004, Jaaskelainen et al., 2008, Lahnakoski et al., 2012, Naselaris et al., 2009), no studies have focused on the neural basis of how filmmakers create these images and sounds in their minds. The process of constructing and manipulating mental representations in the human brain is called mental imagery. Professional filmmakers, such as cinematographers and sound designers, can be argued to stand as experts in modality-specific mental imagery. On a daily basis they need to imagine and mentally manipulate either visual (cinematographers) or auditory (sound designers) information in order to reach technical and artistic decisions. In this study we report neuroimaging findings suggesting that cinematographers and sound designers use differentiated brain regions to construct visual and auditory mental representations.
Neuroimaging studies on mental imagery have shown that when we imagine something that we cannot directly see or hear, the brain utilizes similar brain regions as during related sensory perception. The imagery of objects, such as houses or faces, activates category-specific regions in ventral temporal cortex (Ishai et al., 2000, Ishai et al., 2002, O'Craven and Kanwisher, 2000). Tasks involving spatial manipulation of imagined content revealed activity in the dorsal stream, in particular in the posterior parietal cortex (Bien and Sack, 2014, Cohen et al., 1996, Formisano et al., 2002, Mellet et al., 1996, Sasaoka et al., 2014, Tagaris et al., 1996, Trojano et al., 2000). In addition, a set of regions in the frontal cortex, including medial superior frontal gyrus, premotor cortex and middle frontal cortex, have shown to control construction and maintenance of the mental image (D'Esposito et al., 1995, de Borst et al., 2012, Marklund et al., 2007). Whether the similarities between imagery and perception include the primary sensory cortices has been a matter of discussion. While for visual imagery some studies reported early visual cortex activation (Ganis et al., 2004, Ishai et al., 2002, Klein et al., 2000, Kosslyn et al., 1993, Kosslyn et al., 1999), others did not (Formisano et al., 2002, Ishai et al., 2000, Knauff et al., 2000, Mellet et al., 1996, Mellet et al., 2000b, Trojano et al., 2000). The involvement of early visual cortex seems to especially rely on the mental representation of fine-grained details rather than the spatial properties of objects or scenes (Kosslyn and Thompson, 2003, Mellet et al., 2000a). For auditory imagery, most studies find activation of the auditory association regions, but not of primary auditory cortex (Bunzeck et al., 2005, Daselaar et al., 2010, Halpern and Zatorre, 1999, Halpern et al., 2004, Herholz et al., 2012, Kleber et al., 2007, Zvyagintsev et al., 2013).
While these studies showed what regions in the brain are involved in imagery, e.g. the localization, there is still a lot of information lacking about how the brain represents these mental images. This is mainly caused by the fact that predominantly univariate methods have been used to analyze mental imagery data, which do not take into account any information that might be conveyed by multiple voxels simultaneously (O'Toole et al., 2007). Several groundbreaking studies that employed multi-voxel pattern analysis (MVPA) have been able to show that fMRI data contain much more information than previously thought (Formisano et al., 2008, Kamitani and Tong, 2005). Therefore, when investigating specialization in imagery networks, multivariate methods can give additional insight into the fine-grained representational information present in the brain that might otherwise go unnoticed. While the feasibility to predict what type of stimuli experimental participants perceive based on patterns of brain activity has been firmly established now, this is less so for purely mental cognitive states. Nevertheless, in recent years several fMRI studies used multivariate analyses to test the representational similarities between visual perception and visual mental imagery or between visual perception and visual working memory (Albers et al., 2013, Harrison and Tong, 2009, Reddy et al., 2010, Stokes et al., 2011, Vetter et al., 2014, Xing et al., 2013). In most of these studies a classifier was trained on the difference between activity patterns associated with two distinct objects during a perceptual task and tested on the activity patterns during a similar mental imagery or working memory task. The successful classification of visual imagery conditions, when having trained the classifier on visual perception conditions, provides further support for the fact that mental imagery involves primary sensory cortices during certain tasks and moreover, that imagery and perception may rely on similar neural representations (Albers et al., 2013).
Although many parallels can be drawn between imagery and perception, it is unclear whether functional cortical specialization takes place not only in sensory cortices during the acquirement of perceptual expertise, but also in mental imagery networks during mental practice. In the perceptual domain, it has been shown that the brain is strongly shaped through experience, which by continuous training gradually develops into what we may call expertise. Expertise has often been reported to go together with increased neuronal activity in regions specific to the area of expertise and anatomical re-organization (Jancke et al., 2009, Schneider et al., 2005a, Schneider et al., 2005b). For example, professional orchestra members show anatomical and functional specialization in the auditory cortex, specific to their type of instrument (Schneider et al., 2005a, Schneider et al., 2005b). However, while specialization through perception is well established, it is less clear whether specialization can also take place in the mental imagery domain (Buschkuehl et al., 2012). Although expertise in working memory has been theoretically suggested to involve functional reorganization (Guida et al., 2013), few studies have experimentally investigated the effect of expertise level on mental imagery processes (Herholz et al., 2008, Plailly et al., 2012). Therefore, it remains to be understood whether the way in which mental imagery content is represented changes based on experience, and if this is paralleled and supported by fine-grained changes in neural activity patterns in specific regions of the brain.
In this experiment, we studied professional cinematographers and sound designers, whose practical working processes on film image and film sound are to a great extent differentiated and highly modality-specific. Collecting fMRI data from these two expert groups and a control group, allowed us to address the following questions: (1) Does modality-specific specialization take place in the mental imagery domain and if so, what neural representations underlie imagery specialization? (2) Do the neural representations that underlie mental imagery have similarities with perceptual representations during bottom-up processing? (3) Do these neural representations contain expertise-specific information that goes beyond modality information? To tackle these questions, we applied multivariate searchlight mapping (SLM; Kriegeskorte et al., 2006) to whole-brain fMRI data of our three experimental groups. In SLM, several neighboring voxels contained in a moving sphere are considered in a decoding analysis. By centering this sphere on every voxel one obtains a whole-brain map of decoding accuracies, similarly to maps obtained running univariate statistics. We used this technique to predict from brain activity patterns of the participants in what modality (auditory or visual) they were imagining, or what film genre (documentary or feature films) they were imagining. Our results revealed that both cinematographers and sound designers showed distinct multi-voxel patterns that are specific for the modality of their expertise, and are differentiated from control subjects. Furthermore, we showed that part of these modality-specific patterns shared a similar substrate with perception, by applying a pattern classifier trained on perceptual trials to imagery data. Lastly, by decoding the implicit presence of film genre, we showed that cinematographers but not sound designers represented film genre during their mental imagery of film clips.
Section snippets
Participants
Three groups of healthy right-handed volunteers of Finnish nationality were tested: cinematographers (CM), sound designers (SD) and control subjects (CS). Cinematographers (N = 12, mean age = 43 ± 11 years, 12 males) and sound designers (N = 12, mean age = 38 ± 9 years, 8 males) had at least four years of professional experience (of which five cinematographers and five sound designers had over 10 years of experience). The control subjects (N = 12, mean age = 41 ± 12 years, 6 males) did not have any hobbies related
Multi-voxel pattern analyses
The results of the SLM analyses provided clear insight into our three research questions. The first two analyses (“Prediction of imagery modality” section) focused on whether specialization can take place in the mental imagery domain. The third set of analyses (“Prediction across perceptual modalities” section) was aimed at understanding whether differences in modality-specific mental imagery have a perceptual basis. Lastly, we investigated whether filmmakers encoded the implicit presence of
Discussion
Using multivariate analyses we were able to successfully predict from brain activity patterns whether participants were imagining visual or auditory content of film clips. Moreover, the SLM results revealed for the two expert groups distinct voxel patterns that contained information about imagery modality. These patterns were partially modality-specific, as they were localized to parietal and occipito-temporal cortex for cinematographers and to auditory cortex for sound designers. Although the
Acknowledgements
The authors would like to thank Riitta Hari and Yevhen Hlushchuk for their feedback on the design, Maria Palavamäki for the editing of the film and sound clips, as well as Marita Kattelus, Johanna Reinikainen, and Karita Ojala for their help during the experiment. The fMRI data was acquired at the AMI, Aalto Neuroimaging, Aalto University. The research was supported by aivoAALTO research project and Aalto Starting Grant, Aalto University.
References (76)
- et al.
Shared representations for working memory and mental imagery in early visual cortex
Curr. Biol.
(2013) - et al.
Dissecting hemisphere-specific contributions to visual spatial imagery using parametric brain mapping
NeuroImage
(2014) - et al.
Scanning silence: mental imagery of complex sounds
NeuroImage
(2005) - et al.
Neuronal effects following working memory training
Dev. Cogn. Neurosci.
(2012) - et al.
Modality-specific and modality-independent components of the human imagery system
NeuroImage
(2010) - et al.
Integration of “what” and “where” in frontal cortex during visual imagery of scenes
NeuroImage
(2012) - et al.
Tracking the mind's image in the brain I: time-resolved fMRI during visuospatial mental imagery
Neuron
(2002) - et al.
Brain areas underlying visual mental imagery and visual perception: an fMRI study
Brain Res. Cogn. Brain Res.
(2004) - et al.
Behavioral and neural correlates of perceived and imagined musical timbre
Neuropsychologia
(2004) - et al.
Distributed neural systems for the generation of visual images
Neuron
(2000)
Visual imagery of famous faces: effects of memory and attention revealed by fMRI
NeuroImage
Neural correlates of inner speech and auditory verbal hallucinations: a critical review and theoretical integration
Clin. Psychol. Rev.
Overt and imagined singing of an Italian aria
NeuroImage
An efficient algorithm for topologically correct segmentation of the cortical sheet in anatomical mr volumes
NeuroImage
The topography of high-order human object areas
Trends Cogn. Sci.
Sustained and transient neural modulations in prefrontal cortex related to declarative long-term memory, working memory, and attention
Cortex
Neural correlates of topographic mental exploration: the impact of route versus survey perspective learning
NeuroImage
Contribution of striate inputs to the visuospatial functions of parieto-preoccipital cortex in monkeys
Behav. Brain Res.
Bayesian reconstruction of natural images from human brain activity
Neuron
Learning by doing versus learning by thinking: An fMRI study of motor and mental training
Neuropsychologia
Reading the mind's eye: decoding category information during mental imagery
NeuroImage
Tracking the mind's image in the brain II: transcranial magnetic stimulation reveals parietal asymmetry in visuospatial imagery
Neuron
Imagery for shapes activates position-invariant representations in human visual cortex
NeuroImage
Decoding sound and imagery content in early visual cortex
Curr. Biol.
Structure and function of auditory cortex: music and speech
Trends Cogn. Sci.
The brain's conversation with itself: neural substrates of dialogic inner speech
Soc. Cogn. Affect. Neurosci.
Functional brain mapping during free viewing of natural scenes
Hum. Brain Mapp.
Voice-selective areas in human auditory cortex
Nature
Listening to an audio drama activates two processing networks, one for all sounds, another exclusively for speech
PLoS One
Changes in cortical activity during mental rotation. A mapping study using functional MRI
Brain J. Neurol.
The intraparietal sulcus and perceptual organization
J. Cogn. Neurosci.
Neural precursors of delayed insight
J. Cogn. Neurosci.
The neural basis of the central executive system of working memory
Nature
Who” is saying “what”? Brain-based decoding of human voice and speech
Science
Activation of the middle fusiform ‘face area’ increases with expeirtise in recognizing novel objects
Nat. Neurosci.
Analysis of functional image analysis contest (FIAC) data with brainvoyager QX: from single-subject to cortically aligned group general linear model analysis and self-organizing group independent component analysis
Hum. Brain Mapp.
Permutation, Parametric, and Bootstrap Tests of Hypotheses
The effect of long-term working memory through personalization applied to free recall: uncurbing the primacy-effect enthusiasm
Mem. Cogn.
Cited by (13)
Do sparse brain activity patterns underlie human cognition?
2022, NeuroImageVisual processing in expert drivers: What makes expert drivers expert?
2018, Transportation Research Part F: Traffic Psychology and BehaviourCitation Excerpt :From a cognitive perspective, that an applied cognitive task such as driving demonstrates ‘backward transference’ to its underlying visuo-perceptual skills is logically consistent with the idea that the visuo-perceptual skills that are trained in expertise are not task specific, but rather reflect generalised enhancement of the underlying cognitive support networks. Certainly it has been demonstrated elsewhere that expertise in complex cognitive tasks as diverse as imagery in film professionals (De Borst, Valente, Jääskeläinen, & Tikka, 2016), music (Tervaniemi, Janhunen, Kruck, & Putkinen, 2016), sport (Meier, Topka, & Hänggi, 2016) and taste/olfaction (Banks et al., 2016), can result in associated cortical changes that are specific to the cognitive-perceptual characteristics required to perform the task. In a review of the literature on object recognition expertise, Harel (2016) suggests that the development of expertise in a particular applied domain is supported by changes in the underlying cortical substrates that would be reasonably expected to underlie such expertise.
A Free Verbalization Method of Evaluating Sound Design: The Effectiveness of Artificially Intelligent Natural Language Processing Methods and Tools
2023, ACM International Conference Proceeding SeriesMapping Specific Mental Content during Musical Imagery
2021, Cerebral Cortex