Elsevier

NeuroImage

Volume 129, 1 April 2016, Pages 428-438
NeuroImage

Brain-based decoding of mentally imagined film clips and sounds reveals experience-based information patterns in film professionals

https://doi.org/10.1016/j.neuroimage.2016.01.043Get rights and content

Highlights

  • We decoded imagined film clips and sounds from brain activity of film professionals.

  • In cinematographers occipito-temporal and parietal regions predicted imagery modality.

  • In sound designers auditory regions predicted imagery modality.

  • Perception and imagery shared representations in frontal and temporal cortex.

  • Imagined film genre was predicted from frontal brain activity of cinematographers.

Abstract

In the perceptual domain, it has been shown that the human brain is strongly shaped through experience, leading to expertise in highly-skilled professionals. What has remained unclear is whether specialization also shapes brain networks underlying mental imagery. In our fMRI study, we aimed to uncover modality-specific mental imagery specialization of film experts. Using multi-voxel pattern analysis we decoded from brain activity of professional cinematographers and sound designers whether they were imagining sounds or images of particular film clips. In each expert group distinct multi-voxel patterns, specific for the modality of their expertise, were found during classification of imagery modality. These patterns were mainly localized in the occipito-temporal and parietal cortex for cinematographers and in the auditory cortex for sound designers. We also found generalized patterns across perception and imagery that were distinct for the two expert groups: they involved frontal cortex for the cinematographers and temporal cortex for the sound designers. Notably, the mental representations of film clips and sounds of cinematographers contained information that went beyond modality-specificity. We were able to successfully decode the implicit presence of film genre from brain activity during mental imagery in cinematographers. The results extend existing neuroimaging literature on expertise into the domain of mental imagery and show that experience in visual versus auditory imagery can alter the representation of information in modality-specific association cortices.

Introduction

With awe one can watch breath-taking thrillers, or laugh out loud at a silly comedy film. While an increasing body of functional MRI (fMRI) studies have looked at neural responses of film viewers (Bartels and Zeki, 2004, Hasson et al., 2004, Jaaskelainen et al., 2008, Lahnakoski et al., 2012, Naselaris et al., 2009), no studies have focused on the neural basis of how filmmakers create these images and sounds in their minds. The process of constructing and manipulating mental representations in the human brain is called mental imagery. Professional filmmakers, such as cinematographers and sound designers, can be argued to stand as experts in modality-specific mental imagery. On a daily basis they need to imagine and mentally manipulate either visual (cinematographers) or auditory (sound designers) information in order to reach technical and artistic decisions. In this study we report neuroimaging findings suggesting that cinematographers and sound designers use differentiated brain regions to construct visual and auditory mental representations.

Neuroimaging studies on mental imagery have shown that when we imagine something that we cannot directly see or hear, the brain utilizes similar brain regions as during related sensory perception. The imagery of objects, such as houses or faces, activates category-specific regions in ventral temporal cortex (Ishai et al., 2000, Ishai et al., 2002, O'Craven and Kanwisher, 2000). Tasks involving spatial manipulation of imagined content revealed activity in the dorsal stream, in particular in the posterior parietal cortex (Bien and Sack, 2014, Cohen et al., 1996, Formisano et al., 2002, Mellet et al., 1996, Sasaoka et al., 2014, Tagaris et al., 1996, Trojano et al., 2000). In addition, a set of regions in the frontal cortex, including medial superior frontal gyrus, premotor cortex and middle frontal cortex, have shown to control construction and maintenance of the mental image (D'Esposito et al., 1995, de Borst et al., 2012, Marklund et al., 2007). Whether the similarities between imagery and perception include the primary sensory cortices has been a matter of discussion. While for visual imagery some studies reported early visual cortex activation (Ganis et al., 2004, Ishai et al., 2002, Klein et al., 2000, Kosslyn et al., 1993, Kosslyn et al., 1999), others did not (Formisano et al., 2002, Ishai et al., 2000, Knauff et al., 2000, Mellet et al., 1996, Mellet et al., 2000b, Trojano et al., 2000). The involvement of early visual cortex seems to especially rely on the mental representation of fine-grained details rather than the spatial properties of objects or scenes (Kosslyn and Thompson, 2003, Mellet et al., 2000a). For auditory imagery, most studies find activation of the auditory association regions, but not of primary auditory cortex (Bunzeck et al., 2005, Daselaar et al., 2010, Halpern and Zatorre, 1999, Halpern et al., 2004, Herholz et al., 2012, Kleber et al., 2007, Zvyagintsev et al., 2013).

While these studies showed what regions in the brain are involved in imagery, e.g. the localization, there is still a lot of information lacking about how the brain represents these mental images. This is mainly caused by the fact that predominantly univariate methods have been used to analyze mental imagery data, which do not take into account any information that might be conveyed by multiple voxels simultaneously (O'Toole et al., 2007). Several groundbreaking studies that employed multi-voxel pattern analysis (MVPA) have been able to show that fMRI data contain much more information than previously thought (Formisano et al., 2008, Kamitani and Tong, 2005). Therefore, when investigating specialization in imagery networks, multivariate methods can give additional insight into the fine-grained representational information present in the brain that might otherwise go unnoticed. While the feasibility to predict what type of stimuli experimental participants perceive based on patterns of brain activity has been firmly established now, this is less so for purely mental cognitive states. Nevertheless, in recent years several fMRI studies used multivariate analyses to test the representational similarities between visual perception and visual mental imagery or between visual perception and visual working memory (Albers et al., 2013, Harrison and Tong, 2009, Reddy et al., 2010, Stokes et al., 2011, Vetter et al., 2014, Xing et al., 2013). In most of these studies a classifier was trained on the difference between activity patterns associated with two distinct objects during a perceptual task and tested on the activity patterns during a similar mental imagery or working memory task. The successful classification of visual imagery conditions, when having trained the classifier on visual perception conditions, provides further support for the fact that mental imagery involves primary sensory cortices during certain tasks and moreover, that imagery and perception may rely on similar neural representations (Albers et al., 2013).

Although many parallels can be drawn between imagery and perception, it is unclear whether functional cortical specialization takes place not only in sensory cortices during the acquirement of perceptual expertise, but also in mental imagery networks during mental practice. In the perceptual domain, it has been shown that the brain is strongly shaped through experience, which by continuous training gradually develops into what we may call expertise. Expertise has often been reported to go together with increased neuronal activity in regions specific to the area of expertise and anatomical re-organization (Jancke et al., 2009, Schneider et al., 2005a, Schneider et al., 2005b). For example, professional orchestra members show anatomical and functional specialization in the auditory cortex, specific to their type of instrument (Schneider et al., 2005a, Schneider et al., 2005b). However, while specialization through perception is well established, it is less clear whether specialization can also take place in the mental imagery domain (Buschkuehl et al., 2012). Although expertise in working memory has been theoretically suggested to involve functional reorganization (Guida et al., 2013), few studies have experimentally investigated the effect of expertise level on mental imagery processes (Herholz et al., 2008, Plailly et al., 2012). Therefore, it remains to be understood whether the way in which mental imagery content is represented changes based on experience, and if this is paralleled and supported by fine-grained changes in neural activity patterns in specific regions of the brain.

In this experiment, we studied professional cinematographers and sound designers, whose practical working processes on film image and film sound are to a great extent differentiated and highly modality-specific. Collecting fMRI data from these two expert groups and a control group, allowed us to address the following questions: (1) Does modality-specific specialization take place in the mental imagery domain and if so, what neural representations underlie imagery specialization? (2) Do the neural representations that underlie mental imagery have similarities with perceptual representations during bottom-up processing? (3) Do these neural representations contain expertise-specific information that goes beyond modality information? To tackle these questions, we applied multivariate searchlight mapping (SLM; Kriegeskorte et al., 2006) to whole-brain fMRI data of our three experimental groups. In SLM, several neighboring voxels contained in a moving sphere are considered in a decoding analysis. By centering this sphere on every voxel one obtains a whole-brain map of decoding accuracies, similarly to maps obtained running univariate statistics. We used this technique to predict from brain activity patterns of the participants in what modality (auditory or visual) they were imagining, or what film genre (documentary or feature films) they were imagining. Our results revealed that both cinematographers and sound designers showed distinct multi-voxel patterns that are specific for the modality of their expertise, and are differentiated from control subjects. Furthermore, we showed that part of these modality-specific patterns shared a similar substrate with perception, by applying a pattern classifier trained on perceptual trials to imagery data. Lastly, by decoding the implicit presence of film genre, we showed that cinematographers but not sound designers represented film genre during their mental imagery of film clips.

Section snippets

Participants

Three groups of healthy right-handed volunteers of Finnish nationality were tested: cinematographers (CM), sound designers (SD) and control subjects (CS). Cinematographers (N = 12, mean age = 43 ± 11 years, 12 males) and sound designers (N = 12, mean age = 38 ± 9 years, 8 males) had at least four years of professional experience (of which five cinematographers and five sound designers had over 10 years of experience). The control subjects (N = 12, mean age = 41 ± 12 years, 6 males) did not have any hobbies related

Multi-voxel pattern analyses

The results of the SLM analyses provided clear insight into our three research questions. The first two analyses (“Prediction of imagery modality” section) focused on whether specialization can take place in the mental imagery domain. The third set of analyses (“Prediction across perceptual modalities” section) was aimed at understanding whether differences in modality-specific mental imagery have a perceptual basis. Lastly, we investigated whether filmmakers encoded the implicit presence of

Discussion

Using multivariate analyses we were able to successfully predict from brain activity patterns whether participants were imagining visual or auditory content of film clips. Moreover, the SLM results revealed for the two expert groups distinct voxel patterns that contained information about imagery modality. These patterns were partially modality-specific, as they were localized to parietal and occipito-temporal cortex for cinematographers and to auditory cortex for sound designers. Although the

Acknowledgements

The authors would like to thank Riitta Hari and Yevhen Hlushchuk for their feedback on the design, Maria Palavamäki for the editing of the film and sound clips, as well as Marita Kattelus, Johanna Reinikainen, and Karita Ojala for their help during the experiment. The fMRI data was acquired at the AMI, Aalto Neuroimaging, Aalto University. The research was supported by aivoAALTO research project and Aalto Starting Grant, Aalto University.

References (76)

  • A. Ishai et al.

    Visual imagery of famous faces: effects of memory and attention revealed by fMRI

    NeuroImage

    (2002)
  • S.R. Jones et al.

    Neural correlates of inner speech and auditory verbal hallucinations: a critical review and theoretical integration

    Clin. Psychol. Rev.

    (2007)
  • B. Kleber et al.

    Overt and imagined singing of an Italian aria

    NeuroImage

    (2007)
  • N. Kriegeskorte et al.

    An efficient algorithm for topologically correct segmentation of the cortical sheet in anatomical mr volumes

    NeuroImage

    (2001)
  • R. Malach et al.

    The topography of high-order human object areas

    Trends Cogn. Sci.

    (2002)
  • P. Marklund et al.

    Sustained and transient neural modulations in prefrontal cortex related to declarative long-term memory, working memory, and attention

    Cortex

    (2007)
  • E. Mellet et al.

    Neural correlates of topographic mental exploration: the impact of route versus survey perspective learning

    NeuroImage

    (2000)
  • M. Mishkin et al.

    Contribution of striate inputs to the visuospatial functions of parieto-preoccipital cortex in monkeys

    Behav. Brain Res.

    (1982)
  • T. Naselaris et al.

    Bayesian reconstruction of natural images from human brain activity

    Neuron

    (2009)
  • L. Nyberg et al.

    Learning by doing versus learning by thinking: An fMRI study of motor and mental training

    Neuropsychologia

    (2006)
  • L. Reddy et al.

    Reading the mind's eye: decoding category information during mental imagery

    NeuroImage

    (2010)
  • A.T. Sack et al.

    Tracking the mind's image in the brain II: transcranial magnetic stimulation reveals parietal asymmetry in visuospatial imagery

    Neuron

    (2002)
  • M. Stokes et al.

    Imagery for shapes activates position-invariant representations in human visual cortex

    NeuroImage

    (2011)
  • P. Vetter et al.

    Decoding sound and imagery content in early visual cortex

    Curr. Biol.

    (2014)
  • R.J. Zatorre et al.

    Structure and function of auditory cortex: music and speech

    Trends Cogn. Sci.

    (2002)
  • B. Alderson-Day et al.

    The brain's conversation with itself: neural substrates of dialogic inner speech

    Soc. Cogn. Affect. Neurosci.

    (2015)
  • A. Bartels et al.

    Functional brain mapping during free viewing of natural scenes

    Hum. Brain Mapp.

    (2004)
  • P. Belin et al.

    Voice-selective areas in human auditory cortex

    Nature

    (2000)
  • R. Boldt et al.

    Listening to an audio drama activates two processing networks, one for all sounds, another exclusively for speech

    PLoS One

    (2013)
  • M.S. Cohen et al.

    Changes in cortical activity during mental rotation. A mapping study using functional MRI

    Brain J. Neurol.

    (1996)
  • R. Cusack

    The intraparietal sulcus and perceptual organization

    J. Cogn. Neurosci.

    (2005)
  • A. Darsaud et al.

    Neural precursors of delayed insight

    J. Cogn. Neurosci.

    (2011)
  • M. D'Esposito et al.

    The neural basis of the central executive system of working memory

    Nature

    (1995)
  • E. Formisano et al.

    Who” is saying “what”? Brain-based decoding of human voice and speech

    Science

    (2008)
  • I. Gauthier et al.

    Activation of the middle fusiform ‘face area’ increases with expeirtise in recognizing novel objects

    Nat. Neurosci.

    (1999)
  • R. Goebel et al.

    Analysis of functional image analysis contest (FIAC) data with brainvoyager QX: from single-subject to cortically aligned group general linear model analysis and self-organizing group independent component analysis

    Hum. Brain Mapp.

    (2006)
  • P.I. Good

    Permutation, Parametric, and Bootstrap Tests of Hypotheses

    (2005)
  • A. Guida et al.

    The effect of long-term working memory through personalization applied to free recall: uncurbing the primacy-effect enthusiasm

    Mem. Cogn.

    (2013)
  • Cited by (13)

    • Visual processing in expert drivers: What makes expert drivers expert?

      2018, Transportation Research Part F: Traffic Psychology and Behaviour
      Citation Excerpt :

      From a cognitive perspective, that an applied cognitive task such as driving demonstrates ‘backward transference’ to its underlying visuo-perceptual skills is logically consistent with the idea that the visuo-perceptual skills that are trained in expertise are not task specific, but rather reflect generalised enhancement of the underlying cognitive support networks. Certainly it has been demonstrated elsewhere that expertise in complex cognitive tasks as diverse as imagery in film professionals (De Borst, Valente, Jääskeläinen, & Tikka, 2016), music (Tervaniemi, Janhunen, Kruck, & Putkinen, 2016), sport (Meier, Topka, & Hänggi, 2016) and taste/olfaction (Banks et al., 2016), can result in associated cortical changes that are specific to the cognitive-perceptual characteristics required to perform the task. In a review of the literature on object recognition expertise, Harel (2016) suggests that the development of expertise in a particular applied domain is supported by changes in the underlying cortical substrates that would be reasonably expected to underlie such expertise.

    View all citing articles on Scopus
    View full text