Being with virtual others: Neural correlates of social interaction
Introduction
Despite remarkable efforts having been made in the field of social cognitive neuroscience there still is a substantial debate about the neural processes underlying our ability to understand other minds (Gallagher, 2001). Much research has been conducted on what is called Theory of Mind or mentalizing referring to the capacity of constructing a meta-representation to explain and anticipate the behavior of others. From our standpoint, however, it is unclear to what extent human social understanding of others – particularly in everyday-life situations – relies on actual online involvement in a given situation or on inferential, offline modes of meta-representation. Regardless of whether those processes are to be thought of as theories or simulations as reflected in the debate on theory theory and simulation theory, Frith and Frith (2003) draw a distinction between implicit and explicit mentalizing corresponding to Theory of Mind online and offline. Alternatively a single common mentalizing ability has been suggested to underlie understanding mental states of both self and other (Gopnik & Meltzoff, 1994). Likewise implicit, reflexive mechanisms constituting forms of personal involvement are suggested to be involved in understanding social situations (Lieberman, Gaunt, Gilbert, & Trope, 2002).
Neuroimaging studies have only recently begun to target aspects of online interactions that require personal involvement in social communication (Gallagher, Jack, Roeppstorff, & Frith, 2002; McCabe, Houser, Ryan, Smith, & Trouard, 2001; Rilling et al., 2002; Sanfey, Rilling, Aronson, Nystrom, & Cohen, 2003). Thus far, however, the differential effects of self-involvement on the cerebral representation of mentalizing have not been investigated. This is emphasized by Ochsner (2004) who points out that no within-study comparison of self- and other-related activations (often found in MPFC) has been carried out.
How far are the neural mechanisms of perceiving social interactions altered by being personally engaged in social interaction versus being a passive observer who watches others interact? To address this question in an experimentally feasible way forms of virtual reality are useful instruments. Mediated environments have the potential to elicit a sense of being there, a sense of presence in a virtual reality (Heeter, 1992; Ijsselsteijn & Riva, 2003; Moore, Wiederhold, Wiederhold, & Riva 2002; Reeves & Nass, 1996; Steuer, 1992). Furthermore, it has been demonstrated that virtual characters not only convey social information to human observers, but are also perceived as social agents thus exerting social influence on human interactants (Bailenson, Blascovich, Beall, & Loomis, 2003; Pertaub, Slater, & Barker, 2001). Consequently this method has recently begun to be used in social and environmental psychological research (Blascovich, 2002; de Kort, Ijsselsteijn, Kooijman, & Schuurmans, 2003). Social cognitive neuroscience also benefits considerably by using mediated environments and dynamic stimuli of animated virtual characters (Adolphs, 2003; Pelphrey, Viola, & McCarthy, 2004). However, a crucial prerequisite for the employment of virtual characters in research settings is the assurance that these artificial entities actually do evoke the same experiences and reactions as human stimuli. Here, a considerable number of recent studies have yielded consistent evidence that not only experience (e.g. person perception, see Bente, Krämer, Petersen, & de Ruiter, 2001) but also social reactions are strikingly equivalent to human face-to-face interaction (Garau, Slater, Pertaub, & Razzaque, 2005). Virtual characters have for instance been shown to elicit social facilitation (Hoyt, Blascovich, & Swinth, 2003), proxemic behavior (Bailenson, Blascovich, Beall, & Loomis, 2001) and impression management (Sproull, Subramani, Kiesler, Walker, & Waters, 1996). In sum, they have been shown to evoke a sense of social presence (Biocca, Harms, & Burgoon, 2003)—especially if the appearance of the virtual characters is anthropomorphic (Nowak & Biocca, 2003). As the virtual character's morphology, outward appearance and movements in space and time can be varied systematically, virtual agents and environments have become a powerful tool for experimental psychology (Loomis, Blascovich, & Beall, 1999). Sanchez-Vives and Slater (2005) have recently re-emphasized the applicability of virtual reality as a tool in neuroscience and its value as an object of study in its own right.
We here put to use a much-needed and largely unexplored methodological tool in the realm of social cognitive neuroscience that allows the operationalized investigation of second-person phenomena, i.e. aspects of the dynamic interplay of being personally involved in social interactions (Reddy, 2003). We used this tool in a paradigm which enabled us to investigate the neural processes underlying the cognitive function known as mentalizing dependent upon whether the person who is deciphering someone else's mental state is either involved in social interaction (i.e. online) with that other or not (i.e. offline). For this purpose, we created an experimental situation in which subjects would be socially entrained by virtual others in a controllable mediated scene: In this desktop virtual reality (Tarr & Warren, 2002) virtual characters make an appearance and either turn towards and gaze at the human observer directly (ME; Fig. 1, top row) or look away from the human observer towards a third person not visible to the study participant situated at an angle of approximately 30° (OTHER; Fig. 1, bottom row) thereby systematically varying the study participant's state of self-involvement. Secondly, virtual characters exhibit dynamic facial expressions as they would appear in real-life approach situations to initiate social interaction (Cassell, Bickmore, Campbell, Vilhjalmsson, & Yan, 2001; Grammer, Schiefenhövel, Schleidt, Lorenz, & Eibl-Eibesfeldt, 1988; Kendon & Ferber, 1973). These facial expressions are regarded as socially relevant (SOC) in that they are indicative of someone's intention to establish interpersonal contact and elicit a sense of interactional engagement (Table 1). Alternatively facial movements are shown that are perceived as arbitrary and socially irrelevant (ARB) (Table 1). The resulting two factors (i) SELF-INVOLVEMENT (ME versus OTHER) and (ii) SOCIAL INTERACTION (SOC versus ARB), thus, constitute a two-factorial design (Table 2).
We implemented this design in an event-related fMRI study to investigate the neural substrates of both online and offline mentalizing. We hypothesized that differential neural activations would be associated with one's own personal involvement as compared to situations in which interaction between two others is perceived. We expected that SELF-INVOLVEMENT recruits medial prefrontal cortices whereas perception of the involvement of others in SOCIAL INTERACTION would be subserved by temporo-parietal activations as if taking a third-person-perspective (Vogeley & Fink, 2003). Additionally we assumed that the perception of SOCIAL INTERACTION regardless of self-involvement would also rely upon recruitment of medial prefrontal pathways (Damasio, 1996) while perception of arbitrary facial movements might be accomplished by more basic pathways subserving face perception (Haxby, Hoffman, & Gobbini, 2002).
Section snippets
Subjects
Eighteen right-handed, healthy male volunteers (mean age 25.9 years ± standard deviation 4.2 years) with no record of neurological or psychiatric illness participated in the fMRI study. All volunteers were naïve with respect to the experimental task as well as to the purpose of the study. The study was approved by the local ethics committee of the University of Bonn, Germany.
General information
Using the software package “Poser 4.0” (Curious Lab®) dynamic video animation sequences were designed which depict virtual
Behavioral data during fMRI scanning
Dependent variables were the percent correctness scores of answers to the first question (“WHO?”) and mean response scores to the second question (“CONTACT?”) averaged over all events for each particular condition. One test subject was removed from the analysis as he had consistently failed to answer the questions correctly. The mean correctness score for all remaining subjects for answering the first question was 97.35% (n = 17, S.D. = 1.95, standard error of the mean = 0.46). Mean response score
Discussion
With this study, we suggest to have taken a step towards making second-person phenomena accessible to neuroscientific investigation by using fully controllable anthropomorphic virtual characters which present dynamic facial expressions in order to socially entrain human observers. The results show that different regions of the medial prefrontal cortex are recruited during the perception of social communication and the feeling of personal involvement. Whereas the ventral medial prefrontal cortex
Conclusions
Taken together our results demonstrate that human observers can be socially entrained by virtual characters to whom a communicative intention can be attributed. Behavioral ratings show that socially relevant behavior exhibited by virtual others is judged differently depending upon whether it is self- or other-directed. Ecologically relevant, socially realistic virtual characters can serve as a well-functioning tool for operationalizing second-person phenomena.
The neural activations observed
Acknowledgements
We are grateful for the support of our colleagues in the MRI and Cognitive Neurology group of the Institute of Medicine at the Research Center Juelich and to Dr. Frank Schwab (Universität des Saarlandes, Saarbrücken) for performing FACS coding of our video sequences.
L.S. would like to thank Rüdiger Ilg, Susanne Neufang and Motoaki Sugiura in particular for stimulating discussions and helpful advice.
G.R.F. was supported by the Deutsche Forschungsgemeinschaft (DFG-KFO 112).
The MRI facility is
References (88)
- et al.
Social perception from visual cues: Role of the STS region
Trends in Cognitive Science
(2000) - et al.
More than just a pretty face: Conversational protocols and the affordances of embodiment
Knowledge-Based Systems
(2001) - et al.
Shared representations between self and other: A social cognitive neuroscience view
Trends in Cognitive Science
(2003) - et al.
Functional imaging of “Theory of Mind”
Trends in Cognitive Science
(2003) - et al.
Imaging the intentional stance in a competitive game
NeuroImage
(2002) - et al.
Mirror neurons and the simulation theory of mind-reading
Trends in Cognitive Sciences
(1998) - et al.
Distinct neural substrates for the perception of real and virtual visual worlds
NeuroImage
(2005) - et al.
The distributed human neural system for face perception
Trends in Cognitive Science
(2000) - et al.
Human neural systems for face recognition and social communication
Biological Psychiatry
(2002) - et al.
Face and voice expression identication in patients with emotional and behavioral changes following ventral frontal lobe damage
Neuropsychologia
(1996)
A region of mesial prefrontal cortex tracks monetarily rewarding outcomes: Characterization with rapid event-related fMRI
NeuroImage
Reflection and reflexion: A social cognitive neuroscience apparoch to attributional inference
Advances in Experimental Social Psychology
Activity in ventromedial prefrontal cortex varies with sympathetic skin conductance levels: Physiological account of a “default mode” of brain function
NeuroImage
Current directions in social cognitive neuroscience
Current Opinion in Neurobiology
On being the object of attention: Implications for self-other consciousness
Trends in Cognitive Sciences
A neural basis for social cooperation
Neuron
Understanding orbitofrontal contributions to theory-of-mind reasoning: Implications for autism
Brain and Cognition
Making sense of another mind: The role of the right temporo-parietal junction
Neuropsychologia
Cortical activations during judgments about the self and an other person
Neuropsychologia
Brain responses to the acquired moral status of faces
Neuron
Neural correlates of first-person-perspective
Trends in Cognitive Science
The prefrontal cortex generates the basic constituents of the self
Consciousness and Cognition
Mind reading: Neural mechanisms of theory of mind and self-perspective
NeuroImage
Central representation of the sympathetic nervous system in the cerebral cortex
Brain Research
Are emotions contagious? Evoked emotions while viewing emotionally expressive faces: Quality, quantity, time course and gender differences
Psychiatry Research
An fMRI study of joint attention experience
NeuroImage
Cognitive neuroscience of human social behavior
Nature Reviews Neuroscience
Equilibrium revisited: Mutual gaze and personal space in virtual environments
Presence: Teleoperators and Virtual Environments
Interpersonal distance in immersive virtual environments
Personality and Social Psychology Bulletin
fMRI responses to video and point-light displays of moving humans and manipulable objects
Journal of Cognitive Neuroscience
Characterization of the decision-making effect of patients with ventromedial prefrontal cortex lesions
Brain
Computer animated movement and person perception: Methodological advances in nonverbal behavior research
Journal of Nonverbal Behavior
Toward a more robust theory and measure of social presence
Presence: Teleoperators and Virtual Environments
Social influence within immersive virtual environments
Evidence for a dorso-medial parietal system involved in mental transformation of the body
Journal of Neurophysiology
Neural activity relating to generation and representation of galvanic skin conductance responses: An fMRI study
Journal of Neuroscience
The somatic marker hypothesis and the possible functions of the prefrontal cortex
Philosophical Transactions of the Royal Society London
The feeling of what happens: Body and emotion in the making of consciousness
Neurocognitive function and joint attention ability in young children with autism spectrum disorder versus developmental delay
Child Development
Virtual laboratories: Comparability of real and virtual environments for environmental psychology
Presence
Facial action coding system: A technique for the measurement of facial movement
In search of the emotional self: An fMRI study using positive and negative emotional words
American Journal of Psychiatry
Statistical parametric maps in functional imaging: A general linear approach
Human Brain Mapping
Development and neurophysiology of mentalizing.
Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences
Cited by (354)
Aberrant Effective Connectivity During Eye Gaze Processing Is Linked to Social Functioning and Symptoms in Schizophrenia
2023, Biological Psychiatry: Cognitive Neuroscience and NeuroimagingThe (un)learning of social functions and its significance for mental health
2022, Clinical Psychology Review