In social situations, we encounter information transferred in firsthand (egocentric) and secondhand (allocentric) communication contexts. However, the mechanism by which an individual distinguishes whether a past interaction occurred in an egocentric versus allocentric situation is poorly understood. This study examined the neural bases for encoding memories of social interactions through experimentally manipulating the communication context. During fMRI data acquisition, participants watched video clips of an actor speaking and gesturing directly toward them (egocentric context) or toward an unseen third person (allocentric context). After scanning, a recognition task gauged participants' ability to recognize the sentences they had just seen and to recall the context in which the sentences had been spoken. We found no differences between the recognition of sentences spoken in egocentric and allocentric contexts. However, when asked about the communication context ("Had the actor directly spoken to you?"), participants tended to believe falsely that the actor had directly spoken to them during allocentric conditions. Greater activity in the hippocampus was related to correct context memory, whereas the ventral ACC was activated for subsequent inaccurate context memory. For the interaction between encoding context and context memory, we observed increased activation for egocentric remembered items in the bilateral and medial frontal cortex, the BG, and the left parietal and temporal lobe. Our data indicate that memories of social interactions are biased to be remembered egocentrically. Self-referential encoding processes reflected in increased frontal activation and decreased hippocampal activation might be the basis of correct item but false context memory of social interactions.