Using functional magnetic resonance imaging (fMRI), we investigated brain activity evoked by mutual and averted gaze in a compelling and commonly experienced social encounter. Through virtual-reality goggles, subjects viewed a man who walked toward them and shifted his neutral gaze either toward (mutual gaze) or away (averted gaze) from them. Robust activity was evoked in the superior temporal sulcus (STS) and fusiform gyrus (FFG). For both conditions, STS activity was strongly right lateralized. Mutual gaze evoked greater activity in the STS than did averted gaze, whereas the FFG responded equivalently to mutual and averted gaze. Thus, we show that the STS is involved in processing social information conveyed by shifts in gaze within an overtly social context. This study extends understanding of the role of the STS in social cognition and social perception by demonstrating that it is highly sensitive to the context in which a human action occurs.