When the same visual input has conflicting interpretations, conscious perception can alternate spontaneously between each competing percept. Surprisingly, such bistable perception can be stabilized by intermittent stimulus removal, suggesting the existence of perceptual "memory" across interruptions in stimulation. The neural basis of such a process remains unknown. Here, we studied binocular rivalry, one type of bistable perception, in two linked experiments in human participants. First, we showed, in a behavioral experiment using binocular rivalry between face and grating stimuli, that the stabilizing effect of stimulus removal was specific to perceptual alternations evoked by rivalry, and did not occur following physical alternations in the absence of rivalry. We then used functional magnetic resonance imaging to measure brain activity in a variable delay period of stimulus removal. Activity in the fusiform face area during the delay period following removal of rivalrous stimuli was greater following face than grating perception, whereas such a difference was absent during removal of non-rivalrous stimuli. Moreover, activity in areas of fronto-parietal regions during the delay period correlated with the degree to which individual participants tended to experience percept stabilization. Our findings suggest that percept-related activity in specialized extrastriate visual areas help to stabilize perception during perceptual conflict, and that high-level mechanisms may determine the influence of such signals on conscious perception.