The 'eye contact effect' is the phenomenon that perceived eye contact with another human face modulates certain aspects of the concurrent and/or immediately following cognitive processing. In addition, functional imaging studies in adults have revealed that eye contact can modulate activity in structures in the social brain network, and developmental studies show evidence for preferential orienting towards, and processing of, faces with direct gaze from early in life. We review different theories of the eye contact effect and advance a 'fast-track modulator' model. Specifically, we hypothesize that perceived eye contact is initially detected by a subcortical route, which then modulates the activation of the social brain as it processes the accompanying detailed sensory information.