The early postnatal period is a highly sensitive time period for the developing brain, both in humans and rodents. During this time window, exposure to adverse experiences can lastingly impact cognitive and emotional development. In this review, we briefly discuss human and rodent studies investigating how exposure to adverse early life conditions - mainly related to quality of parental care - affects brain activity, brain structure, cognition and emotional responses later in life. We discuss the evidence that early life adversity hampers later hippocampal and prefrontal cortex functions, while increasing amygdala activity, and the sensitivity to stressors and emotional behavior later in life. Exposure to early life stress may thus on the one hand promote behavioral adaptation to potentially threatening conditions later in life -at the cost of contextual memory formation in less threatening situations- but may on the other hand also increase the sensitivity to develop stress-related and anxiety disorders in vulnerable individuals.