In the fifty years since the organizational hypothesis was proposed, many sex differences have been found in behavior as well as structure of the brain that depend on the organizational effects of gonadal hormones early in development. Remarkably, in most cases we do not understand how the two are related. This paper makes the case that overstating the magnitude or constancy of sex differences in behavior and too narrowly interpreting the functional consequences of structural differences are significant roadblocks in resolving this issue.