Fragile X Syndrome (FXS), a neurodevelopmental disorder, is the most prevalent single-gene cause of autism spectrum disorder. Autism has been associated with impaired auditory processing, abnormalities in the auditory brainstem response (ABR), and reduced cell number and size in the auditory brainstem nuclei. FXS is characterized by elevated cortical responses to sound stimuli, with some evidence for aberrant ABRs. Here, we assessed ABRs and auditory brainstem anatomy in Fmr1-/- mice, an animal model of FXS. We found that Fmr1-/- mice showed elevated response thresholds to both click and tone stimuli. Amplitudes of ABR responses were reduced in Fmr1-/- mice for early peaks of the ABR. The growth of the peak I response with sound intensity was less steep in mutants that in wild type mice. In contrast, amplitudes and response growth in peaks IV and V did not differ between these groups. We did not observe differences in peak latencies or in interpeak latencies. Cell size was reduced in Fmr1-/- mice in the ventral cochlear nucleus (VCN) and in the medial nucleus of the trapezoid body (MNTB). We quantified levels of inhibitory and excitatory synaptic inputs in these nuclei using markers for presynaptic proteins. We measured VGAT and VGLUT immunolabeling in VCN, MNTB, and the lateral superior olive (LSO). VGAT expression in MNTB was significantly greater in the Fmr1-/- mouse than in wild type mice. Together, these observations demonstrate that FXS affects peripheral and central aspects of hearing and alters the balance of excitation and inhibition in the auditory brainstem.