This paper presents a statistical data-driven method for learning intrinsic structures of impact sounds. The method applies principal and independent component analysis to learn low-dimensional representations that model the distribution of both the time-varying spectral and amplitude structure. As a result, the method is able to decompose sounds into a small number of underlying features that characterize acoustic properties such as ringing, resonance, sustain, decay, and onsets. The method is highly flexible and makes no a priori assumptions about the physics, acoustics, or dynamics of the objects. In addition, by modeling the underlying distribution, the method can capture the natural variability of ensembles of related impact sounds.