A number of factors could explain the adverse effect that babble noise has on memory for spoken words (Murphy, Craik, Li, & Schneider, 2000). Babble could degrade the perceptual representation of words to such an extent that it compromises their subsequent processing, or the presence of speech noise in the period between word presentations could interfere with rehearsal. Thirdly, the top-down processes needed to extract the words from the babble could draw on resources that otherwise would be used for encoding. We tested all these hypotheses by presenting babble either only during word presentation or rehearsal, or by gating the babble on and off 500 ms before and after each word pair. Only the last condition led to a decline in memory. We propose that this decline in memory occurred because participants were focusing their attention on the auditory stream (to enable them to better segregate the words from the noise background) rather than on remembering the words they had heard. To further support our claim we show that a similar memory deficit results when participants perform the same memory task in quiet together with a nonauditory attention-demanding secondary task.