We used functional magnetic resonance imaging to investigate the effect of two factors on the neural control of temporal sequence performance: the modality in which the rhythms had been trained, and the modality of the pacing stimuli preceding performance. The rhythms were trained 1-2 days before scanning. Each participant learned two rhythms: one was presented visually, the other auditorily. During fMRI, the rhythms were performed in blocks. In each block, four beats of a visual or auditory pacing metronome were followed by repetitive self-paced rhythm performance from memory. Data from the self-paced performance phase was analysed in a 2x2 factorial design, with the two factors Training Modality (auditory or visual) and Metronome Modality (auditory or visual), as well as with a conjunction analysis across all active conditions, to identify activations that were independent of both Training Modality and Metronome Modality. We found a significant main effect only for visual versus auditory Metronome Modality, in the left angular gyrus, due to a deactivation of this region after auditory pacing. The conjunction analysis revealed a set of brain areas that included dorsal auditory pathway areas (left temporo-parietal junction area and ventral premotor cortex), dorsal premotor cortex, the supplementary and presupplementary premotor areas, the cerebellum and the basal ganglia. We conclude that these regions are involved in controlling performance of well-learned rhythms, regardless of the modality in which the rhythms are trained and paced. This suggests that after extensive short-term training, all rhythms, even those that were both trained and paced in visual modality, had been transformed into auditory-motor representations. The deactivation of the angular cortex following auditory pacing may represent cross-modal auditory-visual inhibition.