Neuronal responses during sensory processing are influenced by both the organization of intracortical connections and the statistical features of sensory stimuli. How these intrinsic and extrinsic factors govern the activity of excitatory and inhibitory populations is unclear. Using two-photon calcium imaging in vivo and intracellular recordings in vitro, we investigated the dependencies between synaptic connectivity, feature selectivity and network activity in pyramidal cells and fast-spiking parvalbumin-expressing (PV) interneurons in mouse visual cortex. In pyramidal cell populations, patterns of neuronal correlations were largely stimulus-dependent, indicating that their responses were not strongly dominated by functionally biased recurrent connectivity. By contrast, visual stimulation only weakly modified co-activation patterns of fast-spiking PV cells, consistent with the observation that these broadly tuned interneurons received very dense and strong synaptic input from nearby pyramidal cells with diverse feature selectivities. Therefore, feedforward and recurrent network influences determine the activity of excitatory and inhibitory ensembles in fundamentally different ways.