When embedded in adjacent distractors, a target becomes more difficult to perceive. The neural mechanism for this ubiquitous visual crowding effect remains unresolved. Stimuli presented on opposite sides of the vertical meridian initially project to different hemispheres, whereas stimuli with the same spatial distance but presented to one side of the vertical meridian project to the same hemisphere. Dissociation between visual spatial distance and cortical distance can also be found in V2 and V3 (quadrant representations of the visual hemifield) along the horizontal meridian. In the current study, we observed a strong crowding effect from spatially adjacent distractors with either Gabor or letter targets presented near the vertical or horizontal meridian. Interestingly, for a target presented near the vertical meridian, a distractor from the same side of the meridian (cortically near) had a significantly stronger crowding effect compared with an equidistant distractor presented on the opposite side (cortically remote). No such meridian modulation was observed across the horizontal meridian. These results constrain the cortical locus of the crowding effect to a stage in which left and right visual spaces are represented discontinuously but the upper and lower visual fields are represented continuously, likely beyond the early retinotopic areas.