Crossmodal interactions of haptic and visual texture information in early sensory cortex
Introduction
Humans need to be able to differentiate surface qualities of objects not only by touch but also visually. This is important for object recognition (e.g. nectarine vs. peach) and for the interaction with objects in our environment (Fikes et al., 1994) (e.g. for goal-directed movement: grasping a slippery piece of soap vs. a splintering piece of wood). Behavioral studies showed that both haptic and visual informations add to texture perception (Lederman and Abbott, 1981) and that a crossmodal transfer of texture information between both sensory modalities occurs (Picard, 2006).
However, it is only in the last decade that the neural basis of texture perception and its multidimensional experience have received increased attention. Several neuroimaging studies focused on texture matching and discrimination (Cant and Goodale, 2007, Cavina-Pratesi et al., 2009, Kaas et al., 2012, Peuskens et al., 2004, Sathian et al., 2011, Stilla and Sathian, 2008) as well as on different dimensions of texture perception within the tactile and visual modality; examples include spatial density (Merabet et al., 2004, Zhang et al., 2005), spatial orientation (Kitada et al., 2006, Zhang et al., 2005) and roughness (Burton et al., 2008, Merabet et al., 2004, Kitada et al., 2005, Roland and Brendan, 1998, Simões-Franklin et al., 2011). Most of the tactile studies stress the importance of the parietal operculum and the posterior insula (Kaas et al., 2012, Kitada et al., 2005, Roland and Brendan, 1998, Simões-Franklin et al., 2011, Stilla and Sathian, 2008) for processing surface textures, while studies focusing on visual texture perception often report regions near the collateral sulcus, the lingual gyrus and areas in early visual cortex (Cant and Goodale, 2007, Cant and Goodale, 2011, Cant and Xu, 2012, Cavina-Pratesi et al., 2010, Peuskens et al., 2004, Stilla and Sathian, 2008, Sathian et al., 2011).
Next to the identification of cortical key players in visual and haptic texture perception a recent approach by Hiramatsu et al. (2011) aimed at investigating how visual material properties are coded in the cortex along the ventral visual pathway. The authors reported that while both early and higher-order visual areas seem to contain information distinguishing material categories (including texture information), the neural representation shifts gradually from an image-based representation in early areas (V1/V2 and V3/V4) to a perceptual representation in areas around the fusiform gyrus and the collateral sulcus. Hence, physical and perceptual measures of visual material qualities seem to be processed in a spatially distributed network in the visual cortex, rather than in a single localized region. A similar distributed network was described by Sathian et al. (2011) for the processing of haptic texture information. In connectivity analyses Sathian and colleagues showed a flow of texture information from task-non-selective regions of the postcentral gyrus to texture-selective areas in the parietal operculum and further to regions of the middle occipital cortex. Despite the pure tactile stimulation in many paradigms, consistent visual cortex activation was reported in several of these studies (Merabet et al., 2007, Simões-Franklin et al., 2011, Stilla and Sathian, 2008). Some findings even indicate the existence of bisensory texture-selective regions in the posterior visual cortex and the lingual gyrus by comparing activations elicited by unimodal shape, location and texture matching in both the visual and haptic modality (Sathian et al., 2011, Stilla and Sathian, 2008).
All of the above-mentioned studies investigated texture perception in separate visual and haptic paradigms. The effect of simultaneous visual and haptic exploration of textures has been mostly neglected so far. Hence, we can only assume an overlap of visual and haptic texture representations in some brain areas, but we cannot infer from these studies whether visual and haptic information interacts in these cortical regions. The imaging study by Sathian et al. (2011) gives a first indication that this might indeed be the case. Behavioral studies also indicate the existence of such crossmodal interaction and matching effects in visuo-haptic tasks. It was shown that people consistently and absolutely match specific tactile vibration rates (simulating manual exploration of a textured surface) to visual spatial frequencies (Guzman-Martinez et al., 2012), indicating some kind of crossmodal association effect in visual and haptic texture perception. Furthermore, Lunghi et al. (2010) even showed that simultaneous tactile stimulation can disambiguate binocular rivalry, a process in which two equally salient but dissimilar monocular stimuli are presented to corresponding retinal locations. Both stimuli compete for perceptual dominance and at any instant only one is perceived consciously while the other image is suppressed. In this study subjects haptically explored a linear grating with a matching orientation to either one of two rival visual stimuli. Exploration of the haptic stimulus prolonged dominance or reduced suppression of the matching visual stimulus, indicating a crossmodal interaction. The authors infer from these results that haptic information can modulate visual processing already at a very early stage, probably in V1. This raises the question whether a change in cortical processing can be expected when matching as compared to non-matching visual–haptic texture information is provided, i.e. representing crossmodal interactions at the cortical level.
In a unimodal tactile fMRI study, Kitada et al. (2005) used a parametric stimulus set, i.e. linear gratings varying in spatial period, and demonstrated that differences in tactile roughness yield graded BOLD responses in the parietal operculum, insula and the lateral prefrontal cortex, but only when subjects actively judge rather than merely attend to roughness. Assuming that crossmodal interactions of texture information are not only presented at the behavioral but also at the cortical level, the question arises whether matching visual–haptic information is sufficient for the observation of graded BOLD responses with varying texture characteristics even without an active judgment task.
The main objective of the present study was to investigate texture perception in a paradigm that combines visual and haptic input in a single condition in order to explore crossmodal interactions at the cortical level. We propose differences in cortical processing of matching and non-matching visual–haptic texture information, representing the influence of one sensory modality on information processing in the other modality as indicated by earlier behavioral studies. Based on the studies mentioned above we would expect these crossmodal effects already in early sensory cortices, e.g. postcentral gyrus and posterior occipital cortex (Dionne et al., 2010, Hiramatsu et al., 2011, Merabet et al., 2007, Sathian et al., 2011, Stilla and Sathian, 2008), but perception-related differences rather in higher-order cortical regions, e.g. the collateral sulcus as well as the parietal operculum and the insula (Cavina-Pratesi et al., 2010, Hiramatsu et al., 2011, Kitada et al., 2005). The unfamiliar dot pattern textures used in this experiment varied only along a single texture dimension, i.e. the average center-to-center dot spacing, ensuring that changes in other surface properties like color and friction do not influence the results. Stimulus presentation was always bimodal, but the sensory information content differed as texture information was varied either in the haptic, visual or in both channels.
We analyzed the data in two different ways. First we were interested in the average difference of the BOLD signal between both unimodal and the bimodal texture variation conditions, disregarding dot pattern differences. As we did not ask subjects to perform a cognitive task with the presented textures, we would expect to find differences, if any, at a rather early sensory processing stage. Second we were interested in relative differences of the BOLD response within each of these three conditions, taking into account the parametric dot pattern variation. Is the BOLD response modulated by texture differences when texture characteristics are varied either unimodally or bimodally? Based on previous behavioral studies we expected perceived roughness by touch to be almost perfectly correlated with the inter-dot spacing (Connor et al., 1990, Dépeault et al., 2009, Eck et al., 2011), while visual spatial density estimates should be negatively correlated with average inter-dot distance. Hence, no difference in the parametric BOLD modulation was expected between the two subjective measures tactile roughness and visual spatial density and the objective texture characteristic inter-dot spacing. However, to account for possible subjective perceptual differences we used individual post-fMRI ratings of haptic roughness and visual spatial density as well as the physical inter-dot spacing of the textures in separate parametric models.
Section snippets
Participants
Seventeen right-handed, healthy volunteers (13 women, 4 men; 27 ± 5.9 years) with normal or corrected-to-normal vision participated in the study. Subjects with calluses or injuries to the hands were excluded from participation. All participants were graduate and undergraduate students at Maastricht University. They were naïve to the hypotheses and received course credit or monetary compensation for their participation in the experiment. In accordance with the Declaration of Helsinki, written
Behavioral results
In general perceived haptic roughness increased with increasing inter-dot distance while the visually perceived spatial density decreased (see Fig. 4). This was expected from earlier work conducted by our group (Eck et al., 2011) and it was confirmed by a significant linear effect of spacing in two repeated measures ANOVAs of spacing on perceived roughness by touch and visual spatial density (F(1, 16) = 174.82, p < 0.001, partial η2 = 0.92 and F(1, 16) = 167.8, p < 0.001, partial η2 = 0.91, respectively).
Differences between unimodal and bimodal dot pattern conditions
Texture perception generated a very similar activation pattern irrespective of the modality in which texture information was varied, with activation overlapping primarily in early and higher-order visual, somatosensory and motor regions. These activations coincided with brain regions that were previously suggested to play a role in haptic and visual texture perception, namely the parietal operculum and insula, regions in the early visual cortex as well as the middle occipital gyrus, the
Conclusion
The present study aimed to investigate whether cortical processing in early sensory cortices changes when matching as compared to non-matching visual–haptic texture information is provided. Earlier research has shown that crossmodal interactions for texture perception exist at the behavioral level. However, neuroimaging studies focused primarily on visual and haptic texture processing separately, allowing to investigate only the overlap of both sensory processing streams. The variation of
Acknowledgments
The research leading to these results has received funding from the European Community's Seventh Framework Programme FP7/2007-2013 under grant agreement number PITN-GA-2008-214728. We also gratefully acknowledge the support of the BrainGain Smart Mix Programme of the Netherlands Ministry of Economic Affairs and the Netherlands Ministry of Education, Culture and Science. We would like to thank Ron Hellenbrand and Joost Mulders for their help with the stimulus creation.
Conflit of interest
I am one
References (62)
- et al.
Tactile stimulation accelerates behavioral responses to visual stimuli through enhancement of occipital gamma-band activity
Vision Res.
(2009) Tactual perception of material properties
Vision Res.
(2010)- et al.
Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex
Curr. Biol.
(2000) - et al.
Effective connectivity during haptic perception: a study using Granger causality analysis of functional magnetic resonance imaging data
Neuroimage
(2008) - et al.
Object familiarity modulates effective connectivity during haptic shape perception
Neuroimage
(2010) - et al.
Interactive coding of visual spatial frequency and auditory amplitude–modulation rate
Curr. Biol.
(2012) - et al.
Transformation from image-based to perceptual representation of materials along the human ventral visual pathway
Neuroimage
(2011) - et al.
Multi-objective optimal experimental designs for event-related fMRI studies
Neuroimage
(2009) - et al.
Tactile estimation of the roughness of gratings yields a graded response in the human brain: an fMRI study
Neuroimage
(2005) - et al.
Object familiarity modulates the relationship between visual object imagery and haptic shape perception
Neuroimage
(2010)
Touch disambiguates rivalrous perception at early stages of visual analysis
Curr. Biol.
Feeling by sight or seeing by touch?
Neuron
A multisensory investigation of the functional significance of the “pain matrix”
Neuroimage
Partial perceptual equivalence between vision and touch for texture information
Acta Psychol. (Amst)
Somatosensory detection of microgeometry, macrogeometry and kinesthesia in man
Brain Res. Rev.
Dual pathways for haptic and visual perception of spatial and texture information
Neuroimage
Visuo-haptic object-related activation in the ventral visual pathway
Nat. Neurosci.
SA1 and RA afferent responses to static and vibrating gratings
J. Neurophysiol.
Projections to early visual areas v1 and v2 in the calcarine fissure from parietal association areas in the macaque
Front. Neuroanat.
Linear systems analysis of functional magnetic resonance imaging in human V1
J. Neurosci.
The relationship between task performance and functional magnetic resonance imaging response
J. Neurosci.
Multiple parietal operculum subdivisions in humans: tactile activation maps
Somatosens. Mot. Res.
Attention to form or surface properties modulates different regions of human occipitotemporal cortex
Cereb. Cortex
Scratching beneath the surface: new insights into the functional properties of the lateral occipital area and parahippocampal place area
J. Neurosci.
Object ensemble processing in human anterior–medial ventral visual cortex
J. Neurosci.
Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey
Eur. J. Neurosci.
Separate processing of texture and form in the ventral stream: evidence from fMRI and visual agnosia
Cereb. Cortex
Separate channels for processing form, texture, and color: evidence from fMRI adaptation and visual object agnosia
Cereb Cortex
Fine-scale organization of SI (area 3b) in the squirrel monkey revealed with intrinsic optical imaging
J. Neurophysiol.
Tactile roughness: neural codes that account for psychophysical magnitude estimates
J. Neurosci.
Tactile perception of roughness: raised-dot spacing, density and disposition
Exp. Brain Res.
Cited by (38)
Functional hierarchy for tactile processing in the visual cortex of sighted adults
2019, NeuroImageCitation Excerpt :Our results align with previous studies that have documented cross-modal activations at the various visual processing hierarchy stages in sighted people. Some of these responses emerge as a result of learning new tactile or auditory skills (Amedi et al., 2007; Saito et al., 2006; Siuda-Krzywicka et al., 2016; Zangenehpour and Zatorre, 2010), while others are observed without any specific training, thus suggesting they are part of a functional repertoire allowed by the default cortical organization (Amedi et al., 2001; Campus et al., 2017; Eck et al., 2016, 2013; Lacey et al., 2014, 2010; Sathian et al., 2011; Stilla and Sathian, 2008; Tal et al., 2016). In the context of our work, it is particularly interesting that the fMRI study of Snow et al. (2014) has already suggested that neural populations responsive to tactually perceived shapes exist both in the early visual cortices and in higher-level ventral visual regions (i.e., V4 and the lateral occipital complex) of sighted subjects.
Brain networks underlying tactile softness perception: A functional magnetic resonance imaging study
2019, NeuroImageCitation Excerpt :Neuroimaging studies focusing on one perceptual dimension of material properties have shown that activity in the parietal operculum and insula is related to the magnitude of perceived roughness (Kitada et al., 2005; Eck et al., 2016) and temperature (Craig et al., 2000). A more recent study demonstrated that the activity in a part of the occipital cortex is related to perceived roughness magnitude (Eck et al., 2013). These findings raise the possibility that compared with the macro-spatial properties, the parietal operculum, the insula, and a part of the occipital cortex are involved in extracting intensity information of material properties.
Processing of affective and emotionally neutral tactile stimuli in the insular cortex
2019, Developmental Cognitive NeuroscienceVisuo-haptic object perception
2019, Multisensory Perception: From Laboratory to Clinic