Elsevier

NeuroImage

Volume 75, 15 July 2013, Pages 123-135
NeuroImage

Crossmodal interactions of haptic and visual texture information in early sensory cortex

https://doi.org/10.1016/j.neuroimage.2013.02.075Get rights and content

Highlights

  • Neural correlates of matching versus non-matching visual–haptic texture exploration

  • Parametric variation of texture characteristics (inter-dot spacing)

  • Matching visual–haptic information increased BOLD response in early visual cortex.

  • Parametric variation of matching visual–haptic textures induced BOLD modulation.

  • These crossmodal interactions occurred primarily in early sensory cortices.

Abstract

Both visual and haptic information add to the perception of surface texture. While prior studies have reported crossmodal interactions of both sensory modalities at the behavioral level, neuroimaging studies primarily investigated texture perception in separate visual and haptic paradigms. These experimental designs, however, only allowed to identify overlap in both sensory processing streams but no interaction of visual and haptic texture processing. By varying texture characteristics in a bimodal task, the current study investigated how these crossmodal interactions are reflected at the cortical level. We used fMRI to compare cortical activation in response to matching versus non-matching visual–haptic texture information. We expected that passive simultaneous presentation of matching visual–haptic input would be sufficient to induce BOLD responses graded with varying texture characteristics. Since no cognitive evaluation of the stimuli was required, we expected to find changes primarily at a rather early processing stage. Our results confirmed our assumptions by showing crossmodal interactions of visual–haptic texture information in early somatosensory and visual cortex. However, the nature of the crossmodal effects was slightly different in both sensory cortices. In early visual cortex, matching visual–haptic information increased the average activation level and induced parametric BOLD signal variations with varying texture characteristics. In early somatosensory cortex only the latter was true. These results challenge the notion that visual and haptic texture information is processed independently and indicate a crossmodal interaction of sensory information already at an early cortical processing stage.

Introduction

Humans need to be able to differentiate surface qualities of objects not only by touch but also visually. This is important for object recognition (e.g. nectarine vs. peach) and for the interaction with objects in our environment (Fikes et al., 1994) (e.g. for goal-directed movement: grasping a slippery piece of soap vs. a splintering piece of wood). Behavioral studies showed that both haptic and visual informations add to texture perception (Lederman and Abbott, 1981) and that a crossmodal transfer of texture information between both sensory modalities occurs (Picard, 2006).

However, it is only in the last decade that the neural basis of texture perception and its multidimensional experience have received increased attention. Several neuroimaging studies focused on texture matching and discrimination (Cant and Goodale, 2007, Cavina-Pratesi et al., 2009, Kaas et al., 2012, Peuskens et al., 2004, Sathian et al., 2011, Stilla and Sathian, 2008) as well as on different dimensions of texture perception within the tactile and visual modality; examples include spatial density (Merabet et al., 2004, Zhang et al., 2005), spatial orientation (Kitada et al., 2006, Zhang et al., 2005) and roughness (Burton et al., 2008, Merabet et al., 2004, Kitada et al., 2005, Roland and Brendan, 1998, Simões-Franklin et al., 2011). Most of the tactile studies stress the importance of the parietal operculum and the posterior insula (Kaas et al., 2012, Kitada et al., 2005, Roland and Brendan, 1998, Simões-Franklin et al., 2011, Stilla and Sathian, 2008) for processing surface textures, while studies focusing on visual texture perception often report regions near the collateral sulcus, the lingual gyrus and areas in early visual cortex (Cant and Goodale, 2007, Cant and Goodale, 2011, Cant and Xu, 2012, Cavina-Pratesi et al., 2010, Peuskens et al., 2004, Stilla and Sathian, 2008, Sathian et al., 2011).

Next to the identification of cortical key players in visual and haptic texture perception a recent approach by Hiramatsu et al. (2011) aimed at investigating how visual material properties are coded in the cortex along the ventral visual pathway. The authors reported that while both early and higher-order visual areas seem to contain information distinguishing material categories (including texture information), the neural representation shifts gradually from an image-based representation in early areas (V1/V2 and V3/V4) to a perceptual representation in areas around the fusiform gyrus and the collateral sulcus. Hence, physical and perceptual measures of visual material qualities seem to be processed in a spatially distributed network in the visual cortex, rather than in a single localized region. A similar distributed network was described by Sathian et al. (2011) for the processing of haptic texture information. In connectivity analyses Sathian and colleagues showed a flow of texture information from task-non-selective regions of the postcentral gyrus to texture-selective areas in the parietal operculum and further to regions of the middle occipital cortex. Despite the pure tactile stimulation in many paradigms, consistent visual cortex activation was reported in several of these studies (Merabet et al., 2007, Simões-Franklin et al., 2011, Stilla and Sathian, 2008). Some findings even indicate the existence of bisensory texture-selective regions in the posterior visual cortex and the lingual gyrus by comparing activations elicited by unimodal shape, location and texture matching in both the visual and haptic modality (Sathian et al., 2011, Stilla and Sathian, 2008).

All of the above-mentioned studies investigated texture perception in separate visual and haptic paradigms. The effect of simultaneous visual and haptic exploration of textures has been mostly neglected so far. Hence, we can only assume an overlap of visual and haptic texture representations in some brain areas, but we cannot infer from these studies whether visual and haptic information interacts in these cortical regions. The imaging study by Sathian et al. (2011) gives a first indication that this might indeed be the case. Behavioral studies also indicate the existence of such crossmodal interaction and matching effects in visuo-haptic tasks. It was shown that people consistently and absolutely match specific tactile vibration rates (simulating manual exploration of a textured surface) to visual spatial frequencies (Guzman-Martinez et al., 2012), indicating some kind of crossmodal association effect in visual and haptic texture perception. Furthermore, Lunghi et al. (2010) even showed that simultaneous tactile stimulation can disambiguate binocular rivalry, a process in which two equally salient but dissimilar monocular stimuli are presented to corresponding retinal locations. Both stimuli compete for perceptual dominance and at any instant only one is perceived consciously while the other image is suppressed. In this study subjects haptically explored a linear grating with a matching orientation to either one of two rival visual stimuli. Exploration of the haptic stimulus prolonged dominance or reduced suppression of the matching visual stimulus, indicating a crossmodal interaction. The authors infer from these results that haptic information can modulate visual processing already at a very early stage, probably in V1. This raises the question whether a change in cortical processing can be expected when matching as compared to non-matching visual–haptic texture information is provided, i.e. representing crossmodal interactions at the cortical level.

In a unimodal tactile fMRI study, Kitada et al. (2005) used a parametric stimulus set, i.e. linear gratings varying in spatial period, and demonstrated that differences in tactile roughness yield graded BOLD responses in the parietal operculum, insula and the lateral prefrontal cortex, but only when subjects actively judge rather than merely attend to roughness. Assuming that crossmodal interactions of texture information are not only presented at the behavioral but also at the cortical level, the question arises whether matching visual–haptic information is sufficient for the observation of graded BOLD responses with varying texture characteristics even without an active judgment task.

The main objective of the present study was to investigate texture perception in a paradigm that combines visual and haptic input in a single condition in order to explore crossmodal interactions at the cortical level. We propose differences in cortical processing of matching and non-matching visual–haptic texture information, representing the influence of one sensory modality on information processing in the other modality as indicated by earlier behavioral studies. Based on the studies mentioned above we would expect these crossmodal effects already in early sensory cortices, e.g. postcentral gyrus and posterior occipital cortex (Dionne et al., 2010, Hiramatsu et al., 2011, Merabet et al., 2007, Sathian et al., 2011, Stilla and Sathian, 2008), but perception-related differences rather in higher-order cortical regions, e.g. the collateral sulcus as well as the parietal operculum and the insula (Cavina-Pratesi et al., 2010, Hiramatsu et al., 2011, Kitada et al., 2005). The unfamiliar dot pattern textures used in this experiment varied only along a single texture dimension, i.e. the average center-to-center dot spacing, ensuring that changes in other surface properties like color and friction do not influence the results. Stimulus presentation was always bimodal, but the sensory information content differed as texture information was varied either in the haptic, visual or in both channels.

We analyzed the data in two different ways. First we were interested in the average difference of the BOLD signal between both unimodal and the bimodal texture variation conditions, disregarding dot pattern differences. As we did not ask subjects to perform a cognitive task with the presented textures, we would expect to find differences, if any, at a rather early sensory processing stage. Second we were interested in relative differences of the BOLD response within each of these three conditions, taking into account the parametric dot pattern variation. Is the BOLD response modulated by texture differences when texture characteristics are varied either unimodally or bimodally? Based on previous behavioral studies we expected perceived roughness by touch to be almost perfectly correlated with the inter-dot spacing (Connor et al., 1990, Dépeault et al., 2009, Eck et al., 2011), while visual spatial density estimates should be negatively correlated with average inter-dot distance. Hence, no difference in the parametric BOLD modulation was expected between the two subjective measures tactile roughness and visual spatial density and the objective texture characteristic inter-dot spacing. However, to account for possible subjective perceptual differences we used individual post-fMRI ratings of haptic roughness and visual spatial density as well as the physical inter-dot spacing of the textures in separate parametric models.

Section snippets

Participants

Seventeen right-handed, healthy volunteers (13 women, 4 men; 27 ± 5.9 years) with normal or corrected-to-normal vision participated in the study. Subjects with calluses or injuries to the hands were excluded from participation. All participants were graduate and undergraduate students at Maastricht University. They were naïve to the hypotheses and received course credit or monetary compensation for their participation in the experiment. In accordance with the Declaration of Helsinki, written

Behavioral results

In general perceived haptic roughness increased with increasing inter-dot distance while the visually perceived spatial density decreased (see Fig. 4). This was expected from earlier work conducted by our group (Eck et al., 2011) and it was confirmed by a significant linear effect of spacing in two repeated measures ANOVAs of spacing on perceived roughness by touch and visual spatial density (F(1, 16) = 174.82, p < 0.001, partial η2 = 0.92 and F(1, 16) = 167.8, p < 0.001, partial η2 = 0.91, respectively).

Differences between unimodal and bimodal dot pattern conditions

Texture perception generated a very similar activation pattern irrespective of the modality in which texture information was varied, with activation overlapping primarily in early and higher-order visual, somatosensory and motor regions. These activations coincided with brain regions that were previously suggested to play a role in haptic and visual texture perception, namely the parietal operculum and insula, regions in the early visual cortex as well as the middle occipital gyrus, the

Conclusion

The present study aimed to investigate whether cortical processing in early sensory cortices changes when matching as compared to non-matching visual–haptic texture information is provided. Earlier research has shown that crossmodal interactions for texture perception exist at the behavioral level. However, neuroimaging studies focused primarily on visual and haptic texture processing separately, allowing to investigate only the overlap of both sensory processing streams. The variation of

Acknowledgments

The research leading to these results has received funding from the European Community's Seventh Framework Programme FP7/2007-2013 under grant agreement number PITN-GA-2008-214728. We also gratefully acknowledge the support of the BrainGain Smart Mix Programme of the Netherlands Ministry of Economic Affairs and the Netherlands Ministry of Education, Culture and Science. We would like to thank Ron Hellenbrand and Joost Mulders for their help with the stimulus creation.

Conflit of interest

I am one

References (62)

  • C. Lunghi et al.

    Touch disambiguates rivalrous perception at early stages of visual analysis

    Curr. Biol.

    (2010)
  • L. Merabet et al.

    Feeling by sight or seeing by touch?

    Neuron

    (2004)
  • A. Mouraux et al.

    A multisensory investigation of the functional significance of the “pain matrix”

    Neuroimage

    (2011)
  • D. Picard

    Partial perceptual equivalence between vision and touch for texture information

    Acta Psychol. (Amst)

    (2006)
  • P.E. Roland et al.

    Somatosensory detection of microgeometry, macrogeometry and kinesthesia in man

    Brain Res. Rev.

    (1987)
  • K. Sathian et al.

    Dual pathways for haptic and visual perception of spatial and texture information

    Neuroimage

    (2011)
  • A. Amedi et al.

    Visuo-haptic object-related activation in the ventral visual pathway

    Nat. Neurosci.

    (2001)
  • S.J. Bensmaia et al.

    SA1 and RA afferent responses to static and vibrating gratings

    J. Neurophysiol.

    (2006)
  • E. Borra et al.

    Projections to early visual areas v1 and v2 in the calcarine fissure from parietal association areas in the macaque

    Front. Neuroanat.

    (2011)
  • G.M. Boynton et al.

    Linear systems analysis of functional magnetic resonance imaging in human V1

    J. Neurosci.

    (1996)
  • G.T. Buracas et al.

    The relationship between task performance and functional magnetic resonance imaging response

    J. Neurosci.

    (2005)
  • H. Burton et al.

    Multiple parietal operculum subdivisions in humans: tactile activation maps

    Somatosens. Mot. Res.

    (2008)
  • J.S. Cant et al.

    Attention to form or surface properties modulates different regions of human occipitotemporal cortex

    Cereb. Cortex

    (2007)
  • J.S. Cant et al.

    Scratching beneath the surface: new insights into the functional properties of the lateral occipital area and parahippocampal place area

    J. Neurosci.

    (2011)
  • J.S. Cant et al.

    Object ensemble processing in human anterior–medial ventral visual cortex

    J. Neurosci.

    (2012)
  • C. Cappe et al.

    Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey

    Eur. J. Neurosci.

    (2005)
  • C. Cavina-Pratesi et al.

    Separate processing of texture and form in the ventral stream: evidence from fMRI and visual agnosia

    Cereb. Cortex

    (2009)
  • C. Cavina-Pratesi et al.

    Separate channels for processing form, texture, and color: evidence from fMRI adaptation and visual object agnosia

    Cereb Cortex

    (2010)
  • L.M. Chen et al.

    Fine-scale organization of SI (area 3b) in the squirrel monkey revealed with intrinsic optical imaging

    J. Neurophysiol.

    (2001)
  • C.E. Connor et al.

    Tactile roughness: neural codes that account for psychophysical magnitude estimates

    J. Neurosci.

    (1990)
  • A. Dépeault et al.

    Tactile perception of roughness: raised-dot spacing, density and disposition

    Exp. Brain Res.

    (2009)
  • Cited by (38)

    • Functional hierarchy for tactile processing in the visual cortex of sighted adults

      2019, NeuroImage
      Citation Excerpt :

      Our results align with previous studies that have documented cross-modal activations at the various visual processing hierarchy stages in sighted people. Some of these responses emerge as a result of learning new tactile or auditory skills (Amedi et al., 2007; Saito et al., 2006; Siuda-Krzywicka et al., 2016; Zangenehpour and Zatorre, 2010), while others are observed without any specific training, thus suggesting they are part of a functional repertoire allowed by the default cortical organization (Amedi et al., 2001; Campus et al., 2017; Eck et al., 2016, 2013; Lacey et al., 2014, 2010; Sathian et al., 2011; Stilla and Sathian, 2008; Tal et al., 2016). In the context of our work, it is particularly interesting that the fMRI study of Snow et al. (2014) has already suggested that neural populations responsive to tactually perceived shapes exist both in the early visual cortices and in higher-level ventral visual regions (i.e., V4 and the lateral occipital complex) of sighted subjects.

    • Brain networks underlying tactile softness perception: A functional magnetic resonance imaging study

      2019, NeuroImage
      Citation Excerpt :

      Neuroimaging studies focusing on one perceptual dimension of material properties have shown that activity in the parietal operculum and insula is related to the magnitude of perceived roughness (Kitada et al., 2005; Eck et al., 2016) and temperature (Craig et al., 2000). A more recent study demonstrated that the activity in a part of the occipital cortex is related to perceived roughness magnitude (Eck et al., 2013). These findings raise the possibility that compared with the macro-spatial properties, the parietal operculum, the insula, and a part of the occipital cortex are involved in extracting intensity information of material properties.

    • Visuo-haptic object perception

      2019, Multisensory Perception: From Laboratory to Clinic
    View all citing articles on Scopus
    View full text