Abstract
Humans use tactile feedback to perform skillful manipulation. When tactile sensory feedback is unavailable, for instance, if the fingers are anesthetized, dexterity is severely impaired. Imaging the deformation of the finger pad skin when in contact with a transparent plate provides information about the tactile feedback received by the central nervous system. Indeed, skin deformations are transduced into neural signals by the mechanoreceptors of the finger pad skin. Understanding how this feedback is used for active object manipulation would improve our understanding of human dexterity. In this paper, we present a new device for imaging the skin of the finger pad of one finger during manipulation performed with a precision grip. The device’s mass (300 g) makes it easy to use during unconstrained dexterous manipulation. Using this device, we reproduced the experiment performed in Delhaye et al. (2021). We extracted the strains aligned with the object’s movement, i.e., the vertical strains in the ulnar and radial parts of the fingerpad, to see how correlated they were with the grip force (GF) adaptation. Interestingly, parts of our results differed from those in Delhaye et al. (2021) due to weight and inertia differences between the devices, with average GF across participants differing significantly. Our results highlight a large variability in the behavior of the skin across participants, with generally low correlations between strain and GF adjustments, suggesting that skin deformations are not the primary driver of GF adaptation in this manipulation scenario.
Significance Statement
In this paper, we introduce a new device weighing 300 g and capable of imaging the skin of the finger pad of one finger during manipulation performed with a precision grip. This object is also capable of recording the forces and accelerations applied to the object. We reproduced the experiment performed in Delhaye et al. (2021) using this device. We extracted the strains aligned with the object’s movement to analyze how correlated these strains were with grip force (GF) adaptation. The behavior of the skin across participants presented a large variability, and we observed low correlations between strain and GF adjustments in most participants. Our results suggest that skin deformations are not the primary driver of GF adaptation in this manipulation scenario.
Introduction
We are capable of manipulating objects dexterously using our hands and fingers. To maintain a stable grasp, humans continuously adjust their grip force (GF), i.e., the force applied by the fingers normal to the contact surface, to counteract the load force (LF), i.e., the force caused by gravity and the inertia of the object tangential to the contact surface. GF adjustments are performed using a combination of predictions about the object’s response to the hand and arm movements and reactive control, driven by proprioceptive and tactile feedback (Flanagan et al., 1993; Augurelle et al., 2003; White et al., 2008). Dependent on the movements being performed, the adjustments of GF can be either intermittently or totally synchronous with changes in LF (Grover et al., 2018, 2019). Moreover, tactile feedback can elicit a wide range of responses in the arm and hand muscles, affecting fast responses, such as short-latency reflexes and long-latency reflexes, or voluntary responses (Pruszynski et al., 2016; Crevecoeur et al., 2017).
Human dexterity is enabled by the number of degrees of freedom (DOFs) and the remarkable sensing capabilities of the hand. The human hand is innervated with several thousand mechanoreceptors that respond to skin deformations and provide information about the contact between the finger skin and the object being manipulated (Birznieks et al., 2001; Johansson and Birznieks, 2004; Ackerley et al., 2014; Corniani and Saal, 2020). It is assumed that the tactile information these mechanoreceptors provide plays a significant role in dexterous manipulation. Indeed, it has been shown that anesthetizing the tactile feedback at the finger pads significantly impacts dexterity (Augurelle et al., 2003; Monzee et al., 2003). However, what tactile feedback is actually received and how this feedback is used during object manipulation remains poorly understood.
The behavior of the finger pad skin when in contact with a flat glass plate has been extensively investigated in passive situations, i.e., when the finger is immobilized, and a glass plate is brought into contact with it. Indeed, imaging of the skin has shown that when the glass plate was displaced relative to the finger, the skin started slipping at the periphery of the contact, with the slip propagating inwards until all skin was slipping (Levesque and Hayward, 2003; Tada and Kanade, 2004; André et al., 2011; Delhaye et al., 2014). Analysis of the finger pad images allowed the measurement of how the different parts of the skin deform under these tangential and rotational loads (Delhaye et al., 2016; du Bois de Dunilac et al., 2022). These images highlighted patterns of compression and dilation at the edge of the propagating slip wavefront. Microneurography showed that skin mechanoreceptors fire in response to these evolving compressive/dilative strains and that the firing rate depended on the amplitude of the strain rate (Delhaye et al., 2021). These results highlight that skin deformations occurring at the contact interface with an object are translated into sensory signals and provide feedback about the slip wavefront propagating across the finger pad skin. This information is potentially important in enabling the nervous system to control the GF during active object manipulation to prevent slip.
Recently, studies have examined how the finger pad skin behaves when we actively manipulate an object (Delhaye et al., 2021; Schiltz et al., 2022). These studies have highlighted the presence of strain patterns of compression and dilation appearing in specific regions of the finger. These strains could provide important information about the object being manipulated, the movement being performed, and the contact state. However, the device used in these studies and presented in Delhaye et al. (2021) has two important limitations stemming from the large weight of the device (500 g), preventing it from being manipulated easily in a precision grip and requiring a counterweight to allow participants to manipulate it more easily. First, this counterweight creates a discrepancy between the device’s apparent weight and its mass/inertia. Second, the counterweighting cable constrains the device to vertical movements, limiting the experiments that can be performed with it to involve strictly vertical motion.
In this paper, we present a new open-source device capable of imaging the skin of the finger pad of one finger during manipulation performed with a precision grip; we name the device InOb, for instrumented object, in the rest of the paper for brevity. InOb weighs only 300 g and is therefore easily manipulated by human participants. Using InOb, we performed an experiment where participants performed vertical oscillations, similar to the experiment performed in Delhaye et al. (2021). We observed a large variability in the behavior of participants’ skin during the task, with strain levels differing significantly. Across all participants, we found no clear relationship between skin strains and GF adjustments, suggesting that strains might not be the main driver for GF adjustments in this periodic task. Indeed, the periodicity of the task likely allows the central nervous system to predict the changes in LF and adjust GF synchronously.
Methods
Hardware and design
The InOb was designed to image the skin of one finger pad during active manipulation. For this reason, InOb is dimensioned to be manipulated in a precision grip; i.e., pinched between the thumb and the index finger. Its optical system allows the imaging of the contact that one finger pad makes with the transparent plate. A direct illumination system was selected as it results in high contrast between fingerprint ridges and valleys and leads to a compact structure (Bochereau et al., 2017). The custom optical system (Fig. 1C) is composed of a light source (Neewer SL-12), a half-mirror (Edmund Optics part #43-359), a camera (Raspberry Pi Camera v2), and a lens with adjustable focal length (CWBL2.8-12-3MP-C, 8–13.5 mm lens). From the light source, half the light goes through the half-mirror, reflects off the transparent plate in contact with the finger, and goes back to the half-mirror, where half of the light coming from the plate is reflected into the camera lens. The internal walls of InOb were painted using Black 3.0 paint (Stuart Semple) to minimize light reflections from the internal walls. This technique has been used to image the finger of humans during static loading (Delhaye et al., 2014; Bochereau et al., 2017; Delhaye et al., 2021; Khamis et al., 2021; Willemet et al., 2022) and dynamic manipulation (Delhaye et al., 2021).
Methods. A, Photograph of the object showing the glass plate, connected to the force sensor, through which the finger is going to be imaged. B, Side view of the instrumented object. The light source, half-mirror, viewing window, and camera represent the optical system allowing for the finger in contact with the viewing window to be imaged. The force sensor measures the 3D forces and torques. The LEDs are used to verify the synchronization of the force signals and the videos. C, Imaging principle of the instrument object. A light source (light-blue rectangle) illuminates the finger–glass contact. The image of the finger is reflected into the camera by a half-mirror. D, Image of the finger captured by InOb. E, Triangle shape evolution from frame t to frame t + 1 and corresponding representation of compression (red triangle) and dilation (blue triangle). F, Schematic of the forces acting on the object as well as the movement frequency and amplitude. G, Schematic view of the contact area with the division in three parts for analysis of the vertical strains of the finger (
Furthermore, InOb is equipped with a six-axis force/torque sensor (ATI Mini 40, ATI-IA, USA) connected to the plate in contact with the finger, measuring the forces and torques applied to the transparent plate during manipulation. A nine-DOF inertial measurement unit (IMU, BNO055) (Fig. 1A,B) measures the accelerations and orientation of the object. Finally, InOb takes advantage of the (undocumented) light sensitivity of ATI sensors to ensure that the force and video signals are synchronized; at the start of the trial, two LEDs, one next to the ATI sensor and one placed next to the camera lens, are flashed simultaneously for half a second providing a synchronous time event in the video and the force data recordings. The flashing LED leads to a change in the force along the x-axis of 0.05 N. All other parts of the device are 3D printed in tough polylactide (PLA) using a Makerbot Method X printer. The total weight of the object is 300 g and the dimensions are 75 × 77 × 225 mm (width × length × height).
The images from the camera are captured with a resolution of 720 × 1280 px (30 px/mm in the scene) at 60 fps. A lens with an adjustable focal length of 8–13 mm was chosen to achieve a focused image of the participant’s finger. Forces and torques are acquired at 200 Hz, and acceleration and orientation (Euler angles) are acquired at 100 Hz. The data are gathered using a Raspberry Pi 4B and Python 3.7. The acquisition code is provided in the following link.
Pilot study
Participants
Ten participants (four female, age 30.3 ± 4.38 years) participated in this experiment. The experimental protocol was approved by the ethics committee of University College Dublin and informed consent was obtained from all participants prior to commencement.
Experimental protocol
To test whether InOb was capable of accurately imaging the finger pad skin of the index finger, while synchronously acquiring the forces and accelerations applied on the object during a manipulation task, we performed an experiment where participants lifted the object in a precision grip, with the index finger touching the viewing window, and performed vertical oscillations of 20 cm peak-to-peak amplitude at a rate of 0.75 Hz for 20 s (Fig. 1F). Participants were instructed to reach the top and bottom of the movement at each beep of a metronome, which helped regulate the frequency of the movement. The amplitude of the movement was visually indicated using two visual markers placed adjacent to the trajectory of InOb. This experiment followed the same protocol as Delhaye et al. except that the weight was not compensated. After each trial, participants rested for 30 s before the start of the next trial. During this time, the viewing window was cleaned. In total, each participant performed 15 trials. The viewing plate was cleaned between trials.
After the 15 trials, the coefficient of friction (μ) of the contact between each participant’s finger and the viewing window of InOb was evaluated using the procedure described in Barrea et al. (2016). This procedure was conducted using InOb.
Data analysis
Force, torque, and acceleration analysis
Force, torque, acceleration, and orientation data were band-pass filtered using fourth-order low-pass Butterworth filters with cut-off frequencies of 40 Hz using zero-phase filtering. GF was estimated as the z-axis force measurement of the ATI sensor, and LF was estimated as twice the y-axis force of the ATI sensor (Fig. 1F). The start of a trial, i.e., the grasping of the object, was identified using the first peak in GF, and the end of a trial, i.e., the release of the grasp, was identified by the last peak in GF. The vertical velocity of the object was estimated by numerically integrating the z-axis acceleration measured by the IMU in the object’s reference frame. Individual oscillations (periods) were separated using the peaks of the vertical velocity.
We extracted the average GF and LF for each oscillation in order to analyze whether participants adapted their GF within and across trials. Using the coefficient of friction, we computed the friction limit as μ-1 and then computed the slip force (SF), i.e., the minimum GF required to maintain the object in a stable grasp for a given LF, as
Image analysis
To improve the contrast of the fingerprints on the video, each frame was spatially filtered using a 2D Gaussian band-pass filter (0.4 mm). Features in the contact area were detected and tracked across frames using the Lucas–Kanade–Tomasi algorithm (Lucas and Kanade, 1981; Shi and Tomasi, 1994; Bouguet, 2000). The distribution of the tracked features was uniform, with the size of the triangles being 0.123 ± 0.012 mm2 (mean ± SD). These features were used to compute the skin displacement field. Using these features, we tessellated the contact area into triangular skin elements by performing a Delaunay triangulation. The contact area was segmented using a semi-automatic machine-learning algorithm.
The size of the contact area was determined by counting the pixels inside the segmented region and converting the result to millimeter square using the conversion ratio of 30 px/mm given by the checkerboard pattern. This contact area can be modified by skin breaking or making contact, and skin deformations. To estimate the amount of skin lifting off or coming into contact at the edge of the contact area, we summed the areas of all the triangles of skin entering or leaving contact. A triangle was considered inside the contact area if its three vertices are located inside it. We computed the surface deformations leading to area changes by computing the area change of all Delaunay triangles in the contact area from one frame to the next.
The amount of skin slipping in each frame of the video was measured by considering that a triangular skin element had slipped if its centroid had moved more than 0.5 pixels across five consecutive frames. The stick ratio (SR) was then computed as the ratio of the area of skin stuck, i.e., the skin that is in the contact area but has not been detected as slipping as a fraction of the total contact area. We labeled each oscillation as “slip” if the SR dropped below 0.5. We used this label to compare the amplitude of the GF change in each oscillation between oscillations labeled as slip and trials with no slip.
The deformations of each triangular skin element were quantified by a strain tensor,
ϵ, having three independent values: the normal strain along the x-axis (
As the movement was vertical and based on the strain patterns observed by Delhaye et al. (2021), we divided the contact area into three segments in the ulnar to radial direction (Fig. 1G). Indeed, we expected strains to appear mainly in the radial and ulnar parts of the finger (Delhaye et al., 2021). The division between these segments was performed by computing the longest axis in the ulnar to radial direction of the contact area and dividing that length equally into three parts. We then computed the average of
Statistical analysis
To analyze how participants adapted their GF within and across trials, we computed a repeated measures ANOVA with the trial number (
To analyze the effect of slips on GF, we ran a paired t-test with the ΔGF in each oscillation for trials labeled as “slip” and those labeled as “no slip.” We expected to observe that oscillations labeled as “slip” would present a greater ΔGF than the oscillations labeled as “no slip”.
We computed the amplitude of the change of the weighted average of the strain in both the radial and ulnar parts of the finger in each oscillation, thus getting a single value per oscillation. To do so, for each oscillation we computed the difference between the maximum and minimum of
To investigate whether strain patterns could lead to adjustments of GF and explain the continuous modulation we observed, we computed cross-covariance using a sliding time window to analyze how well synchronized both signals were across a complete trial and how the time lag between both signals evolved across trials. We used a time window of 2.25 s to perform the cross-covariance, and this window was moved by 0.1 s at each step. This window size was selected to include two oscillations and reduce the variability of the cross-covariance measure. Indeed, if certain strain patterns trigger adjustments in GF, then we would expect the strains to change before a GF adjustment with a maximum covariance value for a negative time delay, i.e., GF lags strain. We performed a similar analysis between GF and peeling/laying, GF and compression/dilation, and GF and LF to investigate the coupling between GF and these different variables. Indeed, as LF is the main cause of strains appearing, we assumed we would observe similar covariances between strains and GF, and LF and GF.
To further explore changes in the relation between strains and GF, we extracted the correlation coefficient between GF and
Results
During a typical trial, participants lifted the object and performed vertical oscillations with an amplitude of 20 cm peak-to-peak with a frequency of 0.75 Hz for 20 s for a typical amount of 15 oscillations. Measured accelerations were in the range of ±4 m s−2 (Fig. 2B). The changes in the acceleration applied on the object led to the load force changing accordingly during the movement (Fig. 2A), with LF varying between 2 and 4 N. During a trial, participants adjusted their GF continuously to keep a safe grasp during the whole movement (Fig. 2A). The grip security is reflected in the ratio between GF and LF which remains above the friction limit (SF/LF) and remains stable across an entire trial (Fig. 2C), indicating that participants applied a large enough GF to prevent full slip. The SR stayed close to 1 for most oscillations but presented some significant drops close to 0.5 for some oscillations (Fig. 2D), which suggests that participants did not allow large levels of slip to occur during the trial. Skin lost contact and came into contact with the glass plate as oscillations were performed (Fig. 2E). Peeling and laying accounted for changes of around 5% of the initial contact area across oscillations for most participants, and global compression and dilation also accounted for around 1% (Fig. 3B).
Forces, acceleration (Acc), SR, peeling, and strain rate for a single trial of an exemplar participant. A, GF applied (blue line) and LF experienced (black line) by the index finger of the participant during the trial. B, Vertical acceleration of the instrumented object measured by the IMU. C, Ratio of the GF relative to the LF (black line) and friction limit (orange dashed line) which is estimated at the end of each experimental trial. D, SR of the skin, i.e., the region of the contact area where the skin is stuck to the plate as a proportion of the total contact area. E, Peeling and laying, and compression and dilation of the finger, i.e., amount of skin coming into and losing contact with the glass plate. F, Horizontal (
Mean GF and LF evolution, peeling amplitude, minimum slip ratio, and strain amplitude for each oscillation, aggregated across all trials. A, GF (blue line) and LF (black line) evolution within and across the 15 trials in the experiment aggregated across all participants. The shaded areas represent the standard deviation. B, Mean (black line) and SD (gray area) peeling amplitude across participants for each oscillation and each trial. C, Mean (black line) and SD (gray area) of the minimum slip ratio of each oscillation across participants. D, Mean (solid line) and SD (faded area) of the vertical strain modulation for the radial part of the finger (orange line) and the ulnar part of the finger (green line) of each oscillation across participants.
GF and safety margin
The mean GF of each oscillation progressively decreased within a trial, often starting at forces of around 10–12 N before decreasing to forces of around 8 N by the last oscillation in a trial (Figs. 2A, 3A). Moreover, GF also progressively decreased across trials (Fig. 3A). Indeed, a repeated measures ANOVA showed that GF did vary significantly with
Overall, all participants but one applied a GF that was, on average, well above (mean ± SD excess force GF–SF of 4.61 ± 2.58 N across participants) the SF required to maintain a stable grasp, as shown by all participants GF staying above the dashed line in Figure 4A. Indicating they all applied a substantial safety margin to ensure the object would not slip.
A, Participants’ mean GF and mean SF (black dots). The error bars represent the standard deviation. The dashed black line represents the SF limit, below which slip will occur. B, Participants’ mean GF and mean strain amplitude for each oscillation for the ulnar (green dots) and radial (orange dots) parts of the finger.
Peeling and compression of the skin
Skin regions lost contact with the plate and came into contact with the plate regularly during manipulation (Fig. 2E). Indeed, the amount of skin in contact with the plate varied by an average of 5% during trials. A repeated measures ANOVA computed on the amplitude of the change in contact area due to peeling/laying during an oscillation showed a significant effect of
Compression/dilation had a much smaller effect when compared with peeling/laying, accounting for changes in the contact area of about 1% (Fig. 3B). A repeated measures ANOVA computed on the amplitude of the change in contact area due to compression/dilation showed no significant effect of
Stick ratio
The SR varied significantly across participants, with most of the skin remaining stuck for most of a trial with some momentary sudden drops in SR (Fig. 2D) for most participants. It is worth noting that the levels of slip varied across participants, with the minimum SR averaged across participants being close to 0.5 (Fig. 3D). A repeated measures ANOVA on the minimal SR observed on each oscillation did not show a significant effect of
We labeled oscillations as “slip” when the SR dropped below 0.5 and as “no slip” otherwise. We extracted the amplitude of the change in GF for each oscillation and compared it across these two labeled sets. Oscillations labeled as “slip” showed a greater amplitude of change in GF compared to oscillations labeled “no slip” (Fig. 5). A paired t-test showed a significant difference between these two conditions (p < 0.001, CI = [0.9085 − 1.0629]).
Mean difference in GF change amplitude for oscillations labeled as “slip” (light-blue dots) and “no slip” (red dots) for each participant.
Local strain patterns
During the course of a single oscillation, we observed repeating patterns of local compression and dilation happening on the skin. The vertical component of the strains (
To better characterize this pattern, we computed the weighted average of the
Covariance between strains, peeling/laying, compression/dilation, LF, and GF
We computed a time-windowed cross-covariance analysis for each trial to verify how the covariance between GF and strains evolved within a trial. For visualization purposes, we averaged these cross-covariance results across all trials for each participant. Moreover, we used the amplitude of the strains in the radial and ulnar parts of the finger as an indicator of the noise of the cross-covariance analysis (Fig. 6A). If strain patterns in the ulnar and radial parts of the finger happen on the skin at the same frequency as the movement, we would expect a high covariance between GF and these strains. Moreover, if strains and tactile feedback drive GF adjustments, we would expect the strains to lead GF time-wise leading to a covariance peak at a positive lag. We observed that the covariance between strains and GF varied widely across participants (Fig. 6B). Some participants presented a large covariance between strains and GF, while others showed small to no covariance (Table 1). The lag between GF and strains also differed greatly across participants, with a global average of −4.8 ± 528 ms and −7.6 ± 541.3 ms for the covariance between GF and
A, Mean amplitude of
Mean ± SD of the maximum covariance between
We computed a similar analysis to study the covariance of peeling/laying and compression/dilation with GF to see whether changes in the contact area size have the potential to trigger GF adjustments. Similar to what we observed in the covariance between
Heatmap of the mean cross-covariance between GF and the strains for each participant. The cross-covariance was computed using a time window of 2.25 s. The yellow color highlights a positive covariance, and the blue color highlights a negative covariance.
A similar cross-covariance analysis investigating the covariance between GF and LF showed that these two signals were well correlated for all participants, with a covariance coefficient of r = 0.942 ± 0.046 across all participants and trials. Interestingly, the lag between GF and LF was small across all participants and trials, −0.01 ± 0.24 s (Fig. 8), highlighting how synchronous GF was in relation to LF.
Heatmap of the mean time-windowed cross-covariance between GF and LF across all trials for each participant. The yellow color highlights a positive covariance and the blue color highlights a negative covariance.
All these results show that the behavior of the skin in the contact area, or the peeling and laying of the skin, vary significantly across participants, with all these signals not being well representative of GF adaptation for all participants. On the other hand, these results show how well synchronized GF and LF are.
Given the variability across participants, we separated them based on their mean covariance in Table 1. We set a threshold of 0.3 on this covariance and obtained two groups of participants, group 1 containing the participants with a covariance greater or equal to 0.3 and group 2 containing the participants with a covariance less than 0.3. We measured the maximum covariance between
A, Covariance between the strains in the radial part of the finger and GF. B, Covariance between the strains in the ulnar part of the finger and GF. The pink line represents the mean and standard deviation of group 1, i.e., the group with the largest average max covariance, and the light green line represents the mean and standard deviation of group 2.
Correlation between GF and strains
We extracted the correlation coefficient between GF and the strains in the radial and ulnar parts of the finger for each oscillation of each trial of every participant (Fig. 2). The repeated measures ANOVA on the correlation between GF and
Discussion
In this paper, we present InOb, an instrumented object capable of imaging the skin of the finger pad of one finger, measuring the forces applied by the imaged finger, and measuring the accelerations and rotations experienced by the object during unconstrained manipulation. We present a use-case experiment for this object to highlight its usefulness in studying human manipulation and grip force control. In this experiment, 10 participants performed vertical oscillations of 20 cm while holding InOb in a precision grip between the thumb and index finger. Participants continuously adjusted their GF to follow the changes in the LF. We extracted various parameters quantifying the behavior of the skin in contact with the viewing plate. We observed a large variability of skin behaviors across participants, with some showing significant strains well correlated to GF adjustments while others showed small deformations or very little covariance with GF adjustments.
During the vertical oscillations, we observed that GF and LF were well synchronized with very little lag between them (Fig. 8). This indicates that participants continuously adjusted GF to match the changes in LF and maintain a safe grip with a large safety margin. Similar experiments have previously observed synchronicity between GF and LF (Flanagan et al., 1993; Delhaye et al., 2021). This suggests that humans adjust their GF in a predictive way so that GF changes in synchrony with LF. Recent studies have shown that the coupling between GF and LF can be context-dependent, and the strength of this coupling can change depending on the task being performed (Grover et al., 2018, 2019), with movement frequency and task difficulty influencing the GF–LF coupling. Grover et al. observed that more demanding tasks lead to a greater coupling between LF and GF. Indeed, movements performed at a higher frequency lead to a greater GF–LF coupling than movements performed at lower frequencies. Similarly, movements performed with a larger weight lead to a greater GF–LF coupling. This suggests that the combination of weight and frequency in this experiment is demanding and requires a strong coupling between GF and LF, with very little lag between the two signals.
Previous studies have shown that tactile feedback does directly impact GF during simple grip and lift tasks (Nowak et al., 2001; Augurelle et al., 2003; Monzee et al., 2003), with GF increasing significantly when the task was performed under fingertip anesthesia. However, our data showed no consistent latency or covariance between strains, peeling/laying, compression/dilation, and GF, suggesting that the skin behavior was not the direct cause of GF adjustments observed in all trials. Moreover, when the correlation between strains and GF was computed for each individual correlation, we did not observe any particular change across trials. In some participants, we observed strain patterns during vertical oscillations (Fig. 2) that were similar to those observed in Delhaye et al. (2021), with significant vertical strains appearing in the top and bottom parts of the finger. However, we observed significant differences in the behavior of the skin across participants, with some participants presenting large skin deformations and others presenting very small deformations. Similar variability between participant finger pad skin behavior has been observed in passive studies (Wang and Hayward, 2007; du Bois de Dunilac et al., 2022). du Bois de Dunilac et al. (2022) observed that the skin deformation rate varied significantly across participants, leading to significant differences in the timing of strains during the rotational load. It is worth noting that as strains are the consequence of the contact forces, therefore strains and LF are temporally linked. However, our data suggest that this relationship is highly complex. Our data would indicate that GF control likely relies more on proprioceptive feedback or predictive control mechanisms during this task, and suggest that tactile feedback is used sparingly when the task is predictable and could have a more important role in response to unexpected perturbations, in which proprioceptive feedback is not sufficient.
To further examine the differences in skin behavior between participants, we separated our participants into two groups based on the mean covariance between GF,
However, we also observed that oscillations, where slip did occur (SR <0.5), presented a greater change in GF (greater ΔGF) than oscillations where slip did not occur. This suggests that participants did use the SR and the slippage part of the finger pad skin as feedback to adjust their grip during the manipulation, leading to greater adjustment when slips occurred. However, the oscillating pattern of the movement and of GF does not allow us to extract timing information about the timing of a change in GF following a slip of the finger pad skin.
We observed that GF decreased within and across trials as participants either adapted their GF to the weight and inertia of the object or fatigued during the task. GF adaptation across trials has been observed in manipulation tasks (Crevecoeur et al., 2009; Giard et al., 2015). Despite the decrease in GF, we did not observe any significant change in strain levels within or across trials, suggesting that while participants adapted their GF, they did so in a way that did not lead to a significant reduction in grasp security.
Peeling and laying account for changes in the contact area of around 5 %, which is significantly larger than the change in contact area due to compression and dilation, and average strain in the ulnar and radial parts of the fingers. We observed significant changes in peeling/laying within trials. This could indicate that the skin peeling off the contact and coming back into contact with the plate could be the main source of information about the state of the contact between the finger pad and the grasped object. Indeed, slowly adapting type one afferents (SA-I) are sensitive to indentation (Handler and Ginty, 2021). However, the covariance and the lag at max covaraince between peeling and laying and GF varied largely across participants indicating that peeling and laying did not drive GF adjustments.
Comparing the instrumented object presented in Delhaye et al. (2021) and InOb highlights a few important differences. The device presented in Delhaye et al. (2021) weights 500 g, and the object is perfectly counterweighted to be more easily manipulable by participants, therefore having an apparent weight of 0 kg, but having the inertia of a 1 kg object. For this reason, the same experimental protocol as the one presented in this paper leads to important differences in the GF applied by participants during the experiment. Of note, we observed a distribution of mean GF that did not vary much across participants (Fig. 4B), whereas Delhaye et al. observed a much larger variation in mean GF due to the counterweighting of the object. Furthermore, due to the counterweighting mechanism, the object used by Delhaye et al. is constrained to vertical movements only, hence limiting the type of experiments that can be performed. The reduced weight of InOb (300 g) and the increased mobility allow a larger range of experiments to be performed, including experiments involving object rotations in various directions.
In order to have the capacity for unconstrained movement and limit the weight of the InOb device, a trade-off was made in terms of frame rate and pixel count when compared with the instrumented object presented in Delhaye et al. (2021). The reduced frame rate (60 fps vs 100 fps) could limit the capacity of InOb to be used in experiments involving very fast perturbations of the object that could lead to rapid changes at the skin/object interface, for instance in the case of collisions, as skin deformations might happen too rapidly for the camera to capture. Moreover, this device is only equipped with one force/torque sensor and hence is unable to differentiate the forces applied by the thumb and index finger.
In the future, we will use InOb to perform experiments aiming to understand the role of tactile feedback during active object manipulation. We aim at using InOb in experiments requiring a task requiring reactive control of the GF to study the influence of finger pad skin deformation on reactive adjustments of GF to unanticipated challenges to grasp security.
Conclusions
In this study, we introduce a new instrumented object weighing only 300 g and capable of imaging the finger pad skin, measuring the forces applied by participants on the object and the acceleration and orientation experienced by the object. This object is made of 3D-printed elements and off-the-shelf components, allowing it to be easily manufactured. We observed strain patterns appearing on the skin during vertical oscillations but with large variability across participants. We show the use of InOb in an experiment where participants performed a vertical oscillations experiment. We show that the GF adjustments were not well correlated with strains, suggesting participants did not solely rely on tactile feedback continuously during object manipulation. InOb opens up the experimental possibilities around the study of object manipulation and could provide very useful information about how tactile feedback is used during object manipulation.
Footnotes
The authors declare no competing financial interests.
D.C.B., S.B.D., and S.J.R. are supported by a grant from 10.13039/501100001602 Science Foundation Ireland (SFI), Ireland (17/FTL/4832), P.L. is supported by a grant from the 10.13039/501100000844 European Space Agency, B.P.D. is supported by a grant from the 10.13039/501100002661 Fonds De La Recherche Scientifique—FNRS (FNRS).
This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International license, which permits unrestricted use, distribution and reproduction in any medium provided that the original work is properly attributed.
References
In this issue
