Functional Use of Eye Movements for an Acting System

doi:10.1016/j.tics.2020.12.006

Trends in Cognitive Sciences

Volume 25, Issue 3, March 2021, Pages 252-263

https://doi.org/10.1016/j.tics.2020.12.006 Get rights and content

Highlights

Eye movements are tightly coupled with goal-directed hand movements in a variety of tasks.
When reaching for or intercepting an object, humans naturally direct their eyes to the object. Eye movements improve hand-movement accuracy through a combination of visual with efference-copy information.
Eye movements are flexible. They can be suppressed or sped up to optimally support the task.
During skill learning and sensorimotor adaptation, changes to eye and hand movements co-occur, suggesting that eye movements can support motor learning. This provides promising avenues for gaze training in motor learning and clinical rehabilitation.
The information exchange between the eye and hand movement system is bidirectional: execution of a concurrent hand movement can result in improvements in eye movements.

Movements of the eyes assist vision and support hand and body movements in a cooperative way. Despite their strong functional coupling, different types of movements are usually studied independently. We integrate knowledge from behavioral, neurophysiological, and clinical studies on how eye movements are coordinated with goal-directed hand movements and how they facilitate motor learning. Understanding the coordinated control of eye and hand movements can provide important insights into brain functions that are essential for performing or learning daily tasks in health and disease. This knowledge can also inform applications such as robotic manipulation and clinical rehabilitation.

Section snippets

The Supportive Role of Eye Movements in Manual Tasks

Humans and other primates move their eyes continuously to gather visual information and to guide their hands and body toward interesting objects. Despite strong functional coupling of eye and hand movements, research on the control and function of either type of movement has, to a large extent, been carried out in isolation. For example, eye movements are often studied with regard to how they enhance aspects of visual processing, such as spatial resolution [1,2]. Hand movements are considered

Eye Movements during Goal-Directed Hand Movements

Over the past two decades many studies have demonstrated that eye movements are controlled and organized to support accurate hand and body movements (e.g., [4., 5., 6., 7.]). We integrate these studies with recent work to first discuss how fixating or tracking a target with the eyes improves the accuracy of goal-directed hand movements. We then discuss how eye movements optimize performance by adjusting to task requirements, and how eye movements can be indicators of choices between actions or

Eye Movements in Motor Learning

Learning a new movement involves extracting and integrating sensory information about the movement target before, during, and after the movement, deciding which movement to make and when and where to move, and adapting appropriately to sensory feedback and error signals [36]. Because eye movements provide crucial information during goal-directed movements, they are probably also involved in movement learning. We discuss below the role of eye movements in two types of learning: motor skill

The Influence of Hand Movements on Eye Movements

In the paragraphs earlier we described a wide range of tasks in which hand movements are optimized by eye movements. In addition, improvements in eye movements can occur as a result of concurrent hand movements. Evidence for the effect of hand movements on eye movements comes primarily from studies investigating smooth-pursuit eye movements and hand movements while tracking moving targets. Figure 1C shows an example of the trajectories of the moving target, hand, and smooth-pursuit and saccadic

Concluding Remarks

Technological advances and the availability of affordable eye- and hand-tracking technology have boosted the quantity and quality of research simultaneously measuring eye and hand movements. These studies have made it clear that, in motor tasks, eye movements are not merely another effector to be investigated but actively guide hand movements (cf [5]). This review highlights the flexibility of eye movements in adjusting to the motor task as well as the skill level of the observer. Eye movements

Acknowledgments

This work was supported by a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant and Accelerator Supplement to M.S., and a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant to J.R.F.

Citation Diversity Statement

Recent work in several fields of science has identified a bias in citation practices such that papers from women and other minorities are under-cited relative to the number of such papers in the field [103]. We sought to proactively consider choosing references that reflect the diversity of the field in thought, form of contribution, gender, and other factors. We obtained predicted gender of the first and last author of each reference by using databases that store the probability of a name

Glossary

Eye–hand coordination: eye and hand movements that are largely synchronous and spatially aligned, without necessarily affecting one another.
Eye–hand interaction: eye and hand movements that affect one another, either unidirectionally or bidirectionally.
Efference copy: an internal copy of a motor command that is sent to sensory brain areas.
Explicit process (of motor adaptation): a strategic learning process driven by the error between the movement endpoint and the target, resulting in a strategic change

References (105)

J. Diedrichsen
The coordination of movement: optimal feedback control and beyond
Trends Cogn. Sci.
(2010)
M.F. Land et al.
In what ways do eye movements contribute to everyday activities?
Vis. Res.
(2001)
B. Bridgeman et al.
Ocular proprioception and efference copy in registering visual direction
Vis. Res.
(1991)
E. Brenner et al.
Continuous visual control of interception
Hum. Mov. Sci.
(2011)
J.L. Epelboim
The function of visual search and memory in sequential looking tasks
Vis. Res.
(1995)
S. Vieluf
The role of eye movements in motor sequence learning
Hum. Mov. Sci.
(2015)
M. Massing
Magnifying visual target information and the role of eye movements in motor sequence learning
Acta Psychol.
(2016)
M.L.R. Meister et al.
Getting directions from the hippocampus: the neural connection between looking and memory
Neurobiol. Learn. Mem.
(2016)
O. Hikosaka
Parallel neural networks for learning sequential procedures
Trends Neurosci.
(1999)
S.D. McDougle
Taking aim at the cognitive side of learning in sensorimotor adaptation tasks
Trends Cogn. Sci.
(2016)

A. Battaglia-Mayer et al.

Parieto-frontal networks for eye–hand coordination and movements

Handb. Clin. Neurol.

(2018)

H. Cui et al.

Posterior parietal cortex encodes autonomously selected motor plans

Neuron

(2007)

H.L. Dean

Only coherent spiking in posterior parietal cortex coordinates looking and reaching

Neuron

(2012)

L. Lünenburger

A possible role of the superior colliculus in eye–hand coordination

Prog. Brain Res.

(2001)

J.-R. Rizzo

Eye–hand re-coordination: a pilot investigation of gaze and reach biofeedback in chronic stroke

Prog. Brain Res.

(2019)

J. Intoy et al.

Finely tuned eye movements enhance visual acuity

Nat. Commun.

(2020)

K.R. Gegenfurtner

The interaction between vision and eye movements

Perception

(2016)

S.F. Neggers et al.

Ocular gaze is anchored to the target of an ongoing pointing movement

J. Neurophysiol.

(2000)

E.B. Lavoie

Using synchronized eye and motion tracking to determine high-precision eye-movement patterns during object-interaction tasks

J. Vis.

(2018)

L.A. Mrotek et al.

Target interception: hand–eye coordination and strategies

J. Neurosci.

(2007)

C. Prablanc

Optimal response of eye and hand motor systems in pointing at a visual target. I. Spatio-temporal characteristics of eye and hand movements and their relationships when varying the amount of visual information

Biol. Cybern.

(1979)

J.D. Fisk et al.

The organization of eye and limb movements during unrestricted reaching to targets in contralateral and ipsilateral visual space

Exp. Brain Res.

(1985)

U. Sailer

Spatial and temporal aspects of eye–hand coordination across different tasks

Exp. Brain Res.

(2000)

M. Poletti

Optimal multimodal integration in spatial localization

J. Neurosci.

(2013)

J. Paillard

The contribution of peripheral and central vision to visually guided reaching

D.W. Franklin et al.

Specificity of reflex adaptation for task-relevant variability

J. Neurosci.

(2008)

J.A. Saunders et al.

Humans use continuous visual feedback from the hand to control fast reaching movements

Exp. Brain Res.

(2003)

M. Dimitriou

The temporal evolution of feedback gains rapidly update to task demands

J. Neurosci.

(2013)

A.J. de Brouwer

Visuomotor feedback gains are modulated by gaze position

J. Neurophysiol.

(2018)

A.J. de Brouwer

Parallel specification of visuomotor feedback gains during bimanual reaching to independent goals

eNeuro

(2017)

E. Brenner et al.

Continuously updating one’s predictions underlies successful interception

J. Neurophysiol.

(2018)

M. Spering

Keep your eyes on the ball: smooth pursuit eye movements enhance prediction of visual motion

J. Neurophysiol.

(2011)

A.C. Sittig

The contribution of afferent information on position and velocity to the control of slow and fast human forearm movements

Exp. Brain Res.

(1987)

S.J. Sober et al.

Flexible strategies for sensory integration during motor planning

Nat. Neurosci.

(2005)

C. de la Malla

Potential systematic interception errors are avoided when tracking the target with one’s eyes

Sci. Rep.

(2017)

S. Delle Monache

Eye movements and manual interception of ballistic trajectories: effects of law of motion perturbations and occlusions

Exp. Brain Res.

(2015)

A. Goettker

Corrective saccades influence velocity judgments and interception

Sci. Rep.

(2019)

J. Fooken et al.

Decoding go/no-go decisions from eye movements

J. Vis.

(2019)

J. Fooken

Eye movement accuracy determines natural interception strategies

J. Vis.

(2016)

B. Cesqui

Gaze behavior in one-handed catching and its relation with interceptive performance: what the eyes can’t tell

PLoS One

(2015)

C. Cámara

Looking away from a moving target does not disrupt the way in which the movement toward the target is guided

J. Vis.

(2020)

C. de la Malla

The predictability of a target’s motion influences gaze, head, and hand movements when trying to intercept it

J. Neurophysiol.

(2019)

D.A. Barany

Perceptual decisions about object shape bias visuomotor coordination during rapid interception movements

J. Neurophysiol.

(2020)

M.F. Land et al.

From eye movements to actions: how batsmen hit the ball

Nat. Neurosci.

(2000)

M. Wilson

Psychomotor control in a virtual laparoscopic surgery training environment: gaze control parameters differentiate novices from experts

Surg. Endosc.

(2010)

J. Fooken et al.

Eye movements as a readout of sensorimotor decision processes

J. Neurophysiol.

(2020)

D.M. Wolpert

Principles of sensorimotor learning

Nat. Rev. Neurosci.

(2011)

J.V.V. Parr

Visual attention, EEG alpha power and T7-Fz connectivity are implicated in prosthetic hand control and can be optimized through gaze training

J. Neuroeng. Rehabil.

(2019)

S.J. Vine

Gaze training improves the retention and transfer of laparoscopic technical skills in novices

Surg. Endosc.

(2013)

U. Sailer

Eye–hand coordination during learning of a novel visuomotor task

J. Neurosci.

(2005)

Cited by (37)

The eyes anticipate where objects will move based on their shape
2023, Current Biology
Imagine staring into a clear river, starving, desperately searching for a fish to spear and cook. You see a dark shape lurking beneath the surface. It doesn’t resemble any sort of fish you’ve encountered before — but you’re hungry. To catch it, you need to anticipate which way it will move when you lunge for it, to compensate for your own sensory and motor processing delays¹^,²^,³. Yet you know nothing about the behaviour of this creature, and do not know in which direction it will try to escape. What cues do you then use to drive such anticipatory responses? Fortunately, many species⁴, including humans, have the remarkable ability to predict the directionality of objects based on their shape — even if they are unfamiliar and so we cannot rely on semantic knowledge about their movements⁵. While it is known that such directional inferences can guide attention⁵, we do not yet fully understand how such causal inferences are made, or the extent to which they enable anticipatory behaviours. Does the oculomotor system, which moves our eyes to optimise visual input, use directional inferences from shape to anticipate upcoming motion direction? Such anticipation is necessary to stabilise the moving object on the high-resolution fovea of the retina while tracking the shape, a primary goal of the oculomotor system⁶, and to guide any future interactions⁷^,⁸. Here, we leveraged a well-known behaviour of the oculomotor system: anticipatory smooth eye movements (ASEM), where an increase in eye velocity is observed in the direction of a stimulus’ expected motion, before the stimulus actually moves³, to show that the oculomotor system extracts directional information from shape, and uses this inference to predict and anticipate upcoming motion.
Differentiated, rather than shared, strategies for time-coordinated action in social and non-social domains in autistic individuals
2023, Cortex
Autism spectrum disorder (ASD) is a neurodevelopmental condition with a highly heterogeneous adult phenotype that includes social and non-social behavioral characteristics. The link between the characteristics assignable to the different domains remains unresolved. One possibility is that social and non-social behaviors in autism are modulated by a common underlying deficit. However, here we report evidence supporting an alternative concept that is individual-centered rather than deficit-centered. Individuals are assumed to have a distinctive style in the strategies they adopt to perform social and non-social tasks with these styles presumably being structured differently between autistic individuals and typically-developed (TD) individuals. We tested this hypothesis for the execution of time-coordinated (synchronized) actions. Participants performed (i) a social task that required synchronized gaze and pointing actions to interact with another person, and (ii) a non-social task that required finger-tapping actions synchronized to periodic stimuli at different time-scales and sensory modalities. In both tasks, synchronization behavior differed between ASD and TD groups. However, a principal component analysis of individual behaviors across tasks revealed associations between social and non-social features for the TD persons but such cross–domain associations were strikingly absent for autistic individuals. The highly differentiated strategies between domains in ASD are inconsistent with a general synchronization deficit and instead highlight the individualized developmental heterogeneity in the acquisition of domain-specific behaviors. We propose a cognitive model to help disentangle individual-centered from deficit-centered effects in other domains. Our findings reinforce the importance to identify individually differentiated phenotypes to personalize autism therapies.
An EMG-based wearable multifunctional Eye-control glass to control home appliances and communicate by voluntary blinks
2023, Biomedical Signal Processing and Control
The purpose of this work is to first apply the electromyogram (EMG) of corrugator supercilii muscles as the only signal to control home appliances and communicate with others by means of voluntary blinks, especially for the severely disabled. An EMG-based wearable eye-control glass is constructed as a multifunctional human–machine interface (HMI) device in this work. Twelve healthy subjects participated in the experiment to perform the home appliance eye-control task and the eye-communication task. This multifunctional eye-control glass is low-cost, easy-to-use, and wearable, and the experiment results indicate the high selectivity of voluntary blinks/natural blinks and left-eye blinks/right-eye blinks, excellent sensitivity, fast response time, high accuracy, low individual differences, and a prominent information transfer rate of the wearable device. This study provides a novel way for the construction of wearable eye-control devices to help healthy and disabled people conveniently control home appliances and communicate with the external world. It is gentler for users when applying this EMG-based wearable eye-control glass in various practical applications, like in the cinema or a meeting, than gesture and voice interactions. For disabled people without speaking ability, users can perform simple communication and express their needs with clinical staff with the help of this wearable device. Especially, this novel method greatly facilitates the daily lives of patients in the lock-in state through blink control. Moreover, the relationship between voluntary blinks and the movement of corrugator supercilii muscles is proven.
Touching with the eyes: Oculomotor self-touch induces illusory body ownership
2023, iScience
Self-touch plays a central role in the construction and plasticity of the bodily self. But which mechanisms support this role? Previous accounts emphasize the convergence of proprioceptive and tactile signals from the touching and the touched body parts. Here, we hypothesise that proprioceptive information is not necessary for self-touch modulation of body-ownership. Because eye movements do not rely on proprioceptive signals as limb movements do, we developed a novel oculomotor self-touch paradigm where voluntary eye movements generated corresponding tactile sensations. We then compared the effectiveness of eye versus hand self-touch movements in generating an illusion of owning a rubber hand. Voluntary oculomotor self-touch was as effective as hand-driven self-touch, suggesting that proprioception does not contribute to body ownership during self-touch. Self-touch may contribute to a unified sense of bodily self by binding voluntary actions toward our own body with their tactile consequences.
Gaze behaviour: A window into distinct cognitive processes revealed by the Tower of London test
2022, Vision Research
Citation Excerpt :
Gaze behaviour refers to the pattern of fixations and eye movements which enables sampling of high-resolution visual information that is crucial for perceiving and interacting with our environment. Seminal work has demonstrated that gaze behaviour is highly task dependent and tightly linked to the underlying perceptual, cognitive and motor processes (Castelhano, Mack and Henderson, 2009; de Brouwer, Glanagan, and Spering, 2021; Flanagan, Terao, and Johansson, 2008; Land and Hayhoe, 2001; Peltz, Hayhoe, Loeber, 2001; Yarbus, 1967). As such, the analysis of gaze behaviour during complex tasks provides a promising and non-invasive method to examine how specific eye movement patterns relate to various aspects of cognition and action.
The analysis of gaze behaviour during complex tasks provides a promising non-invasive method to examine how specific eye movement patterns relate to various aspects of cognition and action. Notably, the association between aspects of gaze behaviour and subsequent goal-directed action during high-level visuospatial problem solving remains elusive. Therefore, the current study comprehensively examined gaze behaviour using traditional and entropy-based gaze analyses in healthy adults (N = 27) while they performed the Freiburg version of the Tower of London task. Results demonstrated that both gaze analyses provided crucial temporal and spatial information related to planning, solution elaboration and execution. Specifically, gaze biases toward task-relevant areas (i.e., the work space) and an increase in gaze complexity (i.e., gaze transition entropy) during optimal performance reflected changes in cognitive demands as task difficulty increased. A comparison between optimal and non-optimal performance revealed sub-optimal gaze patterns that occurred in the early stages of planning, which were taken to reflect poor information extraction from the task environment and impaired maintenance of information in visuospatial working memory. Gaze behaviour during movement execution indicated an increased need to extract and process information from the goal space. Consequently, movement execution time increased in order to reverse erroneous movements and re-sequence the problem solution. Taken together, the traditional and entropy-based gaze analyses applied in the present study provide a promising approach to identify eye movement patterns that support neurocognitive performance on tasks relying on visuospatial planning and problem solving.
Gaze-specific motor memories for hand-reaching
2022, Current Biology
Citation Excerpt :
Despite the significant role of gaze coordinates in motor planning,22–26 the previous view has overlooked the potential role of gaze-related representations for acquisition of skilled hand reaching. Specifically, it has remained unclear how FOV and PER are associated with learning for reaching.21 Here, we address this issue and show that gaze states (target location on the retina) can contribute to forming and recalling multiple internal models for reaching movements.
Numerous studies have proposed that our adaptive motor behaviors depend on learning a map between sensory information and limb movement,1, 2, 3 called an “internal model.” From this perspective, how the brain represents internal models is a critical issue in motor learning, especially regarding their association with spatial frames processed in motor planning.⁴^,⁵ Extensive experimental evidence suggests that during planning stages for visually guided hand reaching, the brain transforms visual target representations in gaze-centered coordinates to motor commands in limb coordinates, via hand-target vectors in workspace coordinates.6, 7, 8, 9 While numerous studies have intensively investigated whether the learning for reaching occurs in workspace or limb coordinates,10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 the association of the learning with gaze coordinates still remains untested.²¹ Given the critical role of gaze-related spatial coding in reaching planning,22, 23, 24, 25, 26 the potential role of gaze states for learning is worth examining. Here, we show that motor memories for reaching are separately learned according to target location in gaze coordinates. Specifically, two opposing visuomotor rotations, which normally interfere with each other, can be simultaneously learned when each is associated with reaching to a foveal target and peripheral one. We also show that this gaze-dependent learning occurs in force-field adaptation. Furthermore, generalization of gaze-coupled reach adaptation is limited across central, right, and left visual fields. These results suggest that gaze states are available in the formation and recall of multiple internal models for reaching. Our findings provide novel evidence that a gaze-dependent spatial representation can provide a spatial coordinate framework for context-dependent motor learning.

View all citing articles on Scopus

View full text

Trends in Cognitive Sciences

ReviewFunctional Use of Eye Movements for an Acting System

Highlights

Section snippets

The Supportive Role of Eye Movements in Manual Tasks

Eye Movements during Goal-Directed Hand Movements

Eye Movements in Motor Learning

The Influence of Hand Movements on Eye Movements

Concluding Remarks

Acknowledgments

Citation Diversity Statement

Glossary

Trends Cogn. Sci.

Vis. Res.

Vis. Res.

Hum. Mov. Sci.

Vis. Res.

Hum. Mov. Sci.

Acta Psychol.

Neurobiol. Learn. Mem.

Trends Neurosci.

Trends Cogn. Sci.

Handb. Clin. Neurol.

Neuron

Neuron

Prog. Brain Res.

Prog. Brain Res.

Finely tuned eye movements enhance visual acuity

Nat. Commun.

The interaction between vision and eye movements

Perception

Ocular gaze is anchored to the target of an ongoing pointing movement

J. Neurophysiol.

Using synchronized eye and motion tracking to determine high-precision eye-movement patterns during object-interaction tasks

J. Vis.

Target interception: hand–eye coordination and strategies

J. Neurosci.

Optimal response of eye and hand motor systems in pointing at a visual target. I. Spatio-temporal characteristics of eye and hand movements and their relationships when varying the amount of visual information

Biol. Cybern.

The organization of eye and limb movements during unrestricted reaching to targets in contralateral and ipsilateral visual space

Exp. Brain Res.

Spatial and temporal aspects of eye–hand coordination across different tasks

Exp. Brain Res.

Optimal multimodal integration in spatial localization

J. Neurosci.

The contribution of peripheral and central vision to visually guided reaching

Specificity of reflex adaptation for task-relevant variability

J. Neurosci.

Humans use continuous visual feedback from the hand to control fast reaching movements

Exp. Brain Res.

The temporal evolution of feedback gains rapidly update to task demands

J. Neurosci.

Visuomotor feedback gains are modulated by gaze position

J. Neurophysiol.

Parallel specification of visuomotor feedback gains during bimanual reaching to independent goals

eNeuro

Continuously updating one’s predictions underlies successful interception

J. Neurophysiol.

Keep your eyes on the ball: smooth pursuit eye movements enhance prediction of visual motion

J. Neurophysiol.

The contribution of afferent information on position and velocity to the control of slow and fast human forearm movements

Exp. Brain Res.

Flexible strategies for sensory integration during motor planning

Nat. Neurosci.

Potential systematic interception errors are avoided when tracking the target with one’s eyes

Sci. Rep.

Eye movements and manual interception of ballistic trajectories: effects of law of motion perturbations and occlusions

Exp. Brain Res.

Corrective saccades influence velocity judgments and interception

Sci. Rep.

Decoding go/no-go decisions from eye movements

J. Vis.

Eye movement accuracy determines natural interception strategies

J. Vis.

Gaze behavior in one-handed catching and its relation with interceptive performance: what the eyes can’t tell

PLoS One

Looking away from a moving target does not disrupt the way in which the movement toward the target is guided

J. Vis.

The predictability of a target’s motion influences gaze, head, and hand movements when trying to intercept it

J. Neurophysiol.

Review
Functional Use of Eye Movements for an Acting System