Abstract
What we see is intimately linked to how we actively and systematically explore the world through eye movements. However, it is unknown to what degree visual experience during early development is necessary for such systematic visual exploration to emerge. The present study investigated visual exploration behavior in 10 human participants whose sight had been restored only in childhood or adulthood, after a period of congenital blindness because of dense bilateral congenital cataracts. Participants freely explored real-world images while their eye movements were recorded. Despite severe residual visual impairments and gaze instability (nystagmus), visual exploration patterns were preserved in individuals with reversed congenital cataract. Modeling analyses indicated that, similar to healthy control subjects, visual exploration in individuals with reversed congenital cataract was based on the low-level (luminance contrast) and high-level (object components) visual content of the images. Moreover, participants used visual short-term memory representations for narrowing down the exploration space. More systematic visual exploration in individuals with reversed congenital cataract was associated with better object recognition, suggesting that active vision might be a driving force for visual system development and recovery. The present results argue against a sensitive period for the development of neural mechanisms associated with visual exploration.
Significance Statement
Humans explore the visual world with systematic patterns of eye movements, but it is unknown whether early visual experience is necessary for the acquisition of visual exploration. Here, we show that sight recovery individuals who had been born blind demonstrate highly systematic eye movements while exploring real-world images, despite visual impairments and pervasive gaze instability. In fact, their eye movement patterns were predicted by those of normally sighted control subjects and models calculating eye movements based on low-level and high-level visual features and, moreover, taking memory information into account. Since object recognition performance was associated with systematic visual exploration, it was concluded that eye movements might be a driving factor for the development of the visual system.
Introduction
Prolonged visual deprivation from birth has been observed to result in the irreversible impairment of several visual functions (Lewis and Maurer, 2005; Röder and Kekunnaya, 2021). These findings have been taken as evidence for “sensitive periods” in brain development, defined as epochs during which adequate input is essential for full functional development (Knudsen, 2004; Hensch, 2005). In humans, sensitive periods have been studied in individuals who had been born blind or with severe visual impairments because of dense, bilateral cataracts and who later received cataract-removal surgery at different times during infancy, childhood, or even adulthood (Maurer et al., 2007; Röder et al., 2013; Ganesh et al., 2014). Despite improvements in vision after congenital cataract removal (Wright et al., 1992), basic visual abilities such as visual acuity (Ellemberg et al., 1999; Lambert et al., 2006) remain permanently impaired, especially if cataracts are not treated within the first few weeks of life. Moreover, higher-order visual functions such as feature binding and within-category viewpoint-independent discrimination, particularly of faces, have been found to only partially recover after congenital cataract surgery, and not to the degree expected by the recovery of visual acuity (Le Grand et al., 2001; Putzar et al., 2007, 2010; Ostrovsky et al., 2009). In addition to these perceptual deficits, individuals who had prolonged congenital bilateral visual deprivation (>8 weeks) typically also experience nystagmus (Rogers et al., 1981; Lambert et al., 2006; Birch et al., 2009). Nystagmus is a disorder of gaze stability that results in continuous, periodic, and involuntary motion of the eyes.
It has recently been shown that despite some distortions because of the superimposed nystagmus, eye movements to simple visual stimuli were reasonably precise and fast in individuals with reversed congenital cataract (Zerr et al., 2020). However, it is unclear whether higher levels of ocular control, such as the ability to generate typical patterns of active visual exploration of natural stimuli, recover after a transient phase of congenital visual deprivation. Active visual exploration is crucial for visual functions such as visual search and object identification, especially in noisy or ambiguous conditions (Einhäuser et al., 2004; Holm et al., 2008; Kietzmann et al., 2011). Furthermore, active visual exploration has been shown to be relevant for visual memory formation in typically sighted individuals (Hannula, 2010).
Previous research has suggested that visual exploration is guided by both bottom-up (stimulus-driven) and top-down mechanisms, which jointly define the direction toward which the eyes move. Stimulus-driven mechanisms use input characteristics such as luminance, color, orientation, and motion (Veale et al., 2017), whereas top-down mechanisms consider goals, memory, and contextual factors (Eckstein, 2011; Tatler et al., 2011; König et al., 2016). Stimulus-driven “saliency” models have successfully used low-level and high-level visual features to predict human eye movements during free viewing of scenes (Itti and Koch, 2000; Tatler et al., 2005; Kümmerer et al., 2017). Additionally, the repeated presentation of the same image has been used to assess the effects of short-term memory on visual exploration, that is, a nonreflexive aspect of gaze control (Ryan et al., 2000; Smith et al., 2006; Kaspar and König, 2011a,b). If an image is repeatedly encountered, the spread of visual exploration decreases (Hannula, 2010). It has been hypothesized that short-term memory representations provide top-down information, which, combined with bottom-up stimulus-driven maps in so-called priority maps, guide eye movements (Veale et al., 2017).
The degree to which the development of bottom-up and top-down mechanisms of active visual exploration depend on typical visual input after birth is unknown. Theories from developmental psychology have suggested that active visual exploration in infants is instrumental for the development of object knowledge (Johnson and Johnson, 2000). It remains to be investigated whether visual recovery after late sight restoration affects bottom-up, stimulus-driven visual exploration (Einhäuser et al., 2008a; Açık et al., 2009; Nuthmann and Henderson, 2010), and/or top-down, for instance, memory-based, visual exploration (Hannula, 2010).
In the present study, we used a free-viewing task in a sample of 10 individuals who had been born with dense, bilateral cataracts, which had been surgically removed later in life [congenital cataract reversal (CC) group; Table 1] in some participants, only in late childhood or adulthood. The distribution of gazed locations elicited by photographic stimuli (close-up images of different objects, plants, animals, and buildings) were assessed and compared with the typical visual exploration patterns of age-matched, normally sighted controls [sighted control (SC) group]. Further, the CC group was compared with individuals with nystagmus because of reasons other than congenital cataracts [nystagmus control (NC) group], and individuals with a history of developmental cataracts [developmental cataract reversal (DC) group], to isolate group differences specific to early visual deprivation rather than a general history of visual deficits. Finally, to explore top-down influences on visual exploration, the effects of short-term memory on eye movements were assessed by assessing the adaptation of visual exploration patterns for images which were repeatedly presented.
Participants description
Table 1-1
The relationship between visual acuity and age of surgery (CC and DC groups) and age of testing (all groups). Download Table 1-1, EPS file.
Materials and Methods
Participants
A total of 42 participants from four different populations were recruited at the LV Prasad Eye Institute and the local community of Hyderabad (India).
Congenital cataract reversal individuals (CC group)
Individuals were selected from a large number of patients who had been treated with the diagnosis of congenital cataracts. Based on medical records, a clinical history of bilateral congenital cataracts and a history of patterned visual deprivation were confirmed. A lack of fundus view and a lack of retinal glow were considered as evidence for the absence of patterned input reaching the retina before cataract surgery. Additionally, the presence of nystagmus, sensory strabismus, positive family history as well as absorbed lenses aided in the classification of CC participants.
The CC group consisted of 10 participants (2 females; mean age, 20.7 years; age range, 10.7–42.9 years) who had received cataract removal surgery at a mean age of 9.2 years (age range, 3 months to 22 years). These individuals were tested on average 11.4 years after cataract removal surgery (range, 7 months to 23.2 years). Of the 10 participants, 5 had a documented history of strabismus (esotropia, 2 participants; exotropia, 3 participants), 7 had implanted intraocular lenses, and the remaining 3 used corrective glasses. Four CC individuals had a documented family history of congenital cataracts, and four CC individuals had absorbed cataracts when presenting at the LV Prasad Eye Institute. Absorption of cataracts in middle to late childhood has been regularly observed in individuals born with dense congenital cataracts. Absorbed cataracts can be unambiguously differentiated from nondense or partial cataracts by, for instance, the morphology of the lens, anterior capsule wrinkling, as well as plaque or thickness of the stroma. Absorbed cataracts strongly imply dense cataracts, and therefore blindness, at birth. Presurgical visual acuity measurements in severely visually deprived individuals confirmed that at least 7 of 10 CC individuals were blind (i.e., had a visual acuity of <3/60; World Health Organization, 2019). The remaining three CC individuals had absorbed lenses; their presurgery vison corresponded to severe visual impairment, as defined by the World Health Organization. All CC participants additionally had nystagmus, which is strong evidence for the absence of patterned vision at birth. CC participants’ postsurgical visual acuity of the better eye ranged from 0.03 to 0.33 decimal units [geometric mean, 0.14; logarithm of the minimum angle of resolution (logMar), 0.47–1.4; logMar mean, 0.86]. A detailed description of CC participants is presented in Table 1 (see Schulze-Bonsel et al., 2006 for visual acuity equivalences and Extended Data Table 1-1).
Developmental cataract reversal group (DC group)
This control group allowed us to estimate the role of vision at birth for the acquisition of visual exploration behavior. The DC group consisted of nine individuals (four females; mean age, 15.6 years; age range, 11.6–24.4) with a history of bilateral cataracts, but not dense and/or congenital cataracts. These individuals allowed us to control for task-independent effects on eye movements because of cataract surgery (e.g., exploring the images with intraocular lenses). Cataract removal surgery had been performed at a mean age of 7.4 years (age range, 2.8–17.3 years); they were tested on average 8.2 years (range, 1.5–21.8 years) postsurgery. DC participants’ postsurgical visual acuity ranged from 0.46 to 1 decimal units (geometrical mean, 0.7; logMar, 0–0.33; logMar mean, 0.16). All DC participants were fitted with intraocular lenses.
Retrospective classification of CC and DC participants comes with some degree of uncertainty. However, the use of the classification criteria as implemented in the present study have recently been confirmed by an electrophysiological biomarker (Sourav et al., 2020).
Nystagmus group (NC group)
To disentangle the effects of congenital visual deprivation from the effects of prevailing sensory nystagmus, which was present in all CC participants, individuals with nystagmus because of conditions other than congenital cataracts were tested as additional control subjects. Individuals in this group did not experience a phase of severe visual deprivation. Therefore, this group allowed us to distinguish which changes in visual exploration behavior can be attributed to the effects of nystagmus versus congenital visual deprivation. This group comprised 10 participants (1 female; mean age, 15.0 years; age range, 8.7–37.3 years) with infantile nystagmus syndrome (idiopathic, 9 participants; oculocutaneous albinism, 1 participant), without a history of cataracts, severe visual impairment, or blindness. NC participants’ visual acuity ranged from 0.25 to 0.8 decimal units (geometrical mean, 0.45; logMar, 0.1–0.6; logMar mean, 0.35).
The sighted control group (SC group)
The SC group consisted of 13 individuals (3 females; mean age, 23.7 years; age range, 11.2–40.6 years) with normal or corrected-to-normal vision. This group was partially age matched to the CC group (no significant difference in age at testing; t(21) = −0.84, p = 0.41). The SC group allowed us to establish typical eye movement parameters for healthy individuals in the current experimental setting and for the used images.
All individuals were tested at the LV Prasad Eye Institute. None of the participants had any other sensory deficit or neurologic disorder, diagnosed or self-reported. Expenses associated with taking part in the study were reimbursed. Minors additionally received a small present. Participants, and if applicable, their legal guardians, were informed about the study and received the instructions in one of the languages they were able to understand (in most cases Telegu, Hindi, or English). All participants gave written informed consent before participating in the study; in the case of minors, legal guardians additionally provided informed consent. The study was approved by the ethics board of the Faculty of Psychology and Human Movement Science of the University of Hamburg (Germany) and by the ethics board of the LV Prasad Eye Institute.
Stimuli
Forty-nine images from the Natural Face and Object Stimuli image set were used in this study (Rossion et al., 2015; https://face-categorization-lab.webnode.com/resources/natural-face-stimuli/). The images displayed objects representing seven different categories: animals, chairs, fruits, guitars, houses, plants, and telephones. Objects were located close to the center of the image. Images were displayed in grayscale at an 800 × 800 pixel resolution, subtending 19.6 × 19.6 visual degrees. Stimuli were generated in MATLAB (MathWorks) using Psychtoolbox 3 (Brainard, 1997; Kleiner et al., 2007) on a Windows 7 PC and presented with a 24 inch Eizo FG2421 LCD monitor at a resolution of 1920 × 1080 at 120 Hz.
Because of copyright concerns, the figures shown here use line drawings of the actual images presented during the experiment.
Eye-tracking and calibration
Eye movements were recorded with a video-based binocular eye-tracking system at 500 Hz (EyeLink1000 Plus, SR Research). Subjects were seated in a darkened room and placed their heads on a chin rest such that their eyes were at a distance of 60 cm from the screen. Because of several participants presenting with nystagmus, it was not possible to use the built-in standard online calibration method of the eye-tracker system. Instead, a custom-made calibration routine was used. This calibration routine used five screen positions as points (presented at the screen center, 15° right and left of the center and 8.5° above and below the center). Each participant was asked to look at these five points. Next, the screen center position was displayed again, to estimate calibration error. The experimenter manually controlled the calibration. During the presentation of each calibration position, the experimenter decided whether an eye movement was performed to the corresponding point and selected low-velocity periods of the nystagmus at each calibration point. These low-velocity periods typically follow the corrective saccade of the nystagmus; that is, they are aligned with the target position. The online calibration was performed to visually confirm that calibration points aligned with the five position patterns. This confirmation was necessary to decide whether the procedure had to be repeated, or if the calibration was sufficiently precise to continue. During offline calibration, low-velocity periods of nystagmus were selected as described for online calibration.
The median positions of the selected gaze samples were fitted with a polynomial function (Stampe, 1993) to the corresponding screen positions. This is the same algorithm as the one implemented in the EyeLink eye-tracker software. The same calibration procedure was applied to all participants regardless of whether they had nystagmus or not. Calibration error was calculated only for the central position of the screen and did not differ between groups (robust linear model contrasts; all contrasts, p > 0.05; mean CC group: 0.83°; SD, 0.95; mean SC group: 0.31°; SD, 0.12; mean DC group: 0.59°; SD, 0.37; mean NC group: 0.78°; SD, 0.83).
A certain proportion of gaze data was missing when the gaze fell outside of the image or during periods when the eye tracker lost the pupil. On an average, 9.9% of CC participants’ gaze data was missing, compared with 3.69% for SC, 4.27% for DC, and 9.93% for NC participants.
To guarantee a sufficiently reliable estimate, only the visual exploration data from images with at least 50% valid recordings (i.e., gaze location values within the image) were included for analyses. Under this criterion, seven, one, one, and six image explorations had to be disregarded for the CC, SC, DC, and NC group, respectively. In total, we discarded <0.6% of the data.
Procedure
After the calibration was completed, the experiment was conducted in two blocks of 14 trials each. A trial consisted of two images presented sequentially. Each trial started with a white fixation dot (diameter, ∼1.8°) presented in the center of the screen for 1 s, followed by the presentation of the first image for 4 s. Next, a central white fixation dot was shown a second time for 1 s, which was followed by the presentation of the second image for 4 s. Participants were instructed to visually explore the images and report the names of the objects they had encountered in the two images at the end of the trial. After participants provided the names of the two images, the experimenter decided whether each image was correctly named. The experimenters knew about the possible categories and were instructed to accept responses at the exemplar level (e.g., banana) and categorical level (e.g., fruit).
In 7 of the 28 trial image pairs, the same image was presented as the first and second image. In another seven trial pairs, the two images were different, but from the same object category. In the remaining 14 trial pairs, the images were from different categories. Image presentation order was randomized across subjects with respect to pair type (repeated image, repeated category, different category). Order of presentation within a pair in the repeated and different category pairs was randomized across subjects. The experiment took ∼15–20 min, depending on the duration of the eye-tracker calibration procedure.
Data analysis and statistics
The common procedure in eye-tracking research is to use fixation positions as the unit of analysis. However, the nystagmus of the CC and NC individuals made it impossible to define fixations by using typical velocity and acceleration thresholds. Hence, the dependent variables for all participants were calculated with respect to the position of all eye-tracking data samples (subsequently referred to as “gaze” data, obtained at a 500 Hz sampling rate). Several studies have shown that individuals with nystagmus gather information during the complete period of the nystagmus, and not only during the low-velocity phase (Jin et al., 1989; Goldstein et al., 1992; Waugh and Bedell, 1992; Dunn et al., 2014). Therefore, for uniformity of the analysis across groups, all gaze data were used, including high-velocity samples that would normally be considered saccades.
Instantaneous gaze velocity
The extent of gaze instability in all participants was estimated by deriving “instantaneous gaze velocity.” Usually, gaze velocity is calculated from multiple samples to remove high-frequency noise inherent to oculomotor recordings (Stampe, 1993; Holmqvist, 2011). In the present study, we used a modified version of the 2-point central difference algorithm (Bahill et al., 1982) that is standard in the literature (Engbert and Kliegl, 2003; Dimigen et al., 2009; Otero-Millan et al., 2014), and used as the default by the EyeLink eye-tracker (SR Research, 2019). For a sample point n, the corresponding instantaneous gaze velocity [in visual degrees per second (°/s)], is defined as the sum of six nonconsecutive eye-tracking gaze samples (position in screen pixels), as follows:
Entropy
To assess the spread or dispersion of visual exploration, the informational entropy of the spatial distribution of gazed locations was calculated for each image and participant (Açık et al., 2010; Wilming et al., 2011; Shiferaw et al., 2019). Informational entropy is defined as the average amount of information of a random variable. Entropy is higher when uncertainty of an outcome is high and thus when events carry relatively more information. Entropy is maximal for variables with a uniform distribution. In terms of visual exploration patterns, a higher spread or a broader spatial distribution of gazed locations results in higher entropy. Conversely, a narrow spatial distribution results in lower entropy values.
To calculate entropy values for each subject and image, first, a discrete spatial distribution of gazed locations was constructed by dividing each image into a 20 × 20 matrix of 2° x 2° cells. Next, we counted how many times each cell was gazed at by a given participant. Finally, the entropy value of each spatial distribution was calculated by the following:
To confirm the robustness of the results, entropy values were additionally calculated based on gaze distributions for a smaller (1° × 1°) and larger (4° × 4°) cell size. Results did not differ, and thus we report the results based on a cell size of 2° × 2°.
Predictor maps for visual exploration patterns
We evaluated how well each participant’s exploration patterns were explained by the following: (1) the exploration pattern of other participants; (2) the low-level features; and (3) the high-level visual features of the presented images. The following three different predictor maps were correspondingly generated: (1) the visual exploration pattern for each image as assessed in the SC control group; (2) the low-level Intensity Contrast Feature (ICF) model of the images (Kümmerer et al., 2017); and (3) the high-level feature map for the images, as defined by the DeepGaze II (DG-II) model (Kümmerer et al., 2016).
The first predictor was derived from the empirical distribution of gaze locations across all participants of the SC group. A two-dimensional spatial probability distribution was constructed for each image by pooling all the gaze eye-tracking samples of SC individuals for each image. For predictions within the SC group, the SC predictor map was constructed in a leave-one-out cross-validation procedure. Pixel-level gaze counts were spatially smoothed with a two-dimensional Gaussian unit kernel (full-width at half-maximum = 2°) and normalized by dividing by the total count of gaze samples.
The second predictor map, ICF, consisted of a two-dimensional spatial distribution constructed based on the low-level features of images (luminance contrast). Different low-level features are known to be highly correlated (Onat et al., 2014), and simple models based on contrast features seem to perform as well as more complex models that include multiple low-level visual features (Kienzle et al., 2009). Thus, we used a low-level model based solely on luminance contrast.
The third predictor map, DG-II, consisted of a two-dimensional spatial distribution constructed based on features derived by a deep neural network optimized for object recognition (Simonyan and Zisserman, 2014). The DeepGaze II model is currently considered the best performing model for free viewing according to the MIT Saliency Benchmark 2019 (http://saliency.mit.edu/). The DG-II model selects local features that serve as a basis for object classification, but it does not segment or tag objects. Note that the DG-II model typically performs the best at predicting eye movement behavior for images depicting text or faces, which our stimuli dataset did not include. Nevertheless, the DG-II model has been shown to outperform the ICF model even in the absence of such features (Kümmerer et al., 2017).
ICF and DG-II maps were computed for each image using the Python code made available by Matthias Kümmerer and the Bethge Laboratory (https://deepgaze.bethgelab.org). ICF and DG-II maps were generated for each original image, as well as for three low-pass-filtered image versions. The latter were obtained by filtering the images using a 2D Gaussian kernel with frequency cutoffs (reduction, 0.67) at 0.5, 1, and 2 visual degrees, respectively.
Area under the curve
To determine how well a given predictor map explained participants’ visual exploration patterns, we tested whether the values of the predictor map allowed a classification of image locations as gazed versus nongazed. For nongazed locations, values were taken from gazed locations in other images by the same participant. This procedure to define nongazed locations was introduced to avoid an inflated classification success because of possible spatial biases (Tatler et al., 2005; Wilming et al., 2011; Bylinskii et al., 2016), in both human visual exploration patterns and photographic image features (Tatler et al., 2005; Tatler, 2007; Einhäuser et al., 2008b). For each participant, gazed and nongazed values were pooled across images. These values were used to estimate the classification success of a predictor map by calculating the area under the curve (AUC) of the receiver operator characteristic curve (Green and Swets, 1988; Fawcett, 2006). AUC values can be calculated by first taking the Mann–Whitney U statistic (also called Wilcoxon rank-sum test) between gazed and nongazed values of the predictor map, as follows:
AUC values range between 0 and 1, with 0.5 corresponding to chance discrimination and 1 indicating perfect classification. In some analyses, AUC values were obtained per image rather than per subject: gazed and nongazed values were pooled across participants for each image instead across images for each participant.
To further control for any additional potential analyzing bias, control AUC values were calculated as follows: instead of using the predictor map for a given image, images were shuffled; that is, the predictor map of another randomly selected image was used to predict visual exploration of a given image.
Statistical tests
Group differences in instantaneous gaze velocity, entropy, and AUC values were evaluated with robust linear regression models using a categorical group factor. The models used an iteratively reweighted least-squares method using a bisquare weight function, as implemented in the MATLAB R2019b function fitlm (Holland and Welsch, 1977). As there were six possible between-group comparisons, group contrasts were tested at a Bonferroni-corrected significance level of 0.05/6.
Moreover, AUC values for the SC, ICF and DG-II predictor maps were evaluated for different time periods after image presentation. This analysis tested whether classification success depended on the phase of visual exploration. AUC values were computed from data partitions obtained by dividing each participant’s gaze data into eight nonoverlapping 500 ms intervals, from the beginning to the end of the trial. These sets of AUC values, excluding the first interval, were entered in a linear mixed-effects model with group as a categorical factor, a time interval covariate as a fixed effect (seven levels), and participant identity as a random effect.
In the CC and NC groups, we additionally evaluated differences in AUC values generated from gaze locations, obtained by dividing each participant’s gaze data into 10 bins according to the magnitude of instantaneous gaze velocity. This analysis tested whether classification success of the SC predictor map depended on gaze stability in the CC and NC group. The new set of AUC values were entered in a linear mixed-effects model with group as a categorical factor, a velocity quantile covariate as a fixed effect, and participant identity as a random effect.
To assess the effects of short-term memory on visual exploration patterns, the repetition effect was evaluated by comparing the entropy values between the first and the second image of trial pairs. Entropy values for each image were entered in a linear mixed-effects model, with participant group (four levels: CC, SC, DC, NC) and image order within a pair (two levels: first or second) as fixed effect predictors, and participant identity as a random effect. This analysis was performed separately for each type of trial pair (repeated identity, repeated category, unrelated new image).
Linear mixed-effect models were calculated in R (version 3.6.3), using restricted maximum likelihood estimation as implemented in the lme4 package (Bates et al., 2015). The reported p-values were based on the t-distribution using degrees of freedom calculated with the Satterthwaite method, as implemented by the lmerTest package (Kuznetsova et al., 2017).
Differences between groups in object recognition performance were evaluated with a generalized linear model using a binomial distribution and a logit link function, as implemented in the R stats package. The same procedure was used to evaluate the association between visual acuity and the AUC values in CC participants. Detailed specifications and output summaries of all models are described in the corresponding extended data figures.
Data availability
The code for the statistical analyses, figures, and the anonymized, preprocessed data are available at the Research Data Repository of the University of Hamburg (doi: 10.25592/uhhfdm.1520). Original eye-tracking datasets are available on request from the corresponding author.
Results
Gaze stability is severely affected in CC participants
As expected from the prevailing sensory nystagmus, eye movement trajectories were considerably altered in the CC group. Figure 1a displays examples of a single trial eye-tracking recording from one participant of each group (Movies 1, 2, Extended Data Figs. 1-1, 1-2). SC and DC participants showed the prototypical gaze kinematics of visual exploration of static images: their gaze movements were characterized by periods of high stability (i.e., fixations) interrupted by short periods of displacement at a high velocity (i.e., saccades). By contrast, CC and NC participants’ gaze movements were in a continuous, periodic displacement, as is typical of nystagmus. Participants’ gaze stability was quantified in terms of instantaneous gaze velocity; that is, how fast and in which direction the eyes moved from one moment to another (see subsections Data analysis and statistics, and Instantaneous gaze velocity). The magnitude of instantaneous gaze velocity was significantly higher in CC individuals compared with SC individuals (robust linear model contrast, p < 0.001; Fig. 1b, Extended Data Fig. 1-3, full statistical results) and DC individuals (p < 0.001), but was lower than in NC individuals (p = 0.004). Gaze velocities showed no clear direction in CC individuals, whereas for NC individuals, gaze velocities were mostly along the horizontal direction (Fig. 1c). Such a pattern in NC individuals is typical of horizontal jerk nystagmus (Abadi and Bjerre, 2002).
Eye movement kinematics during the visual exploration of an example image. a, Examples of eye movement recordings of one participant from each group. Images were explored for 4 s. The left panels depict the gaze traces overlaid on a line-drawing sketch of the original photographic grayscale image; note that participants watched the original grayscale images. The right panels show eye movement traces as they progress over time and space along the horizontal (dark lines) and vertical (light lines) dimension. Extended Data Figures 1-1 and 1-2 show two other examples of eye movement recordings. b, Distribution of the magnitude of instantaneous gaze velocity. Light lines indicate each participant’s distribution, and dark lines each group’s average distribution. Colored circles display each participant’s median value, and the yellow dots and error bars display the group’s mean and SEM (Extended Data Fig. 1-3, statistics). c, Distribution of instantaneous gaze velocity (bin size, 16°/s; densities were individually generated for each participant and then averaged across the participants of each group). The color scale indicates the probability of a given gaze velocity in log10 scale. Yellow and white contours indicate areas that span ∼75% and 90% of the distribution. In all figures, significant contrasts among groups are indicated as follows: *p < 0.01, **p < 0.001, ***p < 0.0001, respectively.
Figure 1-1
Examples of eye movement recordings of one participant from each of the four groups. Download Figure 1-1, EPS file.
Figure 1-2
Examples of eye movement recordings of one participant from each of the four groups. Download Figure 1-2, EPS file.
Figure 1-3
Instantaneous gaze velocity statistical result. Download Figure 1-3, DOCX file.
Examples of visual exploration patterns. Each subpanel shows the exploration of one participant for the complete period of image presentation. Each red dot represents one eye-tracking gaze sample (downsampled from 500 to 125 Hz for better visualization) overlaid on a line-drawing sketch of the original photographic grayscale image; note that participants watched the original grayscale images.
Examples of visual exploration patterns. Same as in Movie 1.
In sum, these results confirm that, in contrast to SC and DC participants, the gazes of CC and NC participants were in a state of continuous motion; that is, gaze stability was reduced in these two groups.
Visual exploration patterns of CC participants are stimulus driven and similar to those of control subjects
Four examples of group-pooled exploration patterns overlaid over line drawings of the original grayscale images are depicted in Figure 2. These images illustrate the resemblance of visual exploration patterns across groups. To quantitatively evaluate whether visual exploration patterns for natural scenes were stimulus driven in CC participants and to what degree they followed the same principles as in normally sighted control subjects, informational entropy was derived to parametrize the width of the spatial distribution of gazed locations (Shiferaw et al., 2019). Low entropy scores indicate a low degree of randomness of visual exploration patterns (see subections Data analysis and statistics, and Entropy).
Examples of visual exploration by group. The subpanels show, for different images and the four groups of participants (Fig. 1, description), the spatial distributions of the probabilities to gaze different locations (pooled across participants and smoothed with a 2D Gaussian unit kernel), superimposed over line-drawing sketches of the original images. Warmer colors indicate a higher probability to gaze a location. Yellow contours indicate areas that span the top 50%, 75%, and 95% of the spatial distribution. As this distribution is constructed from all gaze eye-tracking samples (each occurring every 2 ms), these maps are equivalent to the spatial distributions of dwell time. The mean of entropy and AUC values for each of the four images are indicated by the corresponding symbol (star, square, and left and right pointing triangles) in Figure 3, b and f. The last column shows the DG-II and ICF predictor maps for each image. Extended Data Figure 2-1 shows the grand average of the spatial distributions of the probability to gaze a certain location across all images separately for each of the four groups. In addition, the corresponding grand average DG-II and ICF predictor maps are displayed.
Figure 2-1
Exploratory and feature bias. a, Grand average, across all image and participants (per group), of the spatial distributions of the probability to gaze. b, Grand average, across all images, of DG-II and ICF predictor maps. Download Figure 2-1, EPS file.
As expected from the nystagmus-related gaze instability, visual exploration by CC individuals covered a wider area of the images than visual exploration by SC individuals. CC participants’ entropy values were higher than those for SC and DC participants (robust linear model contrasts, both p < 0.001; Fig. 3a, Extended Data Fig. 3-1, statistics), but were not different from those of the NC participants (p = 0.27). Thus, higher entropy values in the CC group were a consequence of nystagmus, rather than of congenital visual deprivation.
Spatial spread and predictability of visual exploration patterns. a, Mean gaze entropy for each group (yellow dot with error bars, indicating the SEM) as well as for individual participants (colored dots; Extended Data Fig. 3-1, statistics). b, CC participants gaze entropy per image compared with the gaze entropy values of the other three control groups. Colored continuous lines indicate a linear regression line for entropy values of the CC group (x-axis) and each one of the three control groups (SC, blue; DC, green; NC, orange). The top left inset depicts the corresponding Pearson’s correlation values (in a red scale, top right corner) and the corresponding p-values (in green, lower left corner). Asterisks indicate significant correlations after controlling for multiple comparisons (α = 0.05/6). c, AUC values of the SC predictor map per participant and group. Dark-colored dots indicate AUC values for individual participants as derived by the predictor maps of the SC group to classify gaze and nongazed location. Light-colored circles from the corresponding AUC values for the control analysis in which image correspondence was shuffled. Bottom, Colored stars indicate that actual and control analysis values significantly differed. The control analysis values were not different from 0.5 (chance level). Extended Data Figure 3-2 shows statistics, and Extended Data Figure 3-3 shows the relationship between AUC values and different CC participants’ characteristics. d, AUC values of the SC predictor map across time. Curves show, for each group, AUC values calculated from consecutive 500 ms data partitions (Extended Data Fig. 3-4, statistics). e, AUC values of the SC predictor map as a function of instantaneous gaze velocity. SC predictor maps were used to calculate gaze in CC individuals separately for 10 quantiles of instantaneous gaze velocity (Extended Data Fig. 3-5, statistics). Extended Data Figure 3-6 shows the relationship between gaze velocity during fixations (SC and DC groups) and CC and NC participants’ first and second instantaneous gaze velocity quantiles. f, Correlations of entropy and AUC values across all images for the CC group. Different object categories are color coded. Extended Data Figure 3-7 shows the same correlation for SC, NC, and DC groups.
Figure 3-1
Entropy statistical result. Download Figure 3-1, DOCX file.
Figure 3-2
AUC (SC predictor map) statistical result. Download Figure 3-2, DOCX file.
Figure 3-3
a–d, Relationship between SC predictor map AUC values in CC individuals and logMAR visual acuity (a), age at testing (b), age at surgery (c), and time from surgery (d). Download Figure 3-3, EPS file.
Figure 3-4
AUC (SC predictor) per time interval statistical result. Download Figure 3-4, DOCX file.
Figure 3-5
AUC (SC predictor) per velocity quantile statistical result. Download Figure 3-5, DOCX file.
Figure 3-6
Fixational gaze velocity SC and DC groups and first and second quantiles of gaze velocity in the CC and NC groups. Top, The grand average distribution, for the SC and DC groups, of the magnitude of instantaneous gaze velocity for samples corresponding to fixations. Bottom, Gaze instantaneous velocity first and second decile values for each CC and NC participant. Download Figure 3-6, EPS file.
Figure 3-7
Correlation across images between entropy values and AUC values. a–c, SC (a), NC (b), and DC groups (c). Download Figure 3-7, EPS file.
Importantly, the entropy values of the CC group for individual images were significantly correlated with entropy values of the same images for the three control groups (Pearson’s r, all > 0.5, p < 0.003; Fig. 3b). Thus, the relative extent of visual exploration of images was correlated across groups. This correlation suggests that visual exploration by CC individuals was strongly dependent on the characteristics of the images, and that this dependency was qualitatively similar to the dependency on image characteristics that guided visual exploration in control individuals.
Since stimulus entropy assesses the extent of visual exploration, but not the precise locations of gaze shifts, similar entropy values across groups do not unambiguously indicate the same visual exploration patterns. Therefore, we additionally evaluated whether the exploration patterns of the CC group were predicted by the corresponding visual exploration patterns of the SC group. For each image and participant, we used the pooled spatial distribution of gaze locations from the SC group to create an SC predictor map. We then used the latter to predict whether or not an image location was visually explored by CC individuals. Classification success was quantified by the AUC of the receiver operator characteristic (values >0.5 indicate correct prediction; see subsections Data analysis and statistics, and AUC; Swets, 1988; Fawcett, 2006).
SC predictor maps discriminated the gazed versus not-gazed locations above chance of the CC group (all groups AUC values, >0.5; one-sample t tests, p < 0.001; Fig. 3c). To exclude the possibility that common image characteristics or spatial biases artificially enhanced prediction success, we ran a control analysis in which images were shuffled and AUC values from arbitrarily assigned images were derived. No successful prediction was achieved with these values (none of the AUC values in either group differed from 0.5, p > 0.05). Although CC participants’ AUC values were overall lower than those for SC and DC participants (robust linear model contrasts, both contrasts, p < 0.001; Extended Data Fig. 3-2, for statistics), they did not differ from those of the NC participants (p = 0.93).
In the CC group, AUC values were not correlated with visual acuity (Pearson’s r(8) = −0.19, p = 0.59; Extended Data Fig. 3-3), age at testing (r(8) = −0.05, p = 0.87), age at cataract surgery (r(8) = −0.28, p = 0.42), or time since sight restoration (r(8) = 0.2, p = 0.56).
The previous analyses were based on the complete duration of a trial (4 s). To evaluate possible group differences in the temporal dynamics of visual exploration, we additionally ran the same analyses for the SC predictor maps separately for consecutive, nonoverlapping 500 ms time intervals. For the first interval (0–500 ms after image presentation), all groups had low AUC values (AUC values, ∼0.5; Fig. 3d). This result is consistent with previous findings and is most likely because of the starting position being forced to be at the center of the image and thus independent of image content (Schütt et al., 2019). After the first interval, CC participants’ AUC values increased, and remained at the same level throughout image presentation. By contrast, SC, DC, and NC participants reached their highest AUC values in the second interval (500–1000 ms), following which AUC values that progressively decreased until the end of image presentation. This group difference in the dynamics of visual exploration was confirmed by a mixed-effects model with a categorical predictor participant group, a time interval covariate (excluding the first interval), and participant identity as random effect: CC participants’ estimate of a time interval covariate was not different from 0 (p = 0.9; Extended Data Fig. 3-4, statistics). In other words, there was no relationship between time interval and AUC values in the CC group. By contrast, all of the other groups showed a significantly more negative estimate of the time interval covariate than CC participants (all contrasts, p < 0.006), indicating a decrease of AUC values as exploration progressed in the control groups.
Previous research suggested that visual acuity depends on the extent of the stable, “foveation,” period of the nystagmus (Dell’Osso and Daroff, 1975; Dell’Osso and Jacobs, 2002; Felius et al., 2011). Therefore, it is possible that CC and NC participants mainly explore the image during low-velocity periods of their nystagmus. To test this hypothesis in CC and NC participants, AUC values were separately calculated for 10 data partitions according to the magnitude of the gaze instantaneous velocity (Fig. 3e). AUC values were above chance for each velocity bin. A mixed-effects model with the categorical predictor group (run only with CC and NC groups), a velocity quantile covariate, and participant identity as random effect, revealed a significant effect of speed quantile (p = 0.002; Extended Data Fig. 3-5, statistics), without a significant main effect of group (p = 0.6) and without a significant interaction of group and the velocity quantile covariate (p = 0.83). Therefore, across both groups, slower gaze velocities resulted in higher AUC values. The first two gaze velocity quantiles of CC and NC individuals were approximately comparable to SC and DC individuals’ instantaneous gaze velocity during fixations (Extended Data Fig. 3-6). This result suggests that CC and NC individuals were able to systematically adjust visual exploration to gaze at the most relevant parts of an image during the low-speed phase of the nystagmus, when visual discrimination seemed to be best in individuals with nystagmus.
Entropy and AUC values were correlated in all groups, demonstrating that the lower the spread of visual exploration, the higher the agreement of visual exploration patterns across participants (Fig. 3f, results of the CC group, Extended Data Fig. 3-7, corresponding results of the other groups). This correlation did not differ between any of the four groups (comparison of Fischer’s z-transformed r values, all p > 0.05).
In summary, CC individuals gazed at similar locations of the image as normally sighted control subjects. These results support the hypothesis that the CC individuals’ visual exploration was based on the same underlying mechanisms. Thus, neither the acquisition of these representations, nor their use for visual exploration via eye movements, seem to require patterned vision at birth.
Exploration patterns of CC participants are guided by both low-level and high-level visual features
Next, we evaluated to what degree visual exploration patterns were guided by low-level versus high-level visual information. For this purpose, predictor maps from two different saliency models were computed for each image: a first, low-level predictor map was constructed from local contrast as defined by the ICF model (Kümmerer et al., 2017). The second, high-level predictor map was constructed from features resulting from a deep neural network trained for object recognition, as defined by the DG-II model (Kümmerer et al., 2016, 2017; Schütt et al., 2019).
In all groups, visual exploration patterns were classified above chance by both the low-level ICF and high-level DG-II predictor maps (Fig. 4a,b; all AUC > 0.5, p < 0.001). The high-level DG-II model predicted the visual exploration patterns better (i.e., resulted in higher AUC values) than the low-level ICF model for all groups (paired t test: CC group, p = 0.01; SC group, p < 0.001; DC group, p < 0.001) except for the NC group (p = 0.13). We confirmed that this classification accuracy was not an artifact of general image characteristics or spatial biases: neither of the two models significantly predicted gaze patterns in either group after images were shuffled. These results suggest that CC individuals were able to make use of both low-level and high-level visual information for guiding visual exploration, similar to SC individuals.
Degree of explained visual exploration behavior for low-level and high-level visual information and context. a, AUC values resulting from the low-level ICF predictor maps (Extended Data Fig. 4-1, statistics). b, AUC values resulting from the high-level DG-II predictor maps (Extended Data Fig. 4-2, statistics). c, Ratio between ICF and DG-II AUC values (Extended Data Fig. 4-3, statistics). Extended Data Figures 4-4, 4-5, and 4-6 show AUC values of the ICF and DG-II predictor map across time and the corresponding statistics. Extended Data Figure 4-7 shows the ratio between ICG and DG-II AUC values obtained from low-pass-filtered versions of the images and the AUC values obtained from the nonfiltered images (Extended Data Figs. 4-8, 4-9, statistics).
Figure 4-1
AUC (ICF predictor map) statistical result. Download Figure 4-1, DOCX file.
Figure 4-2
AUC (DG-II predictor map) statistical result. Download Figure 4-2, DOCX file.
Figure 4-3
ICF/DG-II AUC ratio statistical result. Download Figure 4-3, DOCX file.
Figure 4-4
AUC values across time. Curves show, for each group, AUC values calculated from consecutive 500 ms data partitions. a, ICF predictor. b, DG-II predictor. Extended Data Figures 4-5 and 4-6 show statistics. Download Figure 4-4, EPS file.
Figure 4-5
AUC (ICF predictor) per time interval statistical result. Download Figure 4-5, DOCX file.
Figure 4-6
AUC (DG-II predictor) per time interval statistical result. Download Figure 4-6, DOCX file.
Figure 4-7
a, b, Ratio between AUC values for saliency predictor map obtained from low-pass-filtered versions of the images and the AUC values obtained from the nonfiltered images, for the ICF (a) and DG-II (b) predictor maps. ICF and DG-II saliency models were run with low-pass-filtered versions of the images using 0.5, 1, and 2 visual degrees spatial frequency cutoff. Group differences were evaluated separately per predictor map and filtered version with robust linear model contrasts. The ratio between ICF predictor map AUC values from images filtered at 0.5° and nonfiltered images was higher in the CC group compared with the SC group (t(38) = –2.82, p = 0.007; Extended Data Fig. 4-8, statistics) and the DC group (t(38) = –3.51, p = 0.001), but was not significantly different from the NC group (t(38) = –1.06, p = 0.29). No difference between groups was found for the 1° ratio or 2° ratio AUC. The ratio between DG-II predictor map AUC values from images filtered at 0.5° and nonfiltered images was higher in the CC and NC groups compared with the SC and DC group (all four contrast, p < 0.007; Extended Data Fig. 4-9), but was not different in the comparison between CC and NC (p = 0.79). The ratio between AUC values from images filtered at 1° and nonfiltered images was higher in the CC group compared with the DC group (p = 0.005). No other group contrast was significant. No group difference was found for the DG-II 2°. In summary, compared with SC and DC participants, for CC and NC participants gazed and nongazed locations were to a larger degree predicted by the low spatial spectral content of the images. Download Figure 4-7, EPS file.
Figure 4-8
AUC (ICF predictor map low-pass filtered, 0.5) statistical result. Download Figure 4-8, DOCX file.
Figure 4-9
AUC (DG-II predictor map low pass 0.5 filtered, 0.5) statistical result. Download Figure 4-9, DOCX file.
AUC values were overall lower in the CC group compared with the SC and DC groups for both low-level and high-level predictor maps (robust linear model contrasts, all p < 0.004; Fig. 4a,b, Extended Data Figs. 4-1, 4-2, for statistics), while they did not significantly differ from the corresponding AUC values of the NC group (both p > 0.05). Importantly, the relative predictive power of the two models (ratio of the ICF and DG-II AUC values) was indistinguishable between the CC group and the three control groups (all p > 0.05; Fig. 4c, Extended Data Fig. 4-3, statistics). This confirmed that CC individuals weighted low-level and high-level visual information for guiding visual exploration similar to the SC, NC, and DC groups.
As in the analysis for the SC predictor map, we ran the analyses for the ICF and DG-II predictor maps separately for consecutive time intervals (Extended Data Fig. 4-4). For the low-level ICF predictor, the time interval covariate was not significant in any group (estimate not different from 0, all p > 0.1; Extended Data Fig. 4-5, statistics). For the high-level DG-II predictor, the SC group showed an effect of interval (p < 0.001). By contrast, this effect was nonsignificant in the CC (p = 0.29), DC (p = 0.06), and NC (p = 0.12) groups. Nevertheless, the estimate of the time interval covariate was more negative in the SC and DG groups than in the CC group (p < 0.0002 and p < 0.04, respectively; Extended Data Fig. 4-6, statistics).
Effect of stimulus repetition and object recognition performance. a, Gaze entropy for the first versus the second presentation of the same image (si), different images from the same object category (soc), and different images from different object categories (doc; Extended Data Fig. 5-1, 5-2, 5-3, statistics. b, Percentage of correct images recognized for in each group (mean group performance in black with error bars indicating the SEM; Extended Data Fig. 5-4, statistics). c, Recognition performance, visual acuity (logMar) and AUC values (obtained using SC predictor maps) for each CC individual. The blue shade mesh depicts the generalized logistic fit. Black lines starting at the red dots indicate the discrepancy between actual performance of a CC participant and model predictions (Extended Data Fig. 5-5, statistics). Extended Data Figures 5-6 and 5-7 show the relationship between performance and age at testing.
Additionally, AUC values obtained from saliency predictor maps computed from low-pass-filtered images explained visual exploration in the CC and NC groups better than in the SC and DC groups (Extended Data Figs. 4-7, 4-8, 4-9, statistics). This is consistent with CC and NC individuals’ reduced visual acuity and reduced sensitivity to higher spatial frequencies (Ellemberg et al., 1999; Bedell, 2006; Hertle and Reese, 2007). Thus, it is justified to conclude that CC and NC individuals’ visual exploration predominantly made use of the low rather than the high spatial frequency components of visual stimulus features.
In sum, these results demonstrate a highly preserved ability of CC individuals to use both low-level and high-level visual information to guide visual exploration.
Changes in visual exploration patterns for repeated images indicate visual short-term memory effects in CC individuals
Visual exploration patterns narrow down after an image has been repeatedly encountered (Noton and Stark, 1971; Ryan et al., 2000; Smith et al., 2006; Kaspar and König, 2011a,b). This result has been taken as evidence for visual exploration being guided not only by stimulus-driven factors but additionally by top-down factors. To assess such short-term memory effects, we analyzed differences in gaze entropy for two consecutive images, for which the second image was (1) identical to the first image, (2) a different image but displaying an item of the same category as the first image, or (3) a different, unrelated image.
Entropy values decreased between the first and second presentations of the same image in all groups, including the CC group (Fig. 5a). Furthermore, this reduction in the spread of visual exploration between repeated images did not differ between groups (no significant interaction between image repetition and group, p > 0.05; Extended Data Fig. 5-1, statistics). Importantly, in all groups, the reduction in entropy for consecutive images was specific for repeated images and did not generalize to category repetitions or different images (Fig. 5a, Extended Data Figs. 5-2, 5-3, statistics).
Figure 5-1
Entropy values for the first versus second image of a pair of identical images statistical result. Download Figure 5-1, DOCX file.
Figure 5-2
Entropy values for the first versus second image of a pair of images from same category statistical result. Download Figure 5-2, DOCX file.
Figure 5-3
Entropy values for the first versus second image of a pair of images from different category statistical result. Download Figure 5-3, DOCX file.
Figure 5-4
Performance for each group statistical result. Download Figure 5-4, DOCX file.
Figure 5-5
CC participants’ performance statistical result. Download Figure 5-5, DOCX file.
Figure 5-6
Performance versus age at testing. Age at testing was not correlated with performance across all participants (r = 0.06, p = 0.69) or when tested only for the CC group (r = 0.1, p = 0.78). For CC participants, a logistic regression model analysis showed no association between object recognition performance and age at testing (p = 0.17; Extended Data Fig. 5-7). Download Figure 5-6, EPS file.
Figure 5-7
CC participants’ performance and age statistical result. Download Figure 5-7, DOCX file.
In summary, CC individuals’ visual exploration patterns showed the same short-term memory-related reduction in spread as found in the control groups and demonstrated in previous research (Noton and Stark, 1971; Ryan et al., 2000; Smith et al., 2006; Kaspar and König, 2011a,b). This result suggests that CC individuals are able to integrate both stimulus-driven and top-down information from short-term memory to guide visual exploration.
Object recognition performance is linked to systematic visual exploration in CC individuals
Object recognition performance was high in all groups (Fig. 5b). All SC participants, independent of their chronological age at testing, performed at 100%. Overall, the performance of the CC group (mean, 84.2% correct; range, 30.3–100% correct) was lower than in the three other groups (p < 0.001; Extended Data Fig. 5-4, statistics). For CC participants, a logistic regression model analysis revealed that object recognition performance was associated with better visual acuity (visual acuity predictor, p < 0.001; Fig. 5c, Extended Data Fig. 5-5, statistics) but not with age at testing (Extended Data Figs. 5-6, 5-7). Crucially, object recognition was additionally related to how well the CC individuals’ exploration patterns were predicted by the exploration patterns of SC participants (AUC predictor, p < 0.001). According to Akaike information criterion and Tjur R2 model-fit metrics (Extended Data Fig. 5-5), a model with both the visual acuity and AUC predictors performed better at explaining the object recognition scores of CC individuals than a model with either predictor in isolation. While high overall object recognition performance in CC individuals is in accordance with previous findings (Ostrovsky et al., 2009; Röder et al., 2013), this result further suggests that object recognition performance in CC individuals might benefit from systematic visual exploration.
Discussion
Visual exploration of natural scenes by means of eye movements is guided by stimulus-driven mechanisms that make use of low-level and high-level visual features as well as by top-down mechanisms such as explicit goals and memory representations. The present study investigated the degree to which the development of the bottom-up and top-down mechanisms guiding systematic visual exploration of natural stimuli relies on early visual experience. Here, we tested visual exploration patterns in 10 individuals who had received delayed treatment for total dense bilateral congenital cataracts (CC group), some only in late childhood or adulthood. Participants watched close-up photographic images of different objects, plants, animals, and buildings. The visual exploration patterns of CC individuals were compared with those of a group of normally sighted control subjects (SC group), individuals treated for late-onset cataracts (DC group), and a group of individuals with pathologic nystagmus, but without a history of congenital cataracts or visual deprivation (NC group). We found remarkably preserved visual exploration behavior in the CC group, despite an absence of visual experience early in life. Indeed, CC individuals’ visual exploration patterns were successfully predicted by those of the SC group. The application of modeling approaches to identify the visual features guiding visual exploration revealed that CC individuals used both low-level and high-level visual information, and did so with a similar relative weighting as observed in the control groups. Furthermore, by analyzing the effects of short-term memory on visual exploration patterns, we demonstrated that CC individuals were able to integrate recently acquired memory representations with stimulus-driven visual information. Finally, despite the high object recognition scores of CC individuals, residual deficits were associated not only with their persistent lower visual acuity, but additionally were associated with the degree to which their visual exploration patterns resembled those of typically sighted individuals.
While most studies in sight-recovery individuals have focused on visual perceptual functions, the interaction of the visual and oculomotor system has hardly ever been investigated in this population. On one hand, this is surprising, given that visual perception crucially depends on overt exploration to align the gaze with the most relevant regions of the visual world. On the other hand, eye movements of sight-recovery individuals born with severe visual impairment or blindness are highly distorted because of a superimposing involuntary nystagmus, making them harder to assess (Abadi et al., 2006). The emergence of nystagmus in CC individuals is a direct consequence of visual deprivation within the first 8–12 weeks of life; the first 12 weeks of life are considered a sensitive period for the development of gaze stability control (Rogers et al., 1981; Gelbart et al., 1982; Lambert et al., 2006; Birch et al., 2009). We observed more irregular nystagmus in CC individuals than in NC individuals, whose nystagmus patterns of horizontal jerk movements with accelerating slow phases were characteristic of infantile nystagmus syndrome. While Abadi et al. (2006) did not directly demonstrate such irregularities in the nystagmus pattern of CC individuals, their study reported that, in accordance with our observations, more irregular nystagmus, that is, with multiplanar rather than uniplanar patterns, seems to emerge in severe cases of congenital cataracts.
To the best of our knowledge, the present study is the first demonstration that individuals with nystagmus, regardless of etiology, are able to systematically explore natural images despite nystagmus-related distortions. Previous research suggested that visual acuity in individuals with nystagmus depends on the duration of the “foveation” periods within their nystagmus (Dell’Osso and Daroff, 1975; Dell’Osso and Jacobs, 2002; Felius et al., 2011). In both the CC and NC groups, we observed that exploration was more predictable during low-velocity periods, that is during periods that, by and large, resemble foveation periods. Thus, individuals with nystagmus are capable of taking into account their idiosyncratic nystagmus pattern while exploring an image. However, it needs to be stressed that visual exploration was predictable in both the CC and the NC groups for the complete range of gaze velocities. This result is in agreement with more recent research on visual acuity during nystagmus, which indicated that visual perception is possible throughout the nystagmus cycle (Dunn et al., 2014).
While a qualitative assessment of simple ocular orienting to light is routinely performed in CC individuals during clinical examination, the presence of nystagmus has made it difficult to quantitatively study systematic eye movement behavior in this group. It was only recently that visually guided behavior was successfully assessed with eye tracking in CC individuals (Zerr et al., 2020). In this study, participants followed a salient, single visual target, which abruptly but regularly changed location. CC individuals showed intact visually guided eye movements, which were as precise and as fast as can be expected after taking their nystagmus into account. While such visually guided eye movements are likely a prerequisite for the exploration of natural scenes, they might be accounted for, to a large degree, by a simple reflexive mechanism based on luminance contrast. By contrast, real-world visual exploration is not just driven by low-level information such as luminance contrast, but additionally uses high-level features, and integrates top-down influences such as goals and prior knowledge retrieved from memory (Tatler et al., 2011; König et al., 2016; Veale et al., 2017). Since previous research has documented better recovery of low-level than high-level visual processing in CC individuals (McKyton et al., 2015; Sourav et al., 2020; Pitchaimuthu et al., 2021), we expected that visual exploration of natural images would be mostly guided by low-level visual features. Contrary to this hypothesis, CC individuals relied on high-level information, and used both low-level and high-level information in a manner similar to that of SC and DC control groups.
For all three predictor maps (SC group predictor maps, ICF and DG-II predictors), the AUC values were significantly higher than chance in predicting the gaze patterns of CC individuals. However, they were overall lower than what has often been reported in similar studies (Wilming et al., 2011; Bylinskii et al., 2016; Kümmerer et al., 2017). This might be because of the characteristics of the images and constraints of the present study. First, all images featured a single central object, which might have reinforced a visual exploration bias toward the center. Since our analysis procedure controlled for this potential central bias, it might have lowered AUC values in the present study. Second, grayscale images were presented, which attenuated features that strongly guide typical visual exploration (Onat et al., 2014). Third, our analysis was not based only on fixations, but rather considered all eye-tracking gaze samples of the complete trial, including saccades. This was necessary because of the prevailing nystagmus in the CC and NC groups, and for a uniformity of the analysis across groups. By contrast, almost all previous studies that evaluate free-viewing behavior are based on fixations excluding saccades, and often excluding the first fixation following image presentation.
Although we found overall broader and less well predicted visual exploration patterns in the CC group than in the SC and DC groups, CC participants’ visual exploration was overall comparable to visual exploration in the NC group. A difference between the CC and NC groups was, however, detected in a time interval resolved analysis: whereas in SC, DC, and NC participants exploration was more predictable at the beginning of visual exploration (500–1000 ms interval) than during later phases, CC participants showed consistent AUC values throughout the exploration period. Decreasing predictability of visual exploration has been observed in previous research (Onat et al., 2014; Schütt et al., 2019). This has been interpreted as an initial bottom-up orienting response, followed by a gradual broadening of visual exploration (Schütt et al., 2019). The initial strong bottom-up response has been shown to be a consequence of the use of high-level features, rather than primary low-level features (Onat et al., 2014; Schütt et al., 2019). Indeed, this is the pattern of visual exploration that was observed in the SC group. In contrast to the SC group, prediction accuracy driven by high-level features did not vary with time in the CC individual. Thus, we speculate that despite using high-level features for visual guidance, high-level information did not interact with the initial phase of bottom-up exploration in the CC group. Future research might confirm this observation, since the dynamic change of predictability of the high-level model was not significant in the DC and NC groups.
It is unclear to what degree CC individuals are capable of visually exploring more complex scenes (e.g., images with multiple items or images that are generally harder to perceive for them). In fact, as a recent study has reported that CC individuals conduct fewer eye movements to the eyes region (Zohary et al., 2022). In the present study, we also avoided high stimulus eccentricities because of well known deficits in the peripheral vision of CC individuals (Lewis and Maurer, 2005), which is likely enhanced by the effects of nystagmus (Chung and Bedell, 1995; Pascal and Abadi, 1995).
Stimulus-driven guidance of visual exploration is thought to emerge from topographical “feature maps” representing visual features such as color, orientation, luminance, and motion (Itti and Koch, 2001; Veale et al., 2017). It is assumed that these feature maps serve as a source for “saliency maps.” Saliency maps represent how conspicuous or “salient” different regions of the visual field might be (Koch and Ullman, 1985; Itti and Koch, 2000). Our results indicated that the emergence of both of these mechanisms—the extraction of visual feature maps as well as the computation of saliency maps—do not seem to depend on early visual experience during a sensitive period.
The computation of feature and saliency maps has been proposed to be followed by the derivation of a “priority map” (Bisley and Goldberg, 2010). Priority maps are thought to combine bottom-up stimulus-driven information and top-down constraints to select the next gaze location (Bisley and Goldberg, 2010; Veale et al., 2017). Top-down influences have often been studied by manipulating task instructions. A special case of a nonreflexive, implicit top-down influence on visual exploration is the effect of short-term memory: If an image is repeated, the distribution of gazed locations narrows down (Hannula, 2010). Short-term memory effects on visual exploration because of image repetition have been reported to be unrelated to changes in low-level visual features (Kaspar and König, 2011b), suggesting that these effects are neither because of low-level adaptation, nor because of a reweighting of low-level and high-level image features. Whether or not the CC group would show memory-based gains on visual exploration over longer delay periods, as used in previous studies (Hannula, 2010), or other task-based top-down effects might be investigated in future studies.
CC participants were able to recognize the visual stimuli, which is in agreement with previous reports showing that even after a long period of congenital blindness, sight-recovery individuals were able to correctly name everyday objects (Maurer et al., 2005; Ostrovsky et al., 2009; Röder et al., 2013) and to recognize artificial objects through temporal integration (Orlov et al., 2021). In contrast to normally sighted participants who performed at ceiling, the performance of the CC group on object recognition was not perfect in the present study. Crucially, better image recognition in CC individuals was associated not only with better visual acuity, but additionally with how much CC participants’ gaze patterns resembled those of normally sighted control subjects. Although this association must be considered preliminary because of the limited sample size in the present study, this finding is compatible with previous research. For ambiguous or noisy stimuli, visual exploration of diagnostic features precedes explicit recognition, rather than object recognition guiding exploration (Holm et al., 2008; Kietzmann et al., 2011). However, similar latencies of the N170 wave of event-related potentials, an electrophysiological component that has been associated with the structural encoding of objects, speaks in favor for a recovery of typical object recognition times in CC individuals (Röder et al., 2013). Since the overall low visual acuity in CC individuals can be considered analogous to noise, we speculate that visual exploration aided rather than interfered with object recognition in CC individuals.
The idea that visual exploration promotes object recognition is reminiscent of theories from developmental psychology on how infants learn to recognize objects. For example, information-processing accounts assume that object recognition emerges in an active interaction with the visual world (Johnson and Johnson, 2000; Johnson, 2001; Johnson et al., 2008). Object recognition advances with an improvement in active sampling, that is, in visual exploration. It has been hypothesized that newborns’ preference for edges and motion, as well as their ability for figure–ground segregation, acts as an initial guide for where to look (Slater et al., 1990; Johnson and Johnson, 2000; Johnson, 2001). Further, it was proposed that object-defining higher-level features are acquired while continuously exploring the visual world (Johnson and Johnson, 2000). For example, the level of object knowledge in 2- to 3.5-month-old infants (Johnson et al., 2004) and the ability to process facial expressions in 6- to 11-month-old infants (Amso et al., 2010) were found to depend on visual exploration patterns. Our results are consistent with the idea of active visual exploration being instrumental for the acquisition of object knowledge. We speculate that CC individuals’ postsurgery visual exploration might initially have made use of the same preferences for edges and motion as suggested for newborns (Johnson and Johnson, 2000; Johnson, 2001). This additionally requires functioning oculomotor control in CC individuals capable of taking nystagmus-related trajectories into account. As children refine visual exploration to rely more on high-level features (Açık et al., 2010; Helo et al., 2014), we assume the same for CC individuals following cataract removal surgery. Indeed, CC individuals of the present study who had acquired the most typical visual exploration patterns were those who performed the best at object recognition.
None of the measures tested (i.e., entropy, AUC, and performance) showed an association with age at testing or time since surgery in CC participants. At first glance, this result seems surprising given the large range of ages at testing and of time passed since surgery. However, the lack of such a significant association requires replication, since our sample size was limited by the availability of a rare population. Further, all CC participants were >10 years of age. Previous research has reported adult-like visual exploration in terms of entropy and AUC measures in children >7 years of age (Açık et al., 2010; Helo et al., 2014). Finally, we tested CC individuals at least 7 months postsurgery. Thus, the duration since surgery within which visual input was available might have been sufficient to acquire visual exploration strategies, and the associated object knowledge. In fact, previous research in cataract-reversal individuals who underwent a long period of visual deprivation has provided evidence that knowledge of object shape emerges within this time period (Wright et al., 1992; Ostrovsky et al., 2009; Held et al., 2011; Chen et al., 2016).
In conclusion, the remarkably preserved exploration patterns of sight-recovery individuals with a history of a transient phase of congenital patterned visual deprivation suggests that the development of visual exploration mechanisms does not depend on experience within a sensitive period. In contrast to prevailing deficits in visual acuity, gaze stability, and other high-level visual functions (Röder and Kekunnaya, 2021), visual exploration mechanisms seem to emerge after sight restoration. We speculate that similar to infants, the newly available, low spatial frequency information might initiate recovery in individuals with reversed congenital cataract; followed by refinement, as in typical ontogenetic development. Finally, it might be hypothesized that visual exploration after sight-restoration surgery might stimulate the acquisition of visual object knowledge despite visual acuity deficits and nystagmus.
Acknowledgments
Acknowledgment: We thank D. Balasubramanian, who made the research at the LV Prasad Eye Institute possible. We also thank Kabilan Pitchaimuthu and Prativa Regmi for clinical data curation, and Suddha Sourav and Rashi Pant for technical support.
Footnotes
The authors declare no competing financial interests.
This work was supported by German Research Foundation (DFG) Grants Ro 2625/10-1 and SFB936 – 178316478-B11.
This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International license, which permits unrestricted use, distribution and reproduction in any medium provided that the original work is properly attributed.
References
In this issue
