Spatiotemporal Coding in the Macaque Supplementary Eye Fields: Landmark Influence in the Target-to-Gaze Transformation

Surgical procedures and recordings of 3D gaze, eye, and head

The experimental protocols complied the guidelines of Canadian Council on Animal Care on the use of laboratory animals and were approved by the York University Animal Care Committee. Neural recordings were performed on two female Macaca mulatta monkeys (Monkeys V and L) and they were implanted with 2D and 3D sclera search coils in left eyes for eye-movement and electrophysiological recordings (Crawford et al., 1999; Klier et al., 2003). The eye coils allowed us to register 3D eye movements (i.e., gaze) and orientation (horizontal, vertical, and torsional components of eye orientation relative to space). During the experiment, two head coils (orthogonal to each other) were also mounted that allowed similar recordings of the head orientation in space. We then implanted the recording chamber centered in stereotaxic coordinates at 25 mm anterior and 0 mm lateral for both animals. A craniotomy of 19 mm (diameter) that covered the chamber base (adhered over the trephination with dental acrylic) allowed access to the right SEF. Animals were seated within a custom-made primate chair during experiments, and this allowed free head movements at the center of three mutually orthogonal magnetic fields (Crawford et al., 1999). The values acquired from 2D and 3D eye and head coils allowed us to compute other variables such as the orientation of the eye relative to the head, eye-velocity and head-velocity, and accelerations (Crawford et al., 1999).

Basic behavioral paradigm

The visual stimuli were presented on a flat screen (placed 80 cm away from the animal) using laser projections (Fig. 1A). The monkeys were trained on a standard memory-guided gaze task to remember a target relative to a visual allocentric landmark (two intersecting lines acted as an allocentric landmark) thus leading to a temporal delay between the target presentation and initiation of the eye movement. This allowed us to separately analyze visual (aligned to target) and eye-movement-related (saccade onset) responses in the SEF. To not provide any additional allocentric cues, the experiment was conducted in a dark room. In a single trial, the animal began by fixating on a red dot (placed centrally) for 500 ms while the landmark was present on the screen. This was followed by a brief flash of visual target (T, white dot) for 100 ms, and then a brief delay (300 ms), a grid-like mask (200 ms, this hides the past visual traces, and also the current and future landmark), and a second memory delay (200–600 ms, i.e., from the onset of the landmark until the go signal). As the red fixation dot extinguished, the animal was signaled to saccade head-unrestrained (indicated by the solid green arrow) toward the memorized location of the target either in the presence of a shifted (indicated by broken green arrow) landmark (90% of trials) or in absence of it (10%, no-shift/zero-shift condition, i.e., landmark was present at the same location as before mask). These trials with zero-shift were used to compute data at the “origin” of the coordinate system for the T-T’ spatial model fits. The saccade targets were flashed one-by-one randomly throughout the response field of a neuron. Note: green color highlights the items that were not presented on the screen (they are only for representational purposes).

The spatial details of the task are in Figure 1B. Figure 1B1 shows an illustration of a gaze shift (blue curve) to an example target (T) in presence of a shifted landmark (blue cross). G refers to the final gaze endpoint, and T’ stands for the virtual target (fixed to the shifted landmark). The landmark vertex could initially appear at one of four spots located 11° obliquely relative to the target and then shift in any one of eight directions (Fig. 1B2). Importantly, the timing and amplitude (8°) of this shift was fixed. Since these animals had been trained, tested behaviorally (Li et al., 2017) and then retrained for this study over a period exceeding two years, it is reasonable to expect that they may have learned to anticipate the timing and the amount of influence of the landmark shift. However, we were careful not to bias this influence: animals were rewarded with a water-drop if gaze was placed (G) within 8–12° radius around the original target (i.e., they were rewarded if they looked at T, toward or away from T’, or anywhere in between). Based on our previous behavioral result in these animals (Li et al., 2017), we expected this paradigm to cause gaze to shift partially toward the virtually shifted target in landmark coordinates (T’).

Note that this paradigm was optimized for our method for fitting spatial models to neural activity (see below in the section ‘Fitting neural response fields against spatial models’), which is based on variable dissociations between measurable parameters such as target location and effectors (gaze, eye, head), and various egocentric/allocentric reference frames (Keith et al., 2009; Sajad et al., 2015). This was optimized by providing variable landmark locations and shift directions, and the use of a large reward window to allow these shifts (and other endogenous factors) to influence gaze errors relative to T. We also jittered the initial fixation locations within a 7–12° window to dissociate gaze-centered and space-centered frames of reference (note that no correlation was observed between the initial gaze location and final gaze errors). Further dissociations between effectors and egocentric frames were provided by the animals themselves, i.e., in the naturally variable contributions of the eye and head to initial gaze position and the amplitude/direction of gaze shifts. Details of such behavior have been described in detail in our previous papers (Sadeh et al., 2015; Sajad et al., 2015).

Behavioral recordings and analysis

During experiments, we recorded the movement of eye and head orientations (in space) with a sampling rate of 1000 Hz. For the analysis of eye movement, the saccade onset (eye movement in space) was marked at the point in time when the gaze velocity exceeded 50°/s, and the gaze offset was marked as the point in time when the velocity declined below 30°/s. The head movement was marked from the saccade onset till the time point at which the head velocity declined below 15°/s.

When the landmark shifted (90% of trials), its influence on measured future gaze position (G) was called allocentric weight (AW), computed as follows: (1)where d is the component of T-G (error space between the actual target location and the final measured gaze position) that projects onto the vector direction of the landmark shift, and D is the magnitude of the landmark shift (Byrne and Crawford, 2010; Li et al., 2017). This was done for each trial, and then averaged to find the representative landmark influence on behavior in a large number of trials. A mean AW score of zero signifies no landmark influence, i.e., gaze shifts headed on average toward T. A mean AW score of 1.0 means that on average, gaze headed toward a virtual target position (T’) that remained fixed to the shifted landmark position. As we shall see (Fig. 1C), AW scores for individual trials often fell between 0 and 1 but varied considerably, possibly because of trial-to-trial variations in landmark influence and/or other sources of variable gaze error that are present without a landmark shift (Sajad et al., 2015, 2016, 2020).

Electrophysiological recordings and response field mapping

Tungsten electrodes (0.2- to 2.0-mΩ impedance, FHC Inc.) were lowered into the SEF [using Narishige (MO-90) hydraulic micromanipulator] to record extracellular activity. Then the recorded activity was digitized, amplified, filtered, and saved for offline spike sorting using template matching and applying principal component analysis on the isolated clusters with Plexon MAP System. The recorded sites of SEF (in head-restrained conditions) were further confirmed by injecting a low-threshold electrical microstimulation (50 μA) as previously used (Bruce et al., 1985). A superimposed pictorial of the recorded sites from both animals is presented in Figure 2A,B (Monkey L in blue and Monkey V in red).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

SEF recordings A, Purple inset represents the location of the SEF and the circle (note that it does not correspond to the original size of chamber) corresponds to the chamber. B, A zoomed-in overlapped section of the chamber and the sites of neural recordings (dots) confirmed with 50-μA current stimulation from Monkey V (red) and Monkey L (blue). C, Mean (±95% confidence) of the spike-density plots [dark red: all trials from all neurons; light red: top 10% best trials most likely representing the hot spot of every neuron’s response field in the visual population, aligned to the onset of the target (blue arrow)]. D, Same as C but for motor responses aligned to the saccade onset (blue arrow). The shaded region denotes the 100-ms sampling window. Note that, in both plots, the “top 10%” neural data were selected using the above sampling windows; therefore, delay activity is not completely represented in these plots (it will be examined later in Results).

We mostly searched for neurons while the monkey freely (head-unrestrained) scanned the environment. Once we noticed that a neuron had reliable spiking, the experiment started. The response field (visual and/or motor) of neuron was characterized while the animal performed the memory-guided gaze shift paradigm as described above in the section ‘Basic behavioral paradigm’. After an initial sampling period to determine the horizontal and vertical extent of the response field, targets were presented (one per trial) in a 4 × 4 to 7 × 7 array (5–10° from each other) spanning 30–80°. This allowed the mapping of visual and motor response fields such as those shown in Results. For analysis we aimed at ∼10 trials per target, so the bigger the response field (and thus the more targets), the more the number of recorded trials were required and vice versa. On average, 343 ± 166 (mean ± SD) trials/neuron were recorded, again depending on the size of the response field. We did such recordings from >200 SEF sites, often in conjunction with simultaneous FEF recordings, as reported previously (Bharmauria et al., 2020).

Data inclusion criteria, sampling window, and neuronal classification

In total, we isolated 256 SEF neurons: 102 and 154 neurons were recorded from Monkeys V and L, respectively. Of these, we only analyzed task-modulated neurons with clear visual burst and/or with perisaccadic movement response (Fig. 2C,D). Neurons that only had postsaccadic activity (activity after the saccade onset) were excluded. Moreover, neurons that lacked significant spatial tuning were also eliminated (see below, Testing for spatial tuning). In the end, after applying our exclusion criteria, we were left with 37 and 31 spatially tuned neurons in Monkeys V and L, respectively. Only those trials were included where monkeys landed their gaze within the reward acceptance window, however, we eliminated gaze end points beyond ±2° of the mean distribution from our analysis. For the analysis of the neural activity, the “visual epoch” sampling window was chosen as a fixed 100-ms window of 80–180 ms aligned to the target onset, and the “movement period” was characterized as a high-frequency perisaccadic 100-ms (−50 to +50 ms relative to saccade onset) period (Sajad et al., 2015). This allowed us to get a good signal/noise ratio for neuronal activity analysis, and most likely corresponded to the epoch in which gaze shifts were influenced by SEF activity. After rigorously employing our exclusion criteria and data analyses (Table 1), we further dissociated the spatially tuned neurons into pure visual (V; n = 6), pure motor (M; n = 26) and VM neurons (which possessed visual, maximal delay, and motor activity, n = 36) based on the common dissociation procedure (Bruce and Goldberg, 1985; Sajad et al., 2015; Schall, 2015; Khanna et al., 2019). Note: the motor population also includes neurons which have delayed memory activity (Sajad et al., 2016).

Spatial models analyzed (egocentric and allocentric)

Figure 3 graphically shows how we derived the 11 egocentric “canonical” models tested in this study and then chose the best egocentric model to compare with the allocentric models. Briefly, based on these models, this figure illustrates the formal means for comparing between target versus gaze position coding and gaze versus eye versus head displacement/position (Fig. 3A–C), with each plotted in different possible frames of reference. We then use the landmark-centered information to test between the egocentric versus allocentric modes of coding (Fig. 3D). Note that in our analysis, these models were based on actual target, gaze, eye, and head data, either derived from geometric calculations (in the case of T) or eye/head coil measurements. As described previously, the variability required to distinguish between these models was either provided by ourselves (i.e., stimulus placement) or the monkeys’ own behavior (i.e., variable gaze errors and variable combinations of eye-head position; Sajad et al., 2015).

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Description of the different spatial models (egocentric and allocentric) tested in this study. A, Physical variables in one trial. The projections of initial gaze (magenta dot), initial head (green cross), final gaze (black dot), and final head (green dot) orientations on the screen are depicted. Note: final gaze often lands between the target (T, red dot) and shifted target (T’, light red dot). B, While head-unrestrained, these positions can be encoded in relation to three egocentric frames of reference: eye (e), head (h), and space/body (s). Since the target was sufficiently far from the head to minimize translation effects, and because our model-fitting method is insensitive to these biases, we centered these frames to be aligned as shown in C. C, Different canonical models derived after plotting the physical parameters in fundamental egocentric frames. dG: gaze displacement referring to final gaze position with respect to the home fixation point (not the eye); dH: displacement of head in space coordinates; dE (data not shown): the displacement of eye in the head accompanying the gaze shift. Target and final gaze, eye, and head positions can be represented in any one of the three initial reference frames. See text for details of all egocentric models. D, Allocentric models that were compared with the most relevant egocentric models (Te, Ge). The broken black cross represents the location of initial landmark (L) and the solid black cross stands for the shifted (indicated by orange arrow) landmark (L’). The red, the black, and the light red solid circles correspond to the target (T), the gaze endpoint (G), and the virtual target (T’) locations, respectively. Tested allocentric models: Ls, landmark relative to space; Le, landmark relative to eye; TL, target relative to landmark; L’s, shifted landmark relative to space; L’e, shifted landmark relative to eye, TL’, target relative to shifted landmark, T’e, shifted target relative to eye; T’s, shifted target relative to space.

We first tested for all the egocentric models as reported in our previous studies (Sajad et al., 2015, 2016) and then tested the best egocentric model with all “pure” allocentric models. Based on different spatial parameters [most importantly target (T) and final gaze position (G)], we fitted visual and motor response fields of SEF against previously tested 11 (Fig. 3A–C) egocentric canonical models in FEF (Sajad et al., 2015). Note that Te and Ge codes were obtained by mathematically rotating the experimental measures of T and G in space coordinates by the inverse of experimentally measured initial eye orientation to obtain the measures in eye coordinates (Klier et al., 2001). In other words, Te is based on the actual target location relative to initial eye orientation, whereas Ge is based on the final gaze position relative to initial eye orientation.

The following models were also tested: dH, difference between the initial and final head orientation in space coordinates; dE: difference between the initial and final eye orientation in head coordinates; dG, difference between the initial and final gaze position in space coordinates; Hs, final position of the head in space coordinates; Eh, final position of the eye in head coordinates; Gs, Ge, Gh: gaze in space, eye, and coordinates, respectively; Ts, Te, Th: target in space, eye, and head coordinates, respectively. The final position refers to orientation of eye/head after the gaze saccade. These models are detailed in the previous study (Sajad et al., 2015). Note that some of these models (like dG and Ge) might not be distinguishable within the range of initial gaze positions and saccade amplitudes described here (Crawford and Guitton, 1997).

Since an allocentric landmark was involved in this task, we analyzed eight additional (allocentric) models of target coding based on the original and shifted landmark location (Fig. 3D). In the allocentric analysis, we retained the best egocentric model (Te for visual neurons and Ge for motor neurons) for comparison. The tested allocentric models are: Ls, landmark within the space coordinates; L’s, shifted landmark within the space coordinates; Le, landmark within eye coordinates; L’e, shifted landmark within eye coordinates; T’s, shifted target in space coordinates; T’e, shifted target in eye coordinates; TL, target relative to landmark; TL’, target relative to shifted landmark. Note: prime (’) stands for the positions that are related to the shifted location of the landmark.

Intermediate spatial models used in main analysis

Previous reports on FEF responses from our lab have reported that responses do not fit exactly against spatial models like Te or Ge, but actually may fit best against intermediate models between the canonical ones (Fig. 4A, lower left; Sajad et al., 2020). As in our previous studies (Sajad et al., 2016; Sadeh et al., 2020), we found that a T-G continuum (specifically, steps along the “error line” between Te and Ge) best quantified the SEF egocentric transformation (Fig. 4A, lower left). This continuum is similar to the concept of an intermediate frame of reference (e.g., between the eye and head) but is instead intermediate between target and gaze position within the same frame of reference. Hereafter, we will sometimes refer to T-G as “the egocentric code.”

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

Schematic representation of spatial parameters and spatial model fitting technique. A, An illustration of different spatial parameters in a single trial. Black dot represents the projections of initial fixation/gaze and the blue dot corresponds to final gaze (G). The red dot depicts the location of target (T) in relation to the original landmark (L) position (broken intersecting lines), and the green dot represents the virtually shifted target (T’) fixed to the shifted landmark (L’) location (solid intersecting lines). Note that the final gaze was placed between the T and T’ (i.e., it shifted toward the shifted landmark). In head-unrestrained conditions, the target can be encoded in egocentric coordinates (eye, head, or body; Sajad et al., 2015). We plotted two continuums: a T-G continuum (egocentric, we divided the space between T and G into 10 equal steps, thus treating these spatial codes as a continuous spatiotemporal variable) to compute the gaze influence and an allocentric shift T-T’ continuum (original target to virtual target) based on the landmark shift to compute the influence of landmark shift on the neuronal activity. B, A logical schematic of response field analysis; x-axis depicts the coordinate frame, and the y-axis corresponds to the related activity to the target. Simply, if the activity related to a fixed target is plotted in the correct reference frame, this will lead to the lowest residuals, i.e., if the neural activity to a fixed target location is fixed (left), then the data (blue) would fit (pink) better on that, leading to lower residuals in comparison with when the activity for the target is plotted in an incorrect frame, thus leading to higher residuals (right).

Further, to quantify the influence of the landmark shift, we created (Fig. 4A, lower right) another continuum (T-T’, the line between the original target and the target if it were fixed to the landmark) between Te (target fixed in eye coordinates) and T’e (virtual target fixed in landmark coordinates), computed in eye coordinates with 10 intermediary steps between, and additional steps on the flanks (10 beyond Te and 10 beyond T’e). These additional 10 steps beyond the canonical models were included (1) to quantify if neurons can carry abstract spatial codes around the canonical models and (2) to eliminate the misleading edge effects (else the best spatial models would cluster around the canonical models). The T-T’ continuum will allow us to test whether SEF code is purely egocentric (based on T), allocentric (based on T’), or contains the integrated allocentric + egocentric information that eventually actuates the behavior. Note that AW and T-T’ are geometrically similar, but the first describes behavioral data, whereas the second describes neural data. Hereafter, we will refer to T-T’ as the “allocentric shift.”

Fitting neural response fields against spatial models

In order to test different spatial models, they should be spatially separable (Keith et al., 2009; Sajad et al., 2015). The natural variation in monkeys’ behavior allowed this spatial separation (for details, see Results). For instance, the variability induced by memory-guided gaze shifts permitted us to discriminate target coding from the gaze coding; the initial eye and head locations allowed us to distinguish between different egocentric reference frames and variable eye and head movements for a gaze shift permitted the separation of different effectors. As opposed to the decoding methods that generally test whether a spatial characteristic is implicitly coded in patterns of population activity (Bremmer et al., 2016; Brandman et al., 2017), the method employed in this study method directly examines which model best predicts the activity in spatially tuned neural responses. The logic of our response field fitting in different reference frames is schematized in Figure 4B. Precisely, if the response field data are plotted in the correct best reference frame, this will yield the lowest residuals (errors between the fit and data points) compared with other models, i.e., if a fit computed to its response field matches the data, then this will yield low residuals (Fig. 4B, left). On the other hand, if the fit does not match the data well, this will lead to higher residuals (Fig. 4B, right). For example, if an eye-fixed response field is computed in eye-coordinates this will lead to lower residuals and if it is plotted in any other inferior/incorrect coordinate, such as space coordinate, this will produce higher residuals (Sajad et al., 2015).

In reality, we employed a non-parametric fitting method to characterize the neural activity with reference to a spatial location and also varied the spatial bandwidth of the fit to plot any response field size, shape, or contour (Keith et al., 2009). We tested between various spatial models using predicted residual error some of squares (PRESS)-statistics. To independently compute the residual for a trial, the actual activity related to it was subtracted from the corresponding point on the fit computed over all the other trials (like cross-validation). Notably, if the spatial physical shift between two models results in a systematic shift (direction and amount), this will appear as a shifted response field or expanded response field and our model fitting approach would not be able to distinguish two models as they would produce virtually indistinguishable residuals. Because in our study, the distribution of relative positions across different models also possesses a non-systematic variable component (e.g., variability in gaze endpoint errors, or unpredictable landmark shifts), the response fields invariably stayed at the same location, but the dissociation between spatial models was based on the residual analysis.

We plotted response fields (visual and movement) of neurons in the coordinates of all the canonical (and intermediate) models. To map the visual response in egocentric coordinates, we took eye and head orientations at the time of target presentation, and for movement response fields, we used behavioral measurements at the time when the gaze shift started (Keith et al., 2009; DeSouza et al., 2011; Sajad et al., 2015). Likewise, for the allocentric models, we used the initial and the shifted landmark location to plot our data. Since we did not know the size and shape of a response field a priori and since the spatial distribution of data were different for every spatial model (e.g., the models would have a smaller range for head than the eye models), we computed the non-parametric fits with different kernel bandwidth (2–25°), thus making sure that we did not bias the fits toward a particular size and spatial distribution. For all the tested models with different non-parametric fits, we computed the PRESS residuals to reveal the best model for the neural activity (that yielded the least PRESS residuals). We then statistically (Keith et al., 2009) compared the mean PRESS residuals of the best model with the mean PRESS residuals of other models at the same kernel bandwidth (two-tailed Brown–Forsythe test). Finally, we performed the same statistical analysis (Brown–Forsythe) at the population level (Keith et al., 2009; DeSouza et al., 2011). For the models in intermediate continua, a similar procedure was used to compute the best fits.

Testing for spatial tuning

The above-described method in the section ‘Fitting neural response fields against spatial models’ assumes that neuronal activity is structured as spatially tuned response fields. This does not imply that other neurons do not subserve the overall population code (Goris et al., 2014; Bharmauria et al., 2016; Leavitt et al., 2017; Chaplin et al., 2018; Zylberberg, 2018; Pruszynski and Zylberberg, 2019), but with our method, only tuned neurons can be explicitly tested. We tested for the neuronal spatial tuning as follows. We randomly (100 times to obtain random 100 response fields) shuffled the firing rate data points across the position data that we obtained from the best model. The mean PRESS residual distribution (PRESS_random) of the 100 randomly generated response fields was then statistically compared with the mean PRESS residual (PRESS_best-fit) distribution of the best-fit model (unshuffled, original data). If the best-fit mean PRESS fell outside of the 95% confidence interval of the distribution of the shuffled mean PRESS, then the neuron’s activity was deemed spatially selective. At the population level, some neurons displayed spatial tuning at certain time-steps and others did not because of low signal/noise ratio. Thus, we removed the time steps where the populational mean spatial coherence (goodness of fit) was statistically indiscriminable from the baseline (before target onset) because there was no task-related information at this time and thus neural activity exhibited no spatial tuning. We defined an index (coherence index; CI) for spatial tuning. CI for a single neuron, which was calculated as (Sajad et al., 2016): (2)

If the PRESS_best-fit was similar to PRESS_random then the CI would be roughly 0, whereas if the best-fit model is a perfect fit (i.e., PRESS_best-fit = 0), then the CI would be 1. We only included those neurons in our analysis that showed significant spatial tuning.

Time normalization for spatiotemporal analysis

A major aim of this study was to track the progression of the T-G and T-T’ codes in spatially tuned neuron populations, from the landmark-shift until the saccade onset. The challenge here was that this period was variable, so that averaging data across trials for each neuron would result in the mixing of very different signals (see Extended Data Fig. 11-1). To compensate for this, we time-normalized this data (Sajad et al., 2016; Bharmauria et al., 2020). To do this, the neural firing rate (in spikes per second; the number of spikes divided by the sampling interval for each trial) was sampled at seven half-overlapping time windows (with a range of 80.5–180.5 ms depending on trial duration). The rationale behind the bin number choice was to make sure that the sampling time window was wide enough, and therefore robust enough to account for the stochastic nature of spiking activity (thus ensuring that there were sufficient neuronal spikes in the sampling window to do effective spatial analysis; Sajad et al., 2016). Once the firing rate was estimated for each trial at a given time-step, they were pooled together for the spatial modeling. Note that the final (seventh) time step also contained some part of the perisaccadic sampling window. Finally, we performed our T-G/T-T’ fits on the data for each of these time bins. In short, this procedure allowed us to treat the whole sequence of memory-motor responses from the landmark shift until saccade onset as a continuum.