Abstract
Segmentation of continuous experience into discrete events is driven by rapid fluctuations in encoding stability at context shifts (i.e., event boundaries), yet the mechanisms underlying the online formation of event memories are poorly understood. We investigated the neural per-time point spatial similarity patterns of the scalp electrophysiological (EEG) activity of 30 human participants (male and female) watching a 50 min movie and found that event boundaries triggered the rapid reinstatement of the just-encoded movie event EEG patterns. We also found that the onset of memory reinstatement at boundary onset was accompanied by a left-lateralized anterior negative ERP effect, which likely reflects the detection of a shift in the narrative structure of the movie. A data-driven approach based on Hidden Markov modeling allowed us to detect event boundaries as shifts between stable patterns of brain EEG activity during encoding, and to identify their reactivation during a free recall task. These results provide the first neurophysiological underpinnings for how the memory systems segment a continuous long stream of experience into episodic events.
SIGNIFICANCE OF STATEMENT Memory for specific episodic events are the building blocks of our autobiographical memory. However, it is still unclear how the memory systems structure the unfolding experience into discrete event units that can be understood and remembered at the long-term. Here, we show that the detection of context shifts, or event boundaries, during a 50 min movie viewing triggers the rapid memory reactivation of the just-encoded event to promote its successful encoding into long-term memory. By finding that memory reactivation, a neural mechanism critical for episodic memory formation and consolidation, takes place under these ecologically valid experimental circumstances, our results provide valuable insights into how the brain shapes the ongoing experience into episodic memories in the real-life.
Introduction
Memory systems transform the stream of our continuous experience into a sequence of episodic memory units to be recalled in the future. While extensive research has been conducted to understand how the brain supports the formation of discrete, brief novel information, it is only recently that psychologists and neuroscientists have started exploring the mechanisms that account for episodic memory formation during a continuous stream of experience.
A widely accepted view is that we naturally segment continuous experience into events, and that event boundaries are driven by moments in time when prediction of the immediate future fails (Zacks et al., 2007) or by fluctuations in contextual stability (Clewett and Davachi, 2017). Segmentation affects not only our perception of the experience, but its subsequent organization in long-term memory (Kurby and Zacks, 2008; Radvansky, 2012; Sargent et al., 2013), such that elements within an event are bound together more cohesively than elements across events (Ezzyat and Davachi, 2011; DuBrow and Davachi, 2013, 2014; Horner et al., 2016). Human neuroimaging studies using naturalistic video clips have set important findings that align well with these behavioral findings. They have shown that a distributed network of brain regions comprising the hippocampus and neocortex are involved during event segmentation and that their dynamics during encoding provide a basis for how we parse the temporally evolving environment into meaningful units. They revealed that the brain organizes the ongoing input into episodic events by detecting changes in the stability of activity patterns. Stable patterns of activity at higher-level brain regions during encoding are thought to maintain a stable event representation despite fluctuations in the ongoing sensory input (Baldassano et al., 2017; Chen et al., 2017). Shifts in stability that coincide with perceived boundaries induce a neural response at the hippocampus (Ben-Yakov and Dudai, 2011; Ben-Yakov et al., 2013; Ben-Yakov and Henson, 2018; for similar findings in rodents, see Bulkin et al., 2018) and the degree to which hippocampal activity at boundaries couples with cortical patterns of activity predicts pattern reinstatement during later free recall (Baldassano et al., 2017), thereby indicating that the hippocampus may be responsible for binding cortical representations into a memory trace online during encoding (McClelland et al., 1995; Norman and O'Reilly, 2003; Moscovitch et al., 2005). However, an important question remains unanswered: which neural mechanisms support the binding of the encoded information of an event upon boundary detection? And more importantly, how can we investigate these neural mechanisms in ecologically valid circumstances that can inform us about their nature in real life environments?
To address this issue, we recorded scalp brain EEG while 30 participants watched a single 50 min movie clip, and asked whether time-resolved fluctuations in neural similarity elicited during movie viewing reflected event segmentation. Leveraging the fine-grained temporal resolution of the EEG signal, we then tested the hypothesis that moments in time after perceived, event boundaries during movie viewing would exhibit reactivation of the just-encoded episode, and that this reactivation would promote consolidation of the encoded event into long-term memory. Indeed, the reactivation of encoded episodes upon boundary detection would be in line with animal research using EEG recordings showing that memory replay of the just-encoded event promoted its memory formation and consolidation (Carr et al., 2011) and with recent EEG research in humans that showed that memory reactivation at picture boundaries during sequence encoding promoted a linked memory representation across events (Sols et al., 2017). The extent to which boundary-triggered memory reactivation impacted memory formation during movie viewing would offer valuable insights into how the brain shapes the unfolding experience into memory under ecologically valid situations.
Materials and Methods
Participant sample.
Thirty-three Spanish speakers (30 right-handed, 20 females, age range 18–46 years, mean = 22 years) participated for pay (10€/h). Participants were recruited from the University of Barcelona and the broader community. All participants were healthy and did not consume psychoactive substances. Informed consent was obtained from participants in accordance with procedures approved by the Ethics Committee of the University of Barcelona. Data from 2 participants were discarded due to falling asleep during the experiment, and 1 due to too much muscular artifact in the data. Thus, the final sample of participants included in the study was 30.
Experimental design.
Our primary dataset consisted of 30 participants who watched the first 50 min of the first episode of BBC's Sherlock, a stimulus already used by Chen et al. (2017) and Baldassano et al. (2017), dubbed in Spanish. They were then asked to freely recall the episode without cues while being recorded using an audio recorder. The audio files were later analyzed to access participants' length and richness of the recall, with total recall times ranging from 6 to 44 min (and a mean of 15 min). At the beginning of the movie, a 30 s audiovisual clip (“Let's All Go to the Lobby”) was presented to set participants' attention. The experimental design was implemented on ePrime 2.0 (Psychology Software Tools).
EEG recording and preprocessing.
EEG was recorded using a 32-channel system at a sampling rate of 500 Hz, using a BrainAmp amplifier and tin electrodes mounted in an electrocap (Electro-Cap International) located at 29 standard positions (Fp1/2, Fz, F7/8, F3/4, Fc1/2, Fc5/6, Cz, C3/4, T3/4, Cp1/2, Cp5/6, Pz, P3/4, T5/6, PO1/2, O1/2) and at the left and right mastoids. An electrode placed at the lateral outer canthus of the right eye served as an online reference. EEG was rereferenced offline to the right and left mastoids. Vertical eye movements were monitored with an electrode at the infraorbital ridge of the right eye, and an independent component analysis was run on MATLAB's EEGLAB toolbox (Delorme and Makeig, 2004) to correct for eye movements and remove extremely noisy components (no more than 6 components were removed). A low pass filter of 20 Hz was applied to reduce the presence of muscular artifacts (Pérez et al., 2017). The data were then normalized before every analysis by z scoring all time points so that the mean of every electrode is 0 across time.
Event boundary annotations by human observers.
Six external participants, who did not take part in the electrophysiological recording session of the study, were asked to watch the movie. Using the standard event segmentation approach (Newtson, 1973; Zacks et al., 2010; also used in Baldassano et al., 2017; Chen et al., 2017; Ben-Yakov and Henson, 2018), participants were requested to annotate with precision the temporal point at which they felt “a new scene is starting; these are points in the movie when there is a major change in topic, location or time.” Participants were also informed that each event should be between 10 s and 3 min long, and we asked them to write down a short title for the event.
With the participants' boundary annotations, we looked for those boundary time points that were consistent across observers. To find a statistical threshold of how many observers should coincide in a given time point to be different from chance in our data, we shuffled the number of observations 1000 times and created a null distribution of the resulting coincident time points. An α = 0.05 as a cutoff for significance indicated that boundary time points at which at least 3 observers coincided in (considering 3 s as possible window of coincidence as in Baldassano et al., 2017) could not be explained by chance. This approach resulted in a model composed by 38 events (minimum = 4 s, maximum = 444 s, and mean = 76.02 s). The model was compared with the one obtained in Baldassano et al. (2017). To do so, the distances between the boundaries in this and the previous study were calculated. A null distribution was then created by shuffling in time the previous study boundaries while maintaining the length of the events, and the distances between the boundaries in the current study and the new shuffled ones were computed. The average real distances were then compared with the null distribution to compute a z value, which was converted to a p value. This analysis resulted in a p = 0.12, which indicates that the human annotation boundaries used in the current study are not statistically different from the ones used in the previous study with another sample of participants.
Verbal recall analysis.
After 15 min of rest, all 30 participants who participated in the EEG encoding session were asked to retell the story they had just watched, without any cues or stimulus. EEG was also collected during this time, and verbal recall was recorded through an audio recorder for later analysis. The audio files recorded during the free verbal recall were analyzed by a laboratory member who was a native Spanish speaker, using the list of events from the human annotations model. An event was counted as recalled if the participant described any part of the event and were counted as out of order if they were initially skipped and later described in the narrative or if an event was recalled earlier than it should. However, the latter case rarely happened (on average, <0.1%).
Finding event structure in the EEG data.
To validate the event segmentation model extracted from human annotations on the EEG data collected from the primary sample in the study, we generated, for each of the individuals, a temporal correlation matrix computed by correlating the 29 electrodes with the same 29 electrodes for each of the time points after downsampling the data by segmenting the EEG into bins of averaged data from 100 sample points (i.e., 200 ms). We chose this interval as a compromise between preserving data structure and reducing computational time in the analyses. Next, we averaged the correlation values within each of the 38 events and ran a permutation test (N = 1000) with null boundaries picked by shuffling the temporal order of the events while maintaining their lengths. The within-event correlation values were compared with the permuted values using an α of 0.05 as a cutoff for significance (see Fig. 2b).
Shared event neural patterns across individuals during movie viewing.
Following previous fMRI findings (Chen et al., 2017), we examined whether neural patterns elicited by events during movie viewing were similar across individuals in our sample. To address this issue, we computed Movie-Movie correlations by comparing movie patterns of each event from one participant with the movie pattern for the same event averaged across the remaining participants. This resulted in an across-participants similarity analysis. To assess whether the correlation values were statistically significant, a permutation test (N = 1000) was computed, using an α of 0.05 as a cutoff for significance, by shuffling the event patterns of the hold-out and compare them with the intact order for the remaining participants.
EEG pattern similarity within and across events.
A similarity analysis was calculated for EEG neural activity before and after boundaries during movie viewing. The similarity analysis was performed at individual level and included spatial (i.e., scalp voltages from all the 29 electrodes) and the temporal features, which were selected in steps of 20 sample points (40 ms) of the resulting z-transformed EEG single trials. The similarity analysis was calculated using Pearson correlation coefficients, which are insensitive to the absolute amplitude and variance of the EEG response.
The similarity analysis was computed on trial-to-trial EEG segments of 10 s preboundaries and postboundaries identified in the event segmentation model. To ensure that differences between before and after the boundary were not arising just due to intrinsic temporal contiguity properties of the EEG signal, we first split preboundary and postboundary 10 s EEG segments in two equal EEG vectors of 5 s. Thus, preboundary event correlations were performed between the interval −10 s to −5 s and the interval −5 to 0 s before the boundary. Between-event correlations were performed between −5 s to 0 and 0 to 5 s, were 0 corresponded to the boundary. Postboundary event correlations were performed on EEG data from the interval 0 to 5 s and the interval 5 to 10 s after the boundary. Point-to-point correlation values were then calculated, and the resulting single trials 2D correlation matrix was smoothed using a mean filter over a rectangle of 10 sample points (i.e., 400 ms). Smoothed single-trial correlation matrices were then averaged for each individual and condition separately. Differences were statistically compared by means of a repeated-measures ANOVA, including type of event as a 3-level factor (preboundary, between-event, and postboundary). Statistical significance was set at p < 0.05. A paired-sample t test was used to test for statistical significance between condition pairs.
An EEG similarity analysis was also performed on 20 s windows of EEG as a function of whether events preceding a boundary were later recalled or forgotten. The 10 s EEG signal included 20 s surrounding a boundary. Similarity analysis was implemented at single-trial level by correlating point-to-point the spatial EEG features surrounding the boundary. As in the previous analysis, single-trial 2D correlation matrices were smoothed using similar parameters before they were averaged. To account for differences between recalled and forgotten conditions, a cluster-based permutation test was used (Maris and Oostenveld, 2007), which identifies clusters of significant points in the resulting 2D matrix in a data-driven manner and addresses the multiple-comparison problem by using a nonparametric statistical method based on cluster-level randomization testing to control the family-wise error rate. Statistics were computed for every time point, and the time points whose statistical values were larger than a threshold (p < 0.05, two-tail) were selected and clustered into connected sets on the basis of x,y adjacency in the 2D matrix. The observed cluster-level statistics were calculated by taking the sum of the statistical values within a cluster. Then, condition labels were permuted 1000 times to simulate the null hypothesis, and the maximum cluster statistic was chosen to construct a distribution of the cluster-level statistics under the null hypothesis. The nonparametric statistical test was obtained by calculating the proportion of randomized test statistics that exceeded the observed cluster-level statistics.
EEG-evoked responses at boundary onset.
Event-related potentials (ERPs) at boundary onset were calculated for each individual as a function of whether events were later recalled or forgotten in the free recall task. To obtain the ERPs, we applied a low-pass filter of 12 Hz to the none-downsampled EEG data. Then, epochs of −1000 to 2000 ms were chosen around each of the boundary time points determined from the event segmentation model. The preboundary interval (−100 to 0 ms) was used for baseline correction. ERP differences between recalled and forgotten conditions were investigated starting at 0 to 2000 ms after each boundary onset. Statistical significance of the differences between conditions was assessed by a cluster-based permutation test.
Brain sources of ERPs.
Low-resolution tomography analysis (sLORETA) (Pascual-Marqui, 2002) was used to reconstruct the source space for ERP differences at boundary onset. This method performs localization inference based on images of standardized current density, which corresponds to the 3D distribution of electric neuronal activity that has maximum similarity (i.e., maximum synchronization), in terms of orientation and strength, between neighboring neuronal populations (represented by adjacent voxels). sLORETA was calculated separately for each participant's averaged ERP triggered by boundaries that followed events that were later recalled and forgotten. Source reconstruction for each condition was compared, and results were displayed by means of t values (paired t test, one-tailed).
Reinstatement of EEG event patterns during free recall.
We adapted the Hidden Markov modeling (HMM) approach used in a recent fMRI study by Baldassano et al. (2017) and tested the extent to which it identified EEG-based latent-states during recall according to the event segmentation model constructed through human annotations during movie watching.
The model is a variant of an HMM in which the latent variables are the event labels st for each time point t and the spatial signatures mk (brain activity patterns across all EEG channels) for each event k. From the observed brain activities bt, we infer both st and mk. The model is set to assume that, for all the participants, the event starts in s1 = 1 and ends with sT = K, where T is the total number of time points and K is the total number of events. We assume that, in each time point, we can either advance to the next state or remain in the same one, which results in a transition matrix where all elements are zero, except for the diagonal and the adjacent off-diagonal.
An isotropic Gaussian model is used to compute the observation model so that the probability that a given observation, bt, is created by a state st = k can be given by the following: where z() denotes the z-score function. The z-scoring of the brain observations and the mean activity patterns result in a proportionality between the log probability of observing brain state bt in an event with signature mk and the Pearson correlation between bt and mk plus a constant offset as follows: To ensure that all states are visited, the observation probabilities P(bt|st = k) are modified by setting P(bT|sT = k) = 0, for all k ≠ K so that, on the final time point, only the final state K could have generated the data. To ensure that all possible event segmentations have the same prior probability, a dummy absorbing state K + 1 is created, so that the transition probabilities for state K are identical to those for previous states. We set P(bt|st = K + 1) = 0 so that this state cannot actually be used.
The data were downsampled by segmenting the EEG into bins of averaged data from 100 sample points (i.e., 200 ms) to reduce the computational time in the analyses. We used the mean EEG patterns of each of the events identified by the human annotations model on encoding to model the EEG data during recall. This was accomplished by using the mean patterns from the human annotations events as input on the forward–backward step. For each participant, the HMM was applied to the continuous recall EEG data to obtain a probabilistic assignment of latent event states consistent with the human annotations model obtained during movie watching. The resulting model probabilities P(st = k) were then used to identify the event transition points during recall, as time points when the most likely event changed. We then tested the extent to which human annotation-based EEG patterns elicited during movie watching were similar to those estimated by HMM search during recall. As in the previous analysis, we ran an event-to-event correlation analysis between movie and recall and calculated an averaged correlation measure, as a proxy of the overall degree of similarity over the entire event segmentation model between the two sets of data. To assess statistical significance, we created a null distribution by shuffling the lengths of the human annotation events before running the HMM and computing the correlation between the movie events and the shuffled HMM-identified recall events. This procedure was applied 1000 times, and an α of 0.05 was used as a cutoff. If the forward–backward step correctly identifies events similar to encoding, we expect to find higher correlation values than the ones obtained by trying to identify the same patterns but shuffled in time. In case the correlation values obtained are similar to the ones obtained by using the null distribution, one can infer that the patterns identified by the model were identified just by chance, and were just a poor match done by the model.
EOG artifacts as potential confounds.
To statistically assess whether the existence of potential eye movement artifacts was systematically associated with event boundaries and, therefore, a potential confounder on our EEG results, we calculated the time points where the amplitude of the EOG channel exceeded 3 SD and selected those whose absolute distance from the nearest boundaries was <10 s around the event boundaries time points. This value was then compared with a null distribution where the real EOG above-threshold time points were shuffled while maintaining the distance lengths between them (N = 1000). The real distances were compared with the null distribution by computing a z value, which was then converted to a p value using the normal distribution. A cutoff of α equal to 0.1 was used to assess statistical significance.
Results
Event segmentation and perceived event boundaries
Temporal points at which at least three external raters (of six) coincided in annotating a boundary were taken as indicative of a “real” event boundary in the movie. This approach resulted in an event segmentation model of 38 event episodes (Fig. 1a), which was consistent with the range and the time shifts of the boundaries found in our previous study (Baldassano et al., 2017).
Movie free recall
Participants' memory accuracy indicated that they were able to recall 51.4% of the encoded events on average (SD = 9.2%) (Fig. 1b). Importantly, we also found that the temporal order of the episodic events at encoding was preserved at recall (Fig. 1c). To statistically assess whether the order of events during movie watching was preserved during free recall, we computed Kendall rank correlation coefficients between each individual event temporal order and a simulated correct linear order. For all participants, the Kendall tau coefficient was positive and close to 1, indicating that the encoded temporal order of the events was highly preserved during their recall, thereby replicating previous results (Chen et al., 2017) that free recall tends to preserve the temporal structure of the encoded memories.
Event segmentation model on the EEG data
Next, we tested whether patterns of EEG activity elicited by the 50 min movie exhibited the event structure hypothesized by our model (periods with stable event patterns punctuated by shifts between events). To address this issue at individual level, we computed a per time point spatial similarity analysis of the EEG data during the 50 min movie watching and calculated the degree of similarity values within each of the events (defined by the human-annotated event boundaries) (Fig. 2a). To statistically assess the extent to which the EEG data fit the model, we averaged the similarity values within each of the 38 events and tested this value against a null distribution generated by running the same analysis 1000 times with a shuffled temporal order distribution of the events (Fig. 2b). This analysis revealed that 22 of the 30 participants in our sample showed a higher degree of similarity values within events from the real segmentation model compared with their individual correlation value cutoff (α = 0.05) from their null distribution (Fig. 2c), and that this was significant at group level (p < 0.05; Fig. 2d).
Shared event neural patterns across participants during movie watching
Having shown that EEG patterns of neural activity were structured according to a general event segmentation model during movie watching, we then tested the prediction that event EEG patterns should be shared across individuals (Chen et al., 2017). To address this question empirically, we computed Movie-Movie correlations by comparing patterns of each event from one participant with the movie pattern for the same event averaged across the remaining participants. An averaged correlation value was obtained for each participant, and its statistical significance was assessed by comparing it with a random distribution obtained by shuffling the event order on the left-out participant. Confirming previous findings on fMRI data (Chen et al., 2017), we found that almost all of the participants (29 of the 30) showed a high degree of shared similarity EEG patterns with the group sample (p < 0.05 at group level) (Fig. 3a,b). These results reveal not only that participants share brain patterns while being presented with the same stimuli but that their activity patterns evolve similarly over time given that they follow the same annotation structure. This analysis does not specifically test whether the annotated events have a coherent neural pattern (e.g., high within-event pattern similarity), which is addressed in the analysis below.
EEG pattern similarity within and across events separated by boundaries
An important assumption derived from event segmentation theory is that patterns of neural activity elicited within an event should be more stable than neural patterns across events, thereby indicating that event neural representations change when boundaries are detected. To test this prediction in our data, we ran a point-to-point spatial similarity analysis throughout EEG segments of −10 to 10 s of averaged EEG trials around the boundary time point. The long EEG segments were then split into EEG epochs of 5 s each, thereby allowing us to examine the extent to which similarity values were higher for neural responses within events. More concretely, the similarity analysis was performed between three different pairs of temporal intervals in the data: preboundary time intervals (−10 to −5 s and −5 to 0 s to the boundary), between-event time intervals (−5 to 0 and 0 to 5 s to the boundary), and postboundary time intervals (0 to 5 s and 5 to 10 s to the boundary) (Fig. 4a). The resulting similarity values for each condition and subject were then averaged, and differences were tested by means of a repeated-measures ANOVA. Notably, the results from this analysis revealed that similarity values differed between conditions (F(2,58) = 14.43, p < 0.001). Post hoc paired t test showed that within-event similarity, both before and after boundary, was higher than between-event similarity (t(29) = 12.49, p < 0.001 and t(29) = 10.86, p < 0.001, respectively) and that similarity values within preboundary and postboundary conditions were statistically equivalent (t(29) = 1.99, p = 0.055) (Fig. 4b).
Boundaries trigger rapid reactivation of the just-encoded event during movie viewing
Leveraged by previous findings indicating that explicit context shifts triggered rapid reinstatement of the just-encoded picture event sequence and that such reinstatement at boundaries promoted the formation of long-term memory for that event (Sols et al., 2017), we tested the prediction that neural reactivation may also support memory formation of the just-encoded event during much more subtle transitions between events during movie watching, providing converging evidence that memory reinstatement at event boundaries facilitates the storage of events into long-term memory. To address this issue, we computed a neural similarity analysis between EEG data epochs of 10 s preceding and following boundary time points and compared the resulting similarity values for events that were later recalled in the free recall task with events that were later forgotten. This analysis revealed that patterns ∼1.5 s after boundary were significantly more similar to patterns ∼5–10 s before boundary when these preboundary events were later recalled (Fig. 5a). These findings provide evidence that neural reactivation is a mechanism supporting the formation of event episodic memories upon boundary detection during a continuous stream of stimuli.
Neural responses accompanying neural reactivation at boundaries
Although memory reactivation was found to take place rapidly upon boundary onset (i.e., ∼1.5 s) in our study, research on ERPs has also revealed the existence of fast neural-evoked responses produced when the input directly disconfirms comprehenders' anticipations of upcoming narrative constituent structure. Specifically, this research used visual narrative picture sequences (comic strips) and showed that a left-lateralized anterior negativity ERP effect appeared between 500 and 700 ms to pictures that disrupted the constituent narrative structure of the sequence (Cohn et al., 2014). Thus, given that memory formation of a meaningful event may depend on the ability to perceive a narrative shift at a boundary (i.e., segmentation) (Kurby and Zacks, 2008), we asked whether perceived movie boundaries following events that were later recalled triggered a lateralized anterior negativity ERP component. To address this issue, we compared the ERPs time-locked to boundaries following events that were later recalled and forgotten, and we found that these two conditions showed a differential ERP in a time window of 600–1400 ms after boundary onset, being more negative in polarity after boundaries following events that were later recalled (Fig. 5b) with a left-lateralized anterior scalp distribution (Fig. 5c). Importantly, this ERP difference was not observed when the same analysis was performed at neural responses time-locked to boundaries preceding later recalled and forgotten events (Fig. 5b), and these findings could not be attributed to a distinct proportion of events that were later recalled or forgotten following the boundary event either (Table 1), indicating that the memory formation processes associated with memory formation had a retrospective nature when boundaries were detected. Furthermore, a source brain analysis revealed that activity from frontal, parietal, and medial temporal regions were involved in the ERP differences between conditions (Fig. 5d), matching brain regions found to be associated with event boundary segmentation in our previous fMRI study (Baldassano et al., 2017). Together, these findings suggest that memory reactivation at event boundaries is accompanied by a rapid neural response signaling a shift between previous and current ongoing event information.
Neural reactivation during free recall
An intriguing finding in our previous fMRI study using the same movie was that the elicited patterns of neural activity associated with event segmentation during encoding were later reinstated during free recall (Baldassano et al., 2017). The extent to which these findings could be replicated using electrophysiological recordings may be relevant to open new venues for examining the neural mechanisms supporting event structure reinstatement patterns. To address this possibility, we adapted the approach implemented in our previous study based on HMM to the present EEG study. Briefly, the HMM approach implements a data-driven segmentation search and returns the most probable division of a given signal to a given number of events. An important advantage of the HMM in the context of this study is that it provides a data-driven solution for how the ongoing pattern of neural activity may be sequenced into a given number of events. This is an attractive approach as it allows searching for the existence of patterns of neural activity related to complex event sequence structure in a flexible manner, as the algorithm can be applied to brain signals of different length. This is particularly relevant in the context of a free recall task, as in the current study, given that total recall length and per-event recall time length varied across participants and within participants, respectively (Fig. 6a). Thus, for each participant, the HMM was used to estimate a 38-event segmentation of the continuous EEG data acquired during recall that most closely corresponded to the 38 neural event patterns elicited during movie watching. If, according to our previous findings using fMRI (Baldassano et al., 2017), participants' recall involved the reinstatement of neural patterns during encoding, we would expect event-elicited EEG activity during encoding and HMM-derived event-elicited EEG activity at recall to be very similar. To measure this, we correlated the EEG patterns elicited during encoding and recall and tested whether the resulting correlation value was higher than in a null distribution obtained by shuffling the duration of the encoding events before computing the mean patterns of each event before running the HMM on recall data. We found that the correlation values between movie and recall could not be attributed to chance at group level (p < 0.01) (Fig. 6b). This result extends previous fMRI findings (Baldassano et al., 2017), demonstrating that memory recall is supported by the reactivation of the electrophysiological patterns elicited during movie viewing.
Possible eye movement artifacts near event boundaries
An important concern that needed to be addressed to attest the validity of our results was the possibility of eye movements being systematically associated with event boundaries, which could potentially account for, or obscure, the EEG results. To discard this possibility, the number of EOG artifacts whose absolute distance from the nearest boundaries was smaller than 10 s around the event boundaries, time points were selected and compared with a null distribution where the real EOG above-threshold time points were shuffled while maintaining the distance lengths between them (N = 1000). The real distances were compared with the null distribution by computing a z value, which was then converted to a p value using the normal distribution. At encoding, we found that the distribution of real EOG above-threshold activity was not statistically different from the averaged null distribution; 27 of 30 participants had p values >0.05; and at the group level, the obtained p value was 0.5, which indicates that there is no evidence that EOG artifacts elicited during movie viewing are associated with boundaries (Fig. 7a). A similar analysis was conducted during recall; but considering the distances between EOG above-threshold activity and boundaries time points obtained using the event segmentation HMM-based approach. This analysis revealed that 30 participants had p values >0.05; and at the group level, the obtained p value was 0.8, which confirmed that potential eye movement artifacts were not associated with event boundaries during recall either (Fig. 7b).
Discussion
Our results provide the first evidence of electrophysiological signatures related to how event segmentation during movie viewing shapes memory formation. They show that patterns of neural activity recorded from the scalp EEG while viewing a 50 min movie fit with an event segmentation model of episodic events punctuated by rapid transitions of content (i.e., event boundaries). We observed that these event-specific patterns of neural activity were reinstated at later recall, thereby corroborating the idea that the event segmentation process shaped memory formation of a continuous stream of stimuli into a structured memory representation that can be accessed long-term. Leveraged by the fine-grained temporal resolution of the EEG data, we showed that event memory formation during movie viewing was mediated by its rapid reactivation at event boundaries and that memory reactivation was accompanied by a left-lateralized anterior negativity ERP effect time-locked to the boundary, which likely reflects the effective detection of a context switch between the current and just-encoded event. These findings indicate that the successful encoding of an episode is regulated by two neural mechanisms that act within the first ∼2 s after an event boundary.
Why would memory reactivation be advantageous during the encoding of a continuous stream of stimuli? Although event segmentation provides a framework to examine how continuous experience can be chunked into a set of discrete episodes in memory through the detection of event boundaries, it does not account for how this sequence of episodes can be integrated into a memory structure that preserves the temporal structure during later recall. Memory reactivation at event boundaries may represent a way to promote temporal event memory organization across boundaries as it may serve to promote the strengthening, or chunking, of that just-encoded event, but it also may help promote binding across boundary episodes as a result of the contemporaneously coactivation of the past and present events (Sols et al., 2017). The extent to which memory reactivation at event boundaries serves to promote the encoding of unique events into memory, the integration of different events into a temporally organized memory structure or both is difficult to disambiguate in our study as participants memory accuracy was obtained through a free recall task, which relies on retrieval processes heavily dependent on clustering properties of the encoded material, such as semantic similarity or temporal proximity (Polyn et al., 2009).
Speculatively, it could be argued that memory reactivation at event boundaries could represent a way to account for how different event episodes that shared contextual semantic properties can be integrated. In support of this hypothesis, previous fMRI studies have shown that temporally extended events sharing contextual information were evaluated as if they appeared closer in time during recall and that this was related to increased hippocampal similarity between these events during encoding (Ezzyat and Davachi, 2011). Interestingly, this effect was only observed for when events that shared contexts were separated by event boundaries during their encoding, suggesting the possibility that neural mechanisms triggered at boundaries were at least partially responsible for memory integration (e.g., memory reactivation). Another set of research studies have emphasized the relevance of memory reactivation to explain how different episodes are integrated as a function of the degree of their overlapping content to allow generalization (Schlichting and Preston, 2015). These studies have shown that memory reactivation is elicited when elements of the experience partially mismatch with stored memory representations, supporting integrative encoding online (Shohamy and Wagner, 2008).
An open question is which specific mechanisms triggers memory reactivation at event boundaries. Advancing on this issue is not trivial given the diverse ranges of stimulus used in event cognition literature: for instance, text narratives (Zwaan, 1996), short video clips (Ben-Yakov and Dudai, 2011; Ben-Yakov et al., 2013), or item sequences (Dubrow and Davachi, 2013). In an attempt to accommodate the literature on this topic, Clewett and Davachi (2017) suggested that event boundaries represent moment-to-moment fluctuations in external and internal contextual states during continuous encoding. In our study, such fluctuations could be understood as moments in time when an internal representation derived from an accumulated contextual encoding suddenly shifts at the start of a novel scene with a change in spatial location, characters, or goals. Interestingly, this effect was only observed for when events that shared contexts were separated by event boundaries during their encoding, suggesting the possibility that neural mechanisms triggered at boundaries were at least partially responsible for memory integration (e.g., memory reactivation) (Griffiths and Fuentemilla, 2019). Another set of research studies have emphasized the relevance of memory reactivation to explain how different episodes are integrated as a function of the degree of their overlapping content to allow generalization (Schlichting and Preston, 2015). These studies have shown that memory reactivation is elicited when elements of the experience partially mismatch with stored memory representations, supporting integrative encoding online (Shohamy and Wagner, 2008). Interestingly, we found that these moments in time were followed by a left-lateralized anterior negative ERP effect, specifically for when the previous event was later recalled but not for those that were forgotten. The fact that this specific ERP response accompanied memory reactivation after an event boundary lends support to the notion that it could reflect a neural process that may be necessary to trigger neural reactivation. However, this idea should be taken cautiously as our data do not provide a quantifiable dependency between the two, other than differences in their temporal onset.
Similarly, it would be important to delimit the representational nature that encompasses rapid memory reactivation at event boundaries. Despite the relevance of this question, our current approach, based on scalp EEG activity recordings from the scalp, is blind on whether the increased neural similarity at event boundaries to recalled events may reflect the representation of episodic details of the just-encoded event or whether, indeed, it encompasses the reactivation of the gist-based memory representation of the encoded event, or both. Disambiguating between these possibilities is not trivial when using scalp EEG recordings for at least two reasons. First, the intrinsic temporal correlation of the EEG data did not allow us to discern whether memory reactivation was associated with the entire neural patterns elicited during the preboundary event, as our findings were based on the comparison of two memory conditions that showed increases in neural similarity for time points surrounding the boundary (see Fig. 5a). And second, as opposed to data from fMRI that can link the neural responses to known representational hierarchy coding along the visual stream (e.g., Baldassano et al., 2017), we are agnostic on how the differential gradient of specific versus abstract memory representations may be distinguished, if they may, in the scalp EEG signal.
In a series of fMRI experiments, Ben-Yakov and Dudai (2011; Ben-Yakov et al., 2013) revealed the importance of studying event offset brain activity in humans at the end of movie clips to understand how episodic memories are formed during the stream of a continuous audiovisual stimulation. These studies, together with those studying abrupt switches between stimulus category and task (DuBrow and Davachi, 2014), and recent studies of event boundaries in movies (Baldassano et al., 2017; Ben-Yakov et al., 2018), offered converging evidence for the specificity and sensitivity of the coupling of the hippocampus to event boundaries during movie viewing. In the current study, we found that the brain sources of the left-lateralized anterior negative ERP effect associated with memory formation at boundaries of the just-encoded event episode included frontal, parietal, and medial temporal lobe regions. These regions highly overlapped with brain regions found in our previous fMRI findings of hippocampally linked event boundaries (Baldassano et al., 2017). The similarity of these sources represents strong evidence that our approach was suitable for identifying the engagement of the same brain network and informing about the temporal properties of their engagement using noninvasive electrophysiological recordings.
In naturalistic scenarios, the study of the recollection of memories of one's past may vary substantially across individuals and within subjects as a function of task contexts. This originates an important challenge in our search for the neural underpinnings, supporting the remembering of autobiographical memories. The implementation of data-driven modeling approaches, such as the ones offered by HMM used in the current study, may foster interesting possibilities in this endeavor. Indeed, being able to identify the reinstatement of memory events from a 50 min movie viewing using HMM extends previous fMRI findings (Baldassano et al., 2017). Our work, however, is the first to show that HMM can be used to model electrophysiological signals, thereby proving its usefulness to test predictions of how perception and memory are supported by brain mechanisms that act at fine-grained temporal resolution. Nevertheless, future work may extend the current approach by examining how HMM can be implemented in electrophysiological signal to search for unknown patterns. Notably, our approach was successful in identifying the appearance of a known pattern during recall extracted from encoding. It would be interesting to discern which electrophysiological features are relevant during encoding, thereby providing a way to contribute to findings set out from our previous fMRI studies (e.g., Baldassano et al., 2017).
Understanding how memories are formed and structured in real-life requires the characterization of neural mechanisms that take place online, during the ongoing encoding of continuous naturalistic stimuli, as our experience unfolds over time. Investigating how memories are formed during audiovisual narratives, such as long movie clips, may provide a valuable approach to bring testable predictions derived from animal and theoretical neuroscience into real-life settings. The current experiment assessed whether memory reinstatement, a neural mechanism critical for memory formation and consolidation, took place under these ecologically valid experimental circumstances. By showing that this is the case, our findings offer insights into how the brain shapes the unfolding experience into long-term memory that can be generalized to real-life.
Footnotes
This work was supported by European Union Erasmum + program to M.S., and Ministerio de Ciencia, Innovación y Universidades to RDB, which is part of Agencia Estatal de Investigación (AEI), through the project PSI2016-80489-P (Co-funded by European Regional Development Fund. ERDF, a way to build Europe), to L.F. We thank CERCA Programme/Generalitat de Catalunya for institutional support.
The authors declare no competing financial interests.
- Correspondence should be addressed to Marta Silva at marta.m.silva22{at}gmail.com