The role of temporal predictability in semantic expectation: An MEG investigation

Abstract

Prior research suggests that prediction of semantic and syntactic information prior to the bottom-up input is an important component of language comprehension. Recent work in basic visual and auditory perception suggests that the ability to predict features of an upcoming stimulus is even more valuable when the exact timing of the stimulus presentation can also be predicted. However, it is unclear whether lexical-semantic predictions are similarly locked to a particular time, as previous studies of semantic predictability have used a predictable presentation rate. In the current study we vary the temporal predictability of target word presentation in the visual modality and examine the consequences for effects of semantic predictability on the event-related N400 response component, as measured with magnetoencephalography (MEG). Although we observe robust effects of semantic predictability on the N400 response, we find no evidence that these effects are larger in the presence of temporal predictability. These results suggest that, at least in the visual modality, lexical-semantic predictions may be maintained over a broad time-window, which could allow predictive facilitation to survive the presence of optional modifiers in natural language settings. The results also indicate that the mechanisms supporting predictive facilitation may vary in important ways across tasks and cognitive domains.

Introduction

Many decades of work in psycholinguistics has established that the broader context in which a word appears has a huge impact on the speed and accuracy of comprehension. More recent research suggests that a major means by which context affects language comprehension is through predictive processing, in which representations are activated or constructed in advance of the corresponding bottom-up input (DeLong et al., 2005, Dikker et al., 2009, Federmeier, 2007, Lau et al., 2008, Van Petten and Luka, 2012, Wicha et al., 2004). For example, given the context ‘It was a windy day, so the boy went out to fly a…’, it has been proposed that the lexical and/or conceptual representation of ‘kite’ will be ‘preactivated’ (e.g., Federmeier & Kutas, 1999) and that this representation will be prematurely added to a working memory representation of the sentence or message (e.g., Lau, Holcomb, & Kuperberg, 2013) in advance of the bottom-up input. Similarly, in the context of a series of highly related word pairs, one might strongly predict the word ‘dog’ following the word ‘cat’.

At the same time, the details of the predictive mechanism utilized by the brain to facilitate on-line comprehension are still largely unknown. In the current study, we seek to gain insights into this mechanism by using magnetoencephalography (MEG) to examine the impact of temporal predictability on predictive semantic facilitation. In particular, given that comprehenders perceive natural speech as proceeding at a rhythmically regular and predictable rate in real-world comprehension contexts (Dilley, Wallace, & Heffner, 2012), we ask whether the processing gains accrued by the comprehender from having predicted the semantic content of a word are contingent on also having predicted when the word would appear.

Past behavioral studies have repeatedly demonstrated that a word is recognized faster when it is preceded by a semantically related word compared to a semantically unrelated word, a phenomenon known as semantic priming (see Neely, 1991, for review), and a similar reaction time benefit is observed for responses to words that appear in supportive sentence contexts (Fischler and Bloom, 1979, Stanovich, 1981). However, because behavioral studies usually measure responses to metalinguistic tasks and do not allow precise estimation of the timing of processes, in the current study we focus on a neural correlate of this contextual facilitation: the amplitude of the electrophysiological component known as the N400. The N400 is a broad peak between 300 and 500 msec in the event-related potential (ERP) associated with stimuli that carry meaning, such as words, pictures, and faces (Kutas & Federmeier, 2011). Critically, the amplitude of this peak has been observed to depend on the broader context, such that in many of the same paradigms in which processing facilitation is observed behaviorally (semantic priming, supportive sentence contexts), the amplitude of the N400 component is reduced (Kutas and Hillyard, 1980, Kutas and Hillyard, 1984, Rugg, 1985).

Behavioral contextual facilitation and the neural modulation associated with it could reflect the result of predictive mechanisms, but it could also reflect more generic mechanisms such as bottom-up spreading activation between stored lexical-semantic representations. Several past findings suggest that the N400 can be modulated by prediction in particular. Federmeier and Kutas (1999) showed that the N400 was reduced in response to words that did not fit the sentence context as long as they shared semantic features with the word most strongly predicted by the context. More conclusively, elegant work by DeLong et al. (2005) demonstrated N400 modulation due to predictibility alone, when semantic association and bottom-up contextual congruity were tightly controlled.

In semantic priming paradigms, facilitation at short prime-target intervals (less than 200 msec) is believed to be due to automatic spreading activation, while facilitation at longer prime-target intervals is believed to be driven by ‘controlled’ mechanisms such as prediction. At long prime-target intervals, behavioral facilitation for semantically related pairs can be increased by increasing the number of related pairs in the experiment (e.g., 10% related trials versus 50% related trials), while relatedness proportion has little effect on the size of the priming effect at short prime-target intervals (e.g., de Groot, 1984, den Heyer et al., 1983, Hutchison et al., 2001; although cf. Bodner & Masson, 2003). This asymmetry can be accounted for by the idea that a higher proportion of related trials in the experiment will lead participants to generate stronger predictions for the target based on the prime, while leaving the automatic process of spreading activation unaffected. Correspondingly, electrophysiological experiments using the relatedness proportion paradigm suggest that N400 effects at long prime-target intervals largely reflect predictive facilitation, as the effect of relatedness on the N400 is quite small at low relatedness proportions and increases with higher relatedness proportion (e.g., Brown et al., 2000, Holcomb, 1988, Lau, Holcomb, et al., 2013, Lau et al., 2015). However, semantic priming at short prime-target intervals also modulates the N400 (e.g., Anderson and Holcomb, 1995, Lau, Gramfort, et al., 2013), indicating that the N400 is also sensitive to facilitation due to bottom-up spreading activation. These findings can be reconciled according to a view in which N400 amplitude indexes the extent of lexical-semantic activation across the network, and in which both prediction and spreading activation act to preactivate the candidate representation such that activation is more narrowly focused when the critical word is presented (Lau et al., in press).

Although there is thus increasing evidence that predictive mechanisms underlie many semantic facilitation effects, the nature and timing of these mechanisms is still poorly understood. While bottom-up spreading activation has been thought to onset and decay rapidly, it is not so clear that this should be the case for preactivation due to predictive processes. On the one hand, predictive activation may need to extend across time to accommodate cases in which the exact moment of the predicted word's representation is not known, for example in the presence of optional modifiers (… the boy went out to fly a new kite…). Furthermore, lexical predictions may depend on higher-level linguistic information, and thus predictions may be delayed until these higher-level representations can be constructed (Chow, 2013).

On the other hand, it may be the case that in a naturalistic setting, the timing of a predicted word can in fact be accurately approximated most of the time. A little remarked-upon property of almost every previous study to examine semantic prediction is that stimulus presentation is fairly rhythmic. Experiments using auditory presentation contain the temporal regularities inherent in natural speech. Although prior work indicates that syllables of perceptually natural speech are not produced at an objectively constant rate (e.g., Fowler, 1979, Hoequist, 1983, Morton et al., 1976), comprehenders have strong intuitions about the timing which results in perceived isochrony, and appear to use this to predict syllable onsets and facilitate speech perception (Dilley et al., 2012). Experiments using visual presentation typically implement a constant stimulus onset asynchrony (SOA) throughout an experimental block of trials. Such temporal regularity makes it possible for listeners or readers to predict approximately when the next word will be presented.

Previous research across multiple cognitive domains has demonstrated that temporal predictability has reliable impacts on both behavioral responses and neural activity, for example in both visual and auditory perception (e.g., Doherty et al., 2005, Jones et al., 2002; see Näätänen & Picton, 1987 for an early review of auditory work and Nobre, Correa, & Coull, 2007 for a more recent cross-domain review). Of note here, a number of studies indicate that such temporal predictions interact synergistically with other kinds of predictive information (e.g., Costa-Faidella et al., 2011, Doherty et al., 2005, Hsu et al., 2013, Kingstone, 1992, Rohenkohl et al., 2014). For example, Rohenkohl et al. (2014) showed that temporal predictability improves performance on a visual discrimination task if participants know ahead of time where the target is likely to appear, but it has no effect in the absence of this kind of spatial expectation. Similarly, Doherty et al. (2005) found that temporal expectation only modulated the P1 attentional component in the presence of spatial expectations. Although less well-studied, there is some evidence that non-spatial predictions for visual features also interact with temporal predictions to facilitate behavioral responses (Kingstone, 1992).

The mechanism by which temporal predictability exerts these effects is only beginning to be understood. Recent work has suggested that temporal expectation facilitates perceptual rather than decision-level processing (Jepma et al., 2012, Rohenkohl et al., 2012). For example, Rohenkohl et al. (2012) show temporal expectation impacting visual perception by increasing the effective contrast of the visual stimulus. It has been widely suggested that such a mechanism could be implemented through modulation of ongoing oscillations in neural firing: oscillatory activity in relevant neural populations becomes entrained to a regular stimulus rate, or a stimulus event acts as a trigger to reset the phase of the oscillations, so that the relevant neurons for the predicted stimulus are in the optimal state to react when the stimulus is presented (e.g., Arnal and Giraud, 2012, Lakatos et al., 2008, Stefanics et al., 2010).

The goal of the current work was to begin to investigate how exactly linguistic predictions are impacted by temporal predictability. If semantic prediction were implemented through an oscillatory signal phase-locked to stimulus presentation, pre-activation would be relatively brief and targeted to the particular time-window in which the expected word was likely to be presented. Conversely, tying lexical-semantic predictions to a narrow time-window may be non-optimal because of the availability of optional modifiers and alternative syntactic frames in natural language contexts. Here, we investigate these possibilities by examining how temporal expectation interacts with the well-established effect of semantic expectation on the N400 response to visual words in a semantic priming paradigm. Following previous work, we manipulate temporal predictability by varying the regularity of stimulus onset asynchrony times (Costa-Faidella et al., 2011, Schwartze et al., 2013). We use visual presentation because it allows tighter control of the onset of semantic information (the entire word is presented instantaneously, rather than being spread across time) and because randomized temporal irregularity is less jarring in the more artificial rapid serial visual presentation paradigm than it would be in otherwise naturalistic speech. We use MEG to record the N400 response rather than EEG because MEG responses are more focal than EEG, allowing us to focus on sensors corresponding to the left-hemisphere neural populations that give rise to the largest N400 effect (Lau et al., 2009, Lau et al., 2008). We also conduct distributed source localization analyses to confirm the standard left anterior temporal location for our semantic predictability effects and to better account for individual differences in head position.