Elsevier

Cognition

Volume 178, September 2018, Pages 26-36
Cognition

Original Articles
Mechanisms of value-learning in the guidance of spatial attention

https://doi.org/10.1016/j.cognition.2018.05.005Get rights and content

Highlights

  • Attention is biased towards previously reward-associated locations within a scene.

  • This spatial attention bias is driven by reinforcement learning mechanisms.

  • The bias is specific to meaningful spatial layouts.

  • Our findings reconcile previously conflicting reports concerning selection history and spatial attention.

Abstract

The role of associative reward learning in the guidance of feature-based attention is well established. The extent to which reward learning can modulate spatial attention has been much more controversial. At least one demonstration of a persistent spatial attention bias following space-based associative reward learning has been reported. At the same time, multiple other experiments have been published failing to demonstrate enduring attentional biases towards locations at which a target, if found, yields high reward. This is in spite of evidence that participants use reward structures to inform their decisions where to search, leading some to suggest that, unlike feature-based attention, spatial attention may be impervious to the influence of learning from reward structures. Here, we demonstrate a robust bias towards regions of a scene that participants were previously rewarded for selecting. This spatial bias relies on representations that are anchored to the configuration of objects within a scene. The observed bias appears to be driven specifically by reinforcement learning, and can be observed with equal strength following non-reward corrective feedback. The time course of the bias is consistent with a transient shift of attention, rather than a strategic search pattern, and is evident in eye movement patterns during free viewing. Taken together, our findings reconcile previously conflicting reports and offer an integrative account of how learning from feedback shapes the spatial attention system.

Introduction

The role of an observer's goals (top-down factors) and the physical salience of objects (bottom-up factors) in the control of attention have been well established and serve as the foundation for prominent models of selective attention (e.g., Corbetta and Shulman, 2002, Desimone and Duncan, 1995, Theeuwes, 2010, Wolfe et al., 1989). More recently, it has been argued that this dichotomy cannot explain the role of selection history in the control of attention, which appears to be both non-strategic and independent of the physical salience of stimuli (Awh, Belopolsky, & Theeuwes, 2012). In this context, an important component of selection history has been argued to reflect associative reward learning, with objects previously associated with reward automatically capturing visual attention (Anderson, 2013).

The role of associative reward learning in the control of attention was initially demonstrated using stimuli defined by shape, with results showing that stimulus competition was biased for or against different shapes based on whether observers were rewarded for selecting or ignoring them, respectively (Della Libera & Chelazzi, 2009). This bias carried over into extinction, suggesting that it was non-strategic. Powerful evidence for the unique role of associative reward learning in the control of attention was provided by a study in which task-irrelevant distractors were rendered in a color that had been predictive of reward during a prior training phase. These distractors were not physically salient (less so than the target), and the color of stimuli was known by participants to be completely irrelevant to the task. Nevertheless, attention was automatically captured by the previously reward-associated colors, suggesting a distinct mechanism of attentional control that has been referred to as value-driven attention (Anderson, Laurent, & Yantis, 2011).

Many subsequent studies have adopted this approach of associating stimulus features (often color) with reward and then presenting the previously reward-associated features as distractors, replicating and extending the phenomenon of value-driven attention (e.g., Anderson, 2016a, Anderson, 2016c, Anderson et al., 2016, Anderson et al., 2012, Anderson and Yantis, 2012, Anderson and Yantis, 2013, Anderson, Kuwabara, et al., 2016, Failing and Theeuwes, 2014; Le Pelley et al., 2015, Mine and Saiki, 2015, Moher et al., 2015, Pool et al., 2014; see Anderson, 2016b, for a recent review). Attention has been successfully trained to favor a variety of stimulus features, ranging from specific colors (e.g., Anderson et al., 2011) and orientations (Laurent et al., 2015, Lee and Shomstein, 2014) to shapes (Della Libera & Chelazzi, 2009) and even object categories (Hickey & Peelen, 2015). In addition to feature-based attention, object-based attention also appears to be strongly modulated by associative reward learning (Lee and Shomstein, 2013, Shomstein and Johnson, 2013).

Few studies have probed the influence of associative reward learning on the control of space-based attention, where reward is not predicted by a particular stimulus feature but rather by where in space attention needs to be directed in order to receive high reward. Value-driven attentional biases for a stimulus feature can be modulated by spatial information, such that the bias is specific to when that feature appears in a region of space in which it was rewarded (Anderson, 2015a). More purely space-based attentional biases have also been shown to be modulated by reward. When a high reward is received for identifying a target appearing in a given location, this location is prioritized on the subsequent trial (Hickey, Chelazzi, & Theeuwes, 2014), extending earlier evidence for reward-mediated priming of stimulus color (Hickey, Chelazzi, & Theeuwes, 2010). A more enduring bias towards a previously rewarded location was demonstrated following a multi-day training protocol in which participants performed a difficult visual search for alphanumeric among nonalphanumeric characters. Which of eight possible stimulus positions a searched-for character appeared in predicted the amount of money earned for reporting that target on a given trial. During extinction, participants were more likely to report a target appearing in a previously high-value location, specifically when two targets were simultaneously presented that competed for attention (Chelazzi et al., 2014).

At least two cases have been reported in which a spatial reward manipulation failed to produce any evidence for an enduring attentional bias, or even an attentional bias during the period in which the reward structure was currently in place. In each study, multiple experiments were conducted in which participants searched for a “T” among offset “L” distractors (Jiang et al., 2015, Won and Leber, 2016). When the target appeared anywhere within a particular quadrant of the screen, it was much more likely to yield a high reward if correctly identified. Under a variety of conditions, including conditions of time pressure in which participants should be highly motivated to preferentially search the high-value quadrant in order to maximize rewards, no measurable spatial attention bias was observed (Jiang et al., 2015, Won and Leber, 2016). Furthermore, under similar conditions in which participants were asked to instead choose a particular stimulus rather than perform visual search, robust spatial preferences for highly rewarded locations were observed. These results suggest that spatial reward can readily influence choice behavior (Won & Leber, 2016), but seemingly not the allocation of attention during visual search (Jiang et al., 2015, Won and Leber, 2016).

It is important to note that space-based attentional biases are robustly influenced by a different source of selection history using the same experimental paradigm that failed to show value-driven biases. Specifically, participants are much faster to report targets in locations that more frequently contained targets in prior trials, even well after such biased probabilities are no longer in place (e.g., Jiang and Swallow, 2013, Jiang et al., 2013, Jiang et al., 2015, Won and Leber, 2016). Similarly, targets can be found more quickly when the position of the target is consistently predicted by the spatial configuration of non-targets (e.g., Chun and Jiang, 1998, Chun and Jiang, 2003), a phenomenon termed contextual cueing. That a different form of selection history can so robustly bias spatial attention in this paradigm argues against a general insensitivity of the paradigm to the ability to detect a learned spatial attention bias.

These repeated failures to observe reliable effects of reward history on the allocation of spatial attention during visual search have naturally led to skepticism concerning whether principles of value-driven attention extend to the spatial domain (Jiang et al., 2015, Won and Leber, 2016). Indeed, it has been suggested that evolutionary pressures imposed by naturally occurring reward structures might strongly favor feature-reward pairings over space-reward pairings, rendering influences of the reward system on spatial attention phylogenetically implausible (Won & Leber, 2016). We would argue that this is a fair point, in the context of how space is defined in these studies.

In traditional visual search paradigms, including those used by Jiang et al. (2015) and Won and Leber (2016), space is defined in a highly abstract manner: a region of a blank computer screen. In fact, there is no clear anchor point for defining where one region would end and another begin, apart from the borders imposed by the edges of the monitor. Such a highly abstract notion of space is unlikely to engage the spatial representations one might use to guide search for a valued item based on learning history, such as where ice cream tends to be stored in the freezer. In this case, the valued location is defined in the context of the spatial arrangement of objects in the scene (e.g., the position of the freezer relative to other objects in the room, and which section of which shelf when looking inside the freezer). The spatial information provided by real-world scenes can serve as the basis for contextual cueing of target position (e.g., Brockmole and Henderson, 2006a, Brockmole and Henderson, 2006b), suggesting a rich source of spatial guidance, although contextual cueing is also evident with the more abstract stimulus displays that have failed to produce evidence of value-based attentional biases (Jiang et al., 2015, Won and Leber, 2016).

Perhaps information pertaining to the spatial layout and arrangement of objects in a scene is useful for guiding spatial attention on the basis of reward history, which might help explain the apparent discrepancy between Chelazzi et al. (2014) on the one hand, and Jiang et al. (2015) and Won and Leber (2016) on the other hand. Chelazzi et al. (2014) only found an effect of reward when two targets simultaneously competed for attention, where reward was not only tied to the absolute spatial location of the targets but also to their relative positions. With the aim of reconciling these conflicting reports, in the present study, we examined the role of value learning in the context of real-world scenes, both scenes containing a rich array of objects with a consistent spatial arrangement and scenes containing no objects (textures).

Section snippets

Experiment 1

In Experiment 1, participants were first trained to associate a specific region of multiple different scenes with monetary reward. On each trial, one of eight scenes remained on the screen until participants clicked on a pixel within the scene using the mouse cursor. Participants were instructed that they would be rewarded for each click, and that the amount of reward received depended on where they clicked. Unbeknownst to the participant, for each of the eight scenes, clicking in one quadrant

Experiment 2

The results of Experiment 1 provide straightforward evidence that selection history can robustly bias spatial attention in a context-specific manner. However, it is unclear to what degree this observed bias is the product of a reward-mediated process specifically. This is because the amount of reward participants earn on each trial is entirely contingent on their orienting behavior. Not only do participants experience greater rewards when selecting the high-value quadrant, they simply select it

Experiment 3

The findings of Experiment 2, in which non-reward feedback both curbed spatial choice behavior and biased spatial attention in a subsequent task, raises an important question concerning the reason why significant biases were observed in the present study but not in Jiang et al. (2015) and Won and Leber (2016). As previously suggested, one possibility is that the scene context allows participants to anchor spatial representations to the configuration of objects within a scene (e.g., to the left

Experiment 4

Another source of ambiguity, both in the spatial attention bias observed in Experiments 1 and 2 and also in prior demonstrations of spatial attention biases driven by selection history (Chelazzi et al., 2014), is the degree to which these biases reflect an automatic stimulus-evoked shift of attention on the one hand versus a bias to execute a particular search pattern or strategy on the other hand. This is because, in the test phase, participants are tasked with searching the display for a

Experiment 5

Experiment 5 sought to provide an even stronger test of whether reinforcement learning biases scene-evoked shifts of spatial attention rather than the execution of a specific search strategy. To this end, we had participants complete a free-viewing task during the test phase, and measured eye position as they viewed the familiar scenes. If reinforcement learning results in an enduring bias to direct attention to a particular region of a scene, participants should preferentially direct their

General discussion

Although the ability of reward learning to modulate feature-based attention is well established (see Anderson, 2016b), whether reward learning can influence spatial attention has been much more controversial (Chelazzi et al., 2014, Jiang et al., 2015, Won and Leber, 2016). In the present study, we sought to characterize whether and how selection history influences spatial attention orienting. Our findings offer new insights into the mechanisms and principles governing the interplay between

Funding statement

The reported research was funded through a start-up package to BAA through Texas A&M University and NARSAD Young Investigator Grant 26008 to BAA.

References (81)

  • J. Moher et al.

    Dissociable effects of salience on attention and goal-directed action

    Current Biology

    (2015)
  • E. Pool et al.

    Where is the chocolate? Rapid spatial orienting toward stimuli associated with primary reward

    Cognition

    (2014)
  • S. Qi et al.

    Neural correlates of reward-driven attentional capture in visual search

    Brain Research

    (2013)
  • P.R. Roelfsema et al.

    Perceptual learning rules based on reinforcers and attention

    Trends in Cognitive Sciences

    (2010)
  • A. Seitz et al.

    Requirement for high-level processing in subliminal learning

    Current Biology

    (2005)
  • A. Seitz et al.

    A unified model for perceptual learning

    Trends in Cognitive Sciences

    (2005)
  • J. Theeuwes

    Top-down and bottom-up control of visual selection

    Acta Psychologica

    (2010)
  • A. Treisman et al.

    A feature-integration theory of attention

    Cognitive Psychology

    (1980)
  • B.A. Anderson

    A value-driven mechanism of attentional selection

    Journal of Vision

    (2013)
  • B.A. Anderson

    Value-driven attentional capture is modulated by spatial context

    Visual Cognition

    (2015)
  • B.A. Anderson

    Value-driven attentional priority is context specific

    Psychonomic Bulletin and Review

    (2015)
  • B.A. Anderson

    Social reward shapes attentional biases

    Cognitive Neuroscience

    (2016)
  • B.A. Anderson

    The attention habit: How reward learning shapes attentional selection

    Annals of the New York Academy of Sciences

    (2016)
  • B.A. Anderson

    Value-driven attentional capture in the auditory domain

    Attention, Perception, and Psychophysics

    (2016)
  • B.A. Anderson et al.

    On the distinction between value-driven attention and selection history: Evidence from individuals with depressive symptoms

    Psychonomic Bulletin and Review

    (2017)
  • B.A. Anderson et al.

    Attentional bias for non-drug reward is magnified in addiction

    Experimental and Clinical Psychopharmacology

    (2013)
  • B.A. Anderson et al.

    Mechanisms of habitual approach: Failure to suppress irrelevant responses evoked by previously reward-associated stimuli

    Journal of Experimental Psychology: General

    (2016)
  • B.A. Anderson et al.

    On the value-dependence of value-driven attentional capture

    Attention, Perception, and Psychophysics

    (2017)
  • B.A. Anderson et al.

    Value-driven attentional capture

    Proceedings of the National Academy of Sciences, USA

    (2011)
  • B.A. Anderson et al.

    Generalization of value-based attentional priority

    Visual Cognition

    (2012)
  • B.A. Anderson et al.

    The attribution of value-based attentional priority in individuals with depressive symptoms

    Cognitive, Affective, and Behavioral Neuroscience

    (2014)
  • B.A. Anderson et al.

    Value-driven attentional and oculomotor capture during goal-directed, unconstrained viewing

    Attention, Perception, and Psychophysics

    (2012)
  • B.A. Anderson et al.

    Persistence of value-driven attentional capture

    Journal of Experimental Psychology: Human Perception and Performance

    (2013)
  • L. Barbaro et al.

    Valence, not utility, underlies reward-driven prioritization in human vision

    Journal of Neuroscience

    (2017)
  • A. Bourgeois et al.

    How does awareness modulate goal-directed and stimulus-driven shifts of attention triggered by value learning?

    PLoS ONE

    (2016)
  • D.H. Brainard

    The psychophysics toolbox

    Spatial Vision

    (1997)
  • J.R. Brockmole et al.

    Recognition and attentional guidance during contextual cueing in real-world scenes: Evidence from eye movements

    Quarterly Journal of Experimental Psychology

    (2006)
  • J.R. Brockmole et al.

    Using real-world scenes as contextual cues for search

    Visual Cognition

    (2006)
  • B. Bucker et al.

    Pavlovian reward learning underlies value driven attentional capture

    Attention, Perception, and Psychophysics

    (2017)
  • L. Chelazzi et al.

    Altering spatial priority maps via reward-based learning

    Journal of Neuroscience

    (2014)
  • Cited by (30)

    • Dynamic modulation of neural feedback processing and attention during spatial probabilistic learning

      2022, iScience
      Citation Excerpt :

      Similar behavioral results are also commonly reported in two-choice reward learning tasks which deploy nonspatial target-features as their to-be-rewarded stimuli (van den Berg et al., 2019; Bourgeois et al., 2017). In addition, the learning of stimulus-reward associations is a prerequisite in most studies investigating their attentional capture (Anderson et al., 2011; Anderson and Kim, 2018; Chelazzi et al., 2014). Given that the average net gain was positive in our study, our data support that participants actively used the feedback information to learn the spatial stimulus-reward association.

    • The past, present, and future of selection history

      2021, Neuroscience and Biobehavioral Reviews
      Citation Excerpt :

      The effect of learned value on attention can be observed even when previously reward-associated stimuli appear as task-irrelevant distractors and when rewards are no longer delivered to motivate performance (e.g., Anderson et al., 2011a, b; Anderson and Halpern, 2017). There is some evidence that reward can similarly bias attention to spatial locations at which identification of a target is more highly rewarded (Chelazzi et al., 2014; Della Libera et al., 2017), although there are limitations to the scope of this bias (Jiang et al., 2015a; Won and Leber, 2016) and it may operate differently than the influence of reward on stimulus features (see Anderson and Kim, 2018a, b), which is a possibility that we will return to later in the paper. Association with aversive outcomes.

    • Reward learning biases the direction of saccades

      2020, Cognition
      Citation Excerpt :

      Saccading to a target in one direction was associated with more reward (probabilistically) than saccading to a target in any of the other directions. In this task, the initial saccade from central fixation is, in principle, analogous to the different contexts (scenes) in Anderson and Kim's (2018a, 2018b) paradigm, as it separated the rewarded direction from a specific spatial location on the screen. We examined whether participants could robustly learn which direction was the optimal direction in which to orient during training, and whether such learning would transfer to performance in an unrewarded visual search task.

    • Neurobiology of value-driven attention

      2019, Current Opinion in Psychology
      Citation Excerpt :

      Given that the superior colliculus plays a role in multisensory integration [61] and receives input from the caudate tail [27], it is possible that the caudate tail might be involved in signaling value-driven attentional priority in more than one sensory modality. Reward learning has also been shown to bias spatial attention [62–64], but the neural mechanisms underlying this dimension of value-driven attention have not been investigated. One possibility would be contextually-dependent weighting of the spatial priority map [52,62].

    View all citing articles on Scopus
    View full text