Neural substrates underlying effort, time, and risk-based decision making in motivated behavior

Abstract

All mobile organisms rely on adaptive motivated behavior to overcome the challenges of living in an environment in which essential resources may be limited. A variety of influences ranging from an organism’s environment, experiential history, and physiological state all influence a cost-benefit analysis which allows motivation to energize behavior and direct it toward specific goals. Here we review the substantial amount of research aimed at discovering the interconnected neural circuits which allow organisms to carry-out the cost-benefit computations which allow them to behave in adaptive ways. We specifically focus on how the brain deals with different types of costs, including effort requirements, delays to reward and payoff riskiness. An examination of this broad literature highlights the importance of the extended neural circuits which enable organisms to make decisions about these different types of costs. This involves Cortical Structures, including the Anterior Cingulate Cortex (ACC), the Orbital Frontal Cortex (OFC), the Infralimbic Cortex (IL), and prelimbic Cortex (PL), as well as the Baso-Lateral Amygdala (BLA), the Nucleus Accumbens (NAcc), the Ventral Pallidal (VP), the Sub Thalamic Nucleus (STN) among others. Some regions are involved in multiple aspects of cost-benefit computations while the involvement of other regions is restricted to information relating to specific types of costs.

Introduction

Some of the earliest laboratory studies of motivated behavior led researchers to observe that most complex behavior tends to occur in bouts and that specific behaviors such as feeding or grooming can be characterized by their frequency, intensity, temporal distribution and direction towards or away from a particular stimulus. One of the prominent researchers of the day went so far as to say that identifying the factors responsible for the initiation and termination of these specific bouts of behavior would be the central problem for experimental psychologists to understand (Richter, 1927). Over the years there have been numerous theories of motivation put forth (Bolles and Moot, 1972, Hebb, 1955, Hull, 1943, Young, 1961), each of which has been influential in stimulating what has been a continuous stream of experiments and research on this topic. There exist excellent reviews of many of these theories and concepts (Berridge, 2004).

Almost a century later, researchers from numerous fields including psychology, psychiatry, and neurobiology are still actively studying goal-directed motivation, which is the name that has been given to the set of biological and psychological processes which guides behavior in pursuit of a goal. Research in this realm of behavioral neuroscience has come a long way toward understanding the wide array of factors which come together to modulate goal-directed action. Neurobiologists are uncovering the widely distributed collection of neural circuits which underlie the various aspects of goal-directed motivation. This has led to the identification of limbic and midbrain regions including the Ventral Tegmental Area (VTA), Nucleus Accumbens (NAcc), and Ventral Pallidum (VP) which appear to be critical for invigorating effortful behavior. Additionally, cortical regions such as the Anterior Cingulate Cortex (ACC) and medial Prefrontal Cortex (mPFC) are crucial for comparing costs and benefits which becomes important when one is faced with several potential response choices. In addition to the basic work being done in animal models, clinicians and psychiatrists using modern brain imaging methods have started to uncover some of the neurobiological correlates of impairments in goal-directed motivation commonly seen in many forms of psychopathology, including schizophrenia and depression. Currently, the unprecedented technical arsenal of neuroscience tools available to researchers makes it an extremely exciting and fruitful time to be studying a question which has captivated researchers for nearly a century.

All mobile organisms are faced with the universal challenge of living in a world in which the resources needed for survival may be limited in number and unevenly dispersed throughout the environment. Obtaining essential resources often requires one to overcome obstacles which inherently contain many different kinds of costs to the organism. When seeking food, water, or potential mates, one might be faced with any number of these costs, including: a physical distance one must traverse, the height of an obstacle one must climb, the number of responses one must make, or the commitment of time one must invest. Goal-directed motivation represents the set of processes which allows an organism to weigh these costs against potential benefits of obtaining a goal. It has been recognized by researcher for a long time that motivation serves two important functions, as it provides both a directional influence on behavior and also has an activational or energizing effect as well as (Duffy, 1957, Hebb, 1955); and more recent work has started to describe the underlying neurobiological substrates of both the directional processes (Kim et al., 2013, Kimchi and Laubach, 2009) as well as activational processes (Anaclet et al., 2009, Pfaff et al., 2012) and whereas the directional component of motivation guides behavior toward a specific goal and away from competing actions (Dickinson & Balleine, 1994), the activational component of motivation provides the energy or vigor needed to overcome the physical costs standing between the animal and its goal. This activational influence on motivation is reflected in the likelihood of initiation, and the speed, vigor and persistence of an action (Floresco, 2015, Salamone, 1992, Salamone and Correa, 2002, Salamone et al., 2012).

The most general way in which the concept of directional motivation is used is to say that animals pursue positive stimuli (e.g. food, water, sex, etc.) and avoid negative stimuli (e.g. painful conditions, predators, stress) (Salamone, Yohn, López-Cruz, San Miguel, & Correa, 2016). A more specific definition of the concept of directional motivation is the processes which cause animals to choose one specific class of behavior to engage in at a given time over all others (i.e. Feeding, Drinking, Mating, Aggressive Behavior, etc.). This concept proves useful in that it allows researchers to attempt to figure out the physiological and environmental variables which influence animals to engage in one class of behaviors over another (e.g. feeding as opposed to drinking). This usage helps to explain observations such as when animals choose to pursue food following a long period of food deprivation, as it is the directional influence of motivation which leads the animal to pursue food while forgoing pursuits of other behaviors. This is unsurprising as there are distinct neural circuits which control food seeking as opposed to something like thirst (Kelley et al., 2005, Oka et al., 2015). There has been an extensive amount of research aimed at understanding what circulating hormones and brain regions are responsible for directional motivational effects for feeding (Belgardt, Okamura, & Brüning, 2009), thirst (Johnson & Thunhorst, 1997), as well as sexual behavior (Davidson, 1966), and other social behaviors (Hong et al., 2014, Wang et al., 2014). We point readers to recent reviews of this literature (Sternson, 2013), as an extensive discussion of these directional effects are beyond the scope of the present review. In the present review, we focus on situations in which subjects are food restricted and working for food rewards (i.e. experimentally manipulated to be directed towards food), and we examine how different types of costs a subject must overcome to obtain the food reward alters both activational aspects of behavior and the choice of what specific action to take to obtain reward.

As animals are deprived of necessary resources their behavior changes in a number of ways: (a) there is often an increase in general locomotor activity, (b) an increase the likelihood of performing actions known to lead to that deprived resource, (c) and an increase in the speed, vigor, and the persistence of these goal directed actions (Floresco, 2015, Salamone, 1992, Salamone and Correa, 2002, Salamone et al., 2012). These changes in behavior are thought to reflect changes in the activational or energizing effects of motivation. It is this activational or energizing influence of motivation which allows animals to overcome the costs standing between them and the goal for which they are working. In this review, we focus specifically on what is known about the neural substrates that influence how the costs of responding affect the activational aspects of motivated behavior. We also examine what is known about the neural machinery involved in processing information about different types of costs that enter into the cost-benefit computation that guides choices about how to allocate effort in situations in which there is more than one response option that could lead to the desired resource.

How does motivation properly guide an organism through the environment to overcome obstacles and meet needs necessary for survival? Current theories suggest that animals incorporate information from many different levels and perform cost-benefit computations which allow for adaptive decision making. A typical laboratory experiment in which a rat has learned to press a lever for a food reward serves as an excellent example of how this might work. A fully sated rat will make a very small number of lever presses for food. The few lever presses it does make will be made slowly with many pauses in between presses, and the rat will spend a substantial amount of time engaging in other behaviors such as exploring the chamber and grooming itself. The same animal’s behavior will look very different when its access to food has been restricted. Both the number of lever presses made as well as the rate/vigor of those responses are highly correlated with the percent body weight loss induced by the food restriction (Collier, 1969, Collier and Levitsky, 1967, Marwine and Collier, 1971). In these two scenarios the cost of responding is constant (i.e. the same number of lever presses is required in both situations), but the benefit or value of the food differs greatly. The difference between the cost and the benefit of pressing in each particular condition determines the direction of behavior (lever pressing and not exploring or grooming/etc.) as well as the intensity or vigor (response rate of the lever presses) with which the behaviors are executed.

Research over that last 5 decades shows that there are many factors which influence the cost-benefit decision making processes. These factors include environment factors (such as local food availability, time of day, or temperature), an animal’s experiential history (whether it was trained on a continuous or intermittent schedule of reinforcement), physiology (circulating hormone levels) and internal biological clocks (e.g. location in a circadian rhythm (Antle & Silver, 2015). Fig. 1 illustrates a conceptual model of how all of these factors might act in concert in a hierarchical manner to modulate goal-directed motivation by influencing the underlying cost-benefit decision making processes, and provides examples of these different factors influencing motivation (Simpson & Balsam, 2016). As shown in this figure, this model posits that the physiological state of the organism, the environment, and past history/learning of the organism interact to influence the representation of costs and benefits that determine the specific types of behavior at any given time. Moreover, the information about the costs and benefits are compared in a cost-benefit computation which then influences the selection and vigor of behavior. We present Fig. 1 to suggest one possible model of how goal-directed motivation may work, and to provide a context in which to place this review. We do not attempt to state which brain regions are definitively involved in specific stages of the Cost/Benefit computation process, rather we examine an array of studies which focus on the cost input to this computation. In doing so we compare 3 different kinds of costs: Effort, Time, and Risk. We bring together these three separate lines of investigation to identify both the overlapping and distinct neurobiological substrates for processing these costs.

The purpose of this review is to summarize and synthesize a number of varying studies which examine different types of motivated behavior through the framework of motivated behavior as relying on a cost-benefit computation to give rise to both the direction and vigor of behavior. The direction and vigor of behavior represent the final behavioral output which one can measure, and a number of studies are reviewed which have been performed to understand the neural locations at which manipulations to the region impact either directional or activational aspects of behavior. Additionally, we give a primary focus to studies which have examined motivated behavior through various forms of cost-benefit decision making. In this review, we systematically focus on studies which have manipulated one of the factors which goes into the cost-benefit calculation: cost, as this represents one critical side of the cost-benefit computations that guide motivated behavior. We first provide a summary of the behavioral data that demonstrates animals’ ability to process information related to various types of costs. We then discuss the more recent work examining the neurobiology of the activational effects of motivation. We finish by reviewing an array of studies aimed at understanding the neurobiological underpinnings of cost-benefit decision making by specifically focusing on studies which employed manipulations of three types of response costs: (1) effort, (2) time, and (3) risk. In doing so we describe studies which have employed neural manipulations such as various types of lesions, as well as locally delivered pharmacological manipulations. While we also discuss a number of results from systemic pharmacology studies, we have limited this to results which further inform our understanding of the neural circuits underlying the different behavioral processes discussed in the review.