Abstract
Studies of ongoing, rapid motor behaviors have often focused on the decision-making implicit in the task. Here, we instead study how decision-making integrates with the perceptual and motor systems and propose a framework of limited-capacity, pipelined processing with flexible resources to understand rapid motor behaviors. Results from three experiments show that human performance is consistent with our framework: participants perform objectively worse as task difficulty increases, and, surprisingly, this drop in performance is largest for the most skilled performers. As well, our analysis shows that the worst-performing participants can perform equally well under increased task demands, which is consistent with flexible neural resources being allocated to reduce bottleneck effects and improve overall performance. We conclude that capacity limits lead to information bottlenecks and that processes like attention help reduce the effects that these bottlenecks have on maximal performance.
Significance Statement
Behavior results from perception, decision-making, and motor control. These processes can be modelled serially, but we must often decide about future actions in parallel with ongoing actions. We use computational models to demonstrate that simply perceiving, deciding, and moving in parallel inevitably produces performance-limiting interference. We see that the best performers in rapid motor behavior tasks are counter-intuitively the most affected by increasing task demands, which is consistent with sharing attention between pipelined processes.
Introduction
Reaction time tasks are a classic way to explore perceptual, decision-making, and motor processes, as response times reflect the summation of these processes. Response times can be as fast as 100 ms for a preplanned motor response (Scott, 2016) or as long as 700 ms when identifying emotions from a person’s face (Nook et al., 2015). This “decide-then-act” paradigm has supplied a number of experiments used to study decision-making during motor actions, including motor lotteries (Trommershäuser et al., 2008; Wu et al., 2009; Nagengast et al., 2011; Jarvstad et al., 2013; Neyedli and Welsh, 2013, 2015; Neyedli and LeBlanc, 2017; Adkins et al., 2022), reaching tasks (Gallivan et al., 2011, 2015; Cos et al., 2014; Enachescu et al., 2021; Ulbrich and Gail, 2021), foraging tasks (Diamond et al., 2017), and intercept tasks (Barany et al., 2020; Fooken and Spering, 2020; Michalski et al., 2020).
Less is known about how perceptual, decision-making, and motor processes interact to generate behavior. Even in the “decide-then-act” paradigm, models such as the affordance competition hypothesis (Cisek, 2007) and embodied decision-making (Cisek and Pastor-Bernier, 2014; Lepora and Pezzulo, 2015) argue that these processes work in parallel, but the situation becomes much more complex when we must “decide-while-acting” (Michalski et al., 2020). Sight-reading piano music, for example, requires the pianist to perceive notes on the sheet music and decide how to play subsequent key strokes while actively playing the musical piece. Future actions are planned while current actions are executed, making future actions a component of the present decision (Gallivan et al., 2018), and providing an opportunity for parallel processing and feedback between processes (Lepora and Pezzulo, 2015). Cisek and Kalaska argued that the parallel characteristics of motor decision-making in humans evolved in response to an environment that requires ongoing, interactive behavior (Cisek and Kalaska, 2010). Parallel processing allows individuals to respond to a stimulus faster when it is the second in a sequence compared with being presented on its own, especially when the two stimuli are compatible, but it breaks down under difficult speeded-response conditions and serial processing is forced (Kahneman, 1973). We ask the question, at what point do the brain’s time-sensitive processing capabilities limit the performance of rapid motor behaviors?
One important consideration is that perceptual, decision-making, and motor processes interact and interfere with one another (Passingham, 1996; Scherbaum et al., 2015; Künstler et al., 2018; Carroll et al., 2019; Thura, 2020; Raßbach et al., 2021; Ulbrich and Gail, 2021). This interference effect has been explained by models that posit either a structural bottleneck or a shared pool of central resources that restrict the brain’s processing capacity (Navon and Miller, 2002). Structural bottleneck models propose that a particular process, such as stimulus perception (Broadbent, 1957) or response selection (Deutsch and Deutsch, 1963), has a fixed capacity. This particular process is the bottleneck and it limits performance. Shared resource models posit instead that the brain must allocate resources, such as attention (Kahneman, 1973) or cognitive maps (Franconeri et al., 2013), between possible activities. For these models, the effects of task characteristics and limited resources can result in any cognitive process becoming the bottleneck.
In this paper, we introduce an information-processing framework with shared resources to interpret how humans produce rapid motor behaviors. We explore how the limited-capacity and pipeline characteristics of this framework lead to information bottlenecks that dictate system performance. We evaluate three models in this framework against datasets of human participants hitting moving objects as part of time-constrained motor behavior tasks in a virtual workspace using an interactive robotic platform (Singh and Scott, 2003). The limited-capacity, pipelined framework correctly predicts that changes in both task demands (e.g., more and faster objects in the workspace) and behavioral demands (e.g., distractor objects adding arbitrary decision rules) lead to complex changes in performance, including that highly skilled individuals tend to be more impacted than those with less skill. These results highlight how changes in a behavioral task may or may not create a bottleneck in performance that partially reflects the individual’s preexisting level of performance.
Materials and Methods
Participants
A total of 2303 human participants (1564 males and 738 females, aged 8–93, 2084 right-handed and 195 left-handed) have performed tasks in the Object-Hit (OH) family (Table 1). Of these, 985 participants were recruited through the “Safe to Play” longitudinal research program.
Participant demographics for rapid motor behavior tasks
Participants were recruited from the communities of Kingston, Ontario and Calgary, Alberta. Participants in Calgary, Alberta were recruited through and tested at the Benson Concussion Institute, Group23 Sports Medicine Clinic (formerly WinSport Medicine Clinic). Test procedures were approved by the Research Ethics Boards of Queen’s University, Providence Care, and the University of Calgary. All participants gave their written and informed consent to have their data collected for research purposes. This written and informed consent was obtained from the individual’s parents/guardians if the individual was under the age of 17.
Healthy human participants had no significant neurologic impairments, were medically stable, had normal or corrected to normal visual acuity, had no ongoing musculoskeletal injuries of the upper limb, and were able to understand task instructions (Tyryshkin et al., 2014; Bourke et al., 2016). All participants who performed the turbo variants of the task had their visual acuity confirmed to be 20/40 uncorrected or better (Mang et al., 2018).
For all three experiments, we compared trial performances for participants who have completed both of the two tasks being compared. This resulted in 487 participants for Experiment 1 (212 male, 432 right-handed, age range 18–93), 1662 participants for Experiment 2 (1290 male, 1510 right-handed, age range 8–55), and 38 participants for Experiment 3 (17 male, 34 right-handed, age range 19–71).
Tasks
The four tasks used in this experimental design have been validated as being suitable for measuring rapid motor behaviors in healthy human participants, as producing reliable test results, and as provoking no learning effects between trials (Mang et al., 2018; Simmatis et al., 2020; Moulton et al., 2022). Although when we compare tasks we might say that one task increases the demands on a particular process, we note that all four tasks require perception, decision-making, and motor control to occur in some proportion. None of these tasks, OH in particular, is made easier by the absence of demands for one of these processes.
Object-hit
The OH task was designed to examine an individual’s ability to generate quick and accurate arm movements throughout the workspace (Tyryshkin et al., 2014). Participants were seated in a bilateral Kinarm exoskeleton lab (Kinarm, Kingston, Ontario; Singh and Scott, 2003; Fig. 1A), which provides full gravitational support to the upper limbs while allowing movement in the horizontal plane (Tyryshkin et al., 2014).
The Object-Hit and Object-Hit-and-Avoid tasks. A, Kinarm Exoskeleton robot, used for the OH and OHA tasks. B, Sample screen shot from the OH task. Objects are dropped in 10 evenly spaced horizontal bins that are not visible to the participant. Arms are shown only for illustrative purposes, only the green paddles are visible during the task. C, Sample screen shot from the OHA task. Objects have eight possible shapes: two shapes indicate target objects and the other six shapes indicate distractor objects.
Participants were instructed to use their hands, represented visually as 5 cm wide green paddles, to hit virtual red objects (circular, 2-cm diameter) that moved toward them in the horizontal plane. These red objects represent targets and appear from 10 different bins (not visible to the participant), with each bin dropping an object in random order before repeating (Fig. 1B). Each bin drops 30 objects over the length of the task, resulting in 300 total targets. Positions and velocities of the hands and objects are recorded with a sampling frequency of 200 Hz and the Kinarm exoskeleton generated a 50-ms force pulse to simulate feedback of contact between paddles and objects. For full details, see the original paper by Tyryshkin et al. (2014).
The OH task challenges participants to produce peak performance. Trials begin with objects created at a slow rate and with slow speeds. During this easy phase, the task is “data-limited” and participants’ performance is limited by the input from the task (Wickens, 1981). Throughout the trial, however, objects are steadily created at a faster rate and with faster speeds, increasing perceptual, decision, and motor processing demands (Vidal et al., 2015), ensuring that participants always finished a trial by being overwhelmed. Once they are overwhelmed, participants’ performance is resource-limited as they trade off speed, accuracy, and effort to maximize their global reward rate (Fitts, 1954; Wickens, 1981; Reynaud et al., 2020). We also saw that participants missed many more peripheral targets when they were overwhelmed (Moulton et al., 2022), potentially maximizing reward rate by minimizing movement between targets and consistent with a preference for less effortful actions (Cos et al., 2014).
Object-Hit-and-Avoid
The Object-Hit-and-Avoid task (OHA) is similar to OH, but now the red objects have different shapes. Two of the eight shapes indicated targets to hit while the remaining shapes indicated distractors to avoid (Fig. 1C). This introduces an arbitrary decision rule and requires some movements to be inhibited. Participants were again seated in a Kinarm exoskeleton lab to record their performance and instructed to hit as many of the 200 targets as possible without hitting the 100 distractor objects (Bourke et al., 2016).
The rate at which objects appeared on the screen and the speed with which they moved steadily increased throughout the trial in an identical manner to OH. Haptic feedback was provided by the exoskeleton on paddle contact with targets. Distractors passed through the paddle with no haptic feedback to provide an instantaneous reminder that it was a distractor. For full details, see the original paper by Bourke et al. (2016).
Turbo Object-Hit and Turbo Object-Hit-and-Avoid
The Turbo Object-Hit (TOH) and Turbo Object-Hit-and-Avoid (TOHA) tasks are speeded versions of the original tasks, with faster moving objects and smaller paddles (2 vs 5 cm). The original intent behind the task was to remove the ceiling effect that was specifically observed when elite athletes performed OH and OHA (Mang et al., 2018).
Participants used a bilateral Kinarm end-point lab (Kinarm, Kingston, Ontario; Fig. 2A). Between 2011 and 2013, participants were seated in front of the robot; beginning in 2014 the robot was modified to allow participants to stand. Participants held on to two handles, which moved in the horizontal plane to interact with an augmented reality system displayed in the same plane (Mang et al., 2018).
The Turbo tasks. A, Kinarm Endpoint robot, used for the TOH and TOHA tasks. B, Sample screen shot from the TOHA task illustrating the smaller paddle size. Arms and traces, indicating the participant’s movement with the left (purple) and right (red) hands, were not visible during the task.
In all other respects the turbo tasks were the same as the original tasks: TOH has 300 targets, TOHA has 200 targets and 100 distractors, and TOHA chooses two shapes out of eight to indicate these targets every trial. See Mang et al. (2018) for full details.
Data analysis
Event codes for object creations and object/paddle contacts were extracted from all trial records and object/paddle contacts were assessed as hits or misses depending on the object’s trajectory at termination (consistent with Tyryshkin et al., 2014). These events were converted into instantaneous target creation and target hit rates using two iterations of a Kolmogorov–Zurbenko filter with a sliding window with 5 s width.
The difference between target creation and target hit rates was calculated at each frame and the median difference over a window with 5 s width was calculated. The beginning of a participant’s overwhelmed phase was then determined to be the latest frame at which the participant was overwhelmed, characterized as when the median rate difference, (1), minus the median absolute deviation, (2), was <0.1 Hz:
where X is a univariate dataset
We characterized participants’ performance during the overwhelmed phase by calculating their steady-state rate: the average target hit rate from the initial frame of the overwhelmed phase to the frame where the trial’s target creation rate peaked. The initial frame for this overwhelmed phase was then brought forward and the steady-state rate recalculated if (1) doing so increased the participant’s steady-state rate or (2) if the median rate difference minus the median absolute deviation was <0.1 Hz at any point in the preceding 5 s. This type of rate parameter is useful for characterizing performance for tasks involving ongoing movement and has been previously used in studying the latency and duration of keystroke sequences for typewriting tasks (Sternberg et al., 1978).
We interpret a participant’s steady-state rate for a trial as representing their peak performance and underlying sensorimotor abilities. Although it is difficult to untangle the effects of factors such as motivation, we believe that this interpretation is justified for three reasons. First, if motivation levels determined steady-state rates within a trial then we would expect to see some participants either substantially increase or substantially decrease their target hit rate during the overwhelmed phase as their motivation fluctuated, but this was not observed. Second, if motivation levels determined steady-state rates between trials then we would expect some participants to exhibit a wide range of steady-state rates across trials. Instead, all four tasks have been validated as evoking consistent performance levels and steady-state rates between tests. Third, if motivation levels determine steady-state rates between participants, this role seems to be consistent both within and between trials for single participants. We would therefore consider this kind of effect as a part of the participant’s underlying ability to perform rapid motor behavior tasks.
Statistical analysis
Because of the problems associated with frequentist null hypothesis testing (Nuzzo, 2014; Kruschke and Liddell, 2018; Amrhein et al., 2019), we assessed the significance of our results using Bayesian data analysis techniques drawn from Kruschke’s textbook on the subject (Kruschke, 2015)
Assessing significant differences
For each experiment we assessed whether or not the difference in steady-state rates was significantly different from 0. For this analysis we assumed that these differences were drawn from a normal distribution with mean μ and SD σ. The question of interest, then, is whether or not 0 Hz is a credible value for the parameter μ (Kruschke, 2015).
Our prior distribution for the parameters was informed by the population-level differences for the tasks. For two arbitrary tasks, our prior belief for μ was centered on the difference in average steady-state rates between population with a SD equal to the precision of the baseline task’s steady-state rates; our prior belief for σ was a uniform distribution over a wide range of values based on the baseline task’s SD:
We established the region of practical equivalence (ROPE) to capture the range of differences that we will consider as practically equivalent to a difference of 0 Hz. The size of this ROPE was calculated as the intertrial variability of steady-state rates for a single participant; in other words, the degree of variability that we would expect to see from a participant performing the baseline task a second time.
Next, we then calculated the difference in steady-state rates for each participant as our evidence. A Markov chain Monte Carlo (MCMC) method was then used to apply Bayesian inference and calculate our posterior belief regarding the values for parameters μ and σ. The representativeness of the MCMC chains was assessed by ensuring overlap of the values assumed by each chain, ensuring overlap of the 95% high-density interval (HDI) from each chain, and verifying convergence of the chains by computing the Brooks-Gelman-Rubin statistic (or shrink factor). The accuracy of the MCMC chains was assessed by ensuring that the chains were not autocorrelated over a series of lags, by computing the effective sample size of the four chains, and by considering the Monte Carlo standard error of the chains (Kruschke, 2015).
Finally, we described our posterior belief for these parameter values by reporting the mode parameter value as well as the 95% HDI of the posterior distribution. The 95% HDI for the parameter μ was then compared against the ROPE around 0 Hz to determine whether or not participants’ difference in steady-state rates between the two tasks are different enough from 0 Hz for practical purposes.
Bayesian linear regression
A similar analysis was performed for each experiment when we conducted a Bayesian linear regression to relate an participant’s steady-state rate in Task 2 with their steady-state rate in Task 1 (Kruschke, 2015). For this analysis, we assumed that steady-state rates for Task 2 were drawn from a t-distribution with centrality parameter μ, scale parameter σ, and normality parameter ν. We further assumed that the centrality parameter for any given participant’s distribution was linearly related to their steady-state rate from Task 1:
Our prior beliefs for these parameters were:
For the linear regression parameters, these prior beliefs implied that participants’ steady-state rates would be roughly equal between the two tasks.
We verified for each regression that the estimated values for σ and ν implied that a linear model was plausible for relating steady-state rates between tasks. The relevant parameters for distinguishing between the three different models, however, were β0 and β1. As the y-intercept parameter, we were interested in determining whether or not β0 was sufficiently different from 0 Hz for practical purposes. We established the ROPE for this value as (–0.12, 0.12), which represented the range of y-intercepts estimated for the Multiplicative Workload systems from our simulations. As the slope parameter, we wanted to determine whether or not β1 was sufficiently different from 1 for practical purposes. The ROPE for this value was (0.95, 1.05) representing the range of slopes estimated for the Additive Workload systems from our simulations.
An MCMC method was then used to apply Bayesian inference and produce estimates for all four parameters. We described our resulting posterior beliefs for these parameter values by again reporting the mode value and the 95% HDI of the posterior distribution. The 95% HDIs for the parameters β0 and β1 were then compared against their respective ROPEs to assess whether or not they reflected a practical difference.
Simulations
We developed three models within the limited-capacity, pipelined framework to explore the principles of information processing for interactive behavior. The time required for the perceptual, decision-making, and motor processes involved depends on the task’s complexity, but can be effectively reduced via parallel processing (Cisek and Kalaska, 2010; Thura and Cisek, 2014; Scott, 2016; Cisek and Thura, 2018; Cisek, 2019; Hommel et al., 2019). Our models each contain two modules that process input simultaneously but are connected in series, leading to pipelined processing. The ability for the modules to process input simultaneously matches the view that parallel processing for goal-directed actions is both more efficient and more biologically plausible (Pezzulo and Ognibene, 2012). While we commonly consider three processes for motor behaviors (perception, decision-making and motor control), the influence of pipelined processing can be easily observed with only two modules. The environment provides input to the system, each module processes input according to its own limited processing capacity, and the second module produces the system output (Fig. 3).
The limited-capacity, pipelined processing framework. A, Our framework envisions information processing systems whose modules have limited processing capacities and that must process input serially while remaining capable of having different modules working on different inputs in parallel. Three separate models are considered within this framework, and we simulate a number of systems for each model to generate testable predictions. B, For all limited-capacity, pipelined processing systems, the system output rate is equal to the environment input rate as long as the rate of environmental input is smaller than system capacity. In this case, the limitation on the output rate is outside the system. C, Once the rate of environmental input exceeds system capacity, the limitation on the system’s output rate is within the system, and the system’s output rate is now equal to the capacity of the system’s most restricted module.
Kahneman posited that the amount of attention available increases with task difficulty. The brain performs processes in parallel by allocating attention between possible processes until demands exceed resources, at which point a degree of serial processing is forced (Kahneman, 1973). Since we are interested in peak performance, we focus on describing how peak performance occurs once task demands exceed the participant’s available attention. Our framework predicts that once the environmental input rate exceeds the system’s processing capacity, the system module with the smallest capacity will form a bottleneck (Fig. 3B). Building on the idea of limited-capacity, pipelined processing, we consider three separate models within this framework: Additive Workload, Multiplicative Workload, and Flexible Capacity. These models are distinguished by two characteristics: the effect of increased task demands and the nature of their processing capacities.
Task demands. Demands are either additive or multiplicative in nature. For systems from the Additive Workload model, more difficult tasks impose an additive workload on system modules: demands are placed on top of existing task input without modifying it.
By contrast, for systems from the Multiplicative Workload and Flexible Capacity models, more difficult tasks impose a multiplicative workload on system modules: demands increase existing task inputs by a common factor across the board.
Processing capacities. Systems from the Additive Workload and Multiplicative Workload models have fixed processing capacities for each module. They cannot adapt their capacity to changing task characteristics and require queueing when demands exceed capacity (Navon and Miller, 2002). This type of capacity represents neural resources that are fixed according to function.
Systems from the Flexible Capacity model have some of their overall capacity in the form of a shared pool of resources. These resources can be flexibly allocated between modules, meaning that sharing must occur when task demands exceed capacity (Navon and Miller, 2002). This is akin to neural resources being allocated to one specific process and improving its performance.
Task specifications
Our task environments were defined by the difficulty for each module to perform its processing task. We expressed this difficulty as a parameter d ∈ [0,1] where 0 indicated the highest level of difficulty and 1 indicated the lowest level of difficulty. A task environment was completely defined by the difficulty for the two modules’ processing tasks and can be thought of as a single point in the space [0, 1] × [0, 1].
We defined 100 task environments by varying the difficulty of each module’s processing from an initial value of 0.1 to a final value of 1.0, incrementing by a step of size 0.1. These task difficulty factors were deterministic and are given in Table 2. Since the task (1.0, 1.0) represents the easiest possible task environment, we use this as the baseline task for our comparisons.
Grid of module difficulty factors
Note that all task environments required both modules to perform their role in producing output. No task environment was made easier by dropping the requirement to process input via one of the modules.
System specifications
We generated individual systems from each of our three models. Module processing capacities were positive real numbers drawn from normal distributions with the means listed in Table 3 and a SD of 0.05 Hz. Each individual system was entirely defined by this pair of positive real numbers and can be expressed as a single point in the space
Grid of mean module capacities
A system was generated for each cell in the table where the absolute difference in module capacities was less than a third of the first entry, preventing one module’s capacity from being completely dominated by the other’s bottleneck. This resulted in 324 systems. For systems from the Flexible Capacity model, each module’s processing capacity was reduced by 0.5 Hz to create a central pool of resources without increasing the system’s cumulative processing capability. This quantity was chosen arbitrarily to illustrate the principle of flexible allocation.
These initial module capacities were modified throughout the simulation according to that module’s task difficulty parameter. The final module capacities for systems in the Additive Workload model were calculated according to Equation 3, while final module capacities for both the Multiplicative Workload and Flexible Capacity models were calculated according to Equation 4:
Module capacities for systems from the Flexible Capacity model were further modified by assigning the shared processing capacity to the modules. This shared capacity is assigned according to the following rules:
If the difference between module capacities is larger than the central capacity pool, assign all central processing capacity to the module with the smaller capacity.
Otherwise, assign the central processing capacity between the two modules so that the final module capacities are equal.
Simulation specifications
All 324 systems were simulated in all 100 task environments with 200 discrete frames recorded per second to match the recording fidelity of the Kinarm lab in the human experiments described below. Environmental input, capacity limits, and system output were all calculated every 100 frames and the input and output values for the intermediate frames were filled in using linear interpolation. The simulation calculated the environment’s steadily increasing input rate throughout the trial according to Equation 5, where t is the frame number, n is an additive noise factor drawn from the standard normal distribution
To model fluctuating task demands over time, a frame-specific difficulty level was randomly generated for each module by sampling from a normal distribution with a mean of that module’s baseline difficulty and a SD of 0.05. Similarly, frame-specific module capacities were sampled from a normal distribution with a mean of the module’s base- line capacity and a SD of 0.05 Hz to model biological noise. These frame-specific difficulty levels and module capacities were then used to calculate the final module capacities for each frame according to Equation 3 or Equation 4 as appropriate.
The total output of a system for frame t was calculated as the minimum of the environment input, input(t), and the final module capacities,
The per-frame input and output were recorded separately for each trial for further analysis.
Code accessibility
The code/software described in the paper is freely available online at https://github.com/richard-moulton/Information-Bottlenecks-in-Rapid-Motor-Behaviour.
Results
Simulation predictions
A consistent observation for all three models was that systems reached a steady-state rate, at which point the same output was produced regardless of how much additional input the environment provided. This steady-state rate was the result of the system’s bottleneck: the smaller module capacity (in this example, Module B) aligned with the steady-state rate and solely determined the system’s overall output (Fig. 4A). A task with different difficulties, however, could see another module (Module A) become the system’s bottleneck (Fig. 4B).
Predictions made by simulations. A, Input rate (black) and output rate (blue) are shown over time for the baseline task. The processing capacity for each module is shown (gray) and the system’s steady-state output rate (red) is equal to the lower module capacity. B, The same system performs a second task where Module A’s processing task is made more expensive; Module A’s capacity for this task is correspondingly reduced. C–E, Scatter plot comparisons of systems’ steady-state rates are made for all three models of systems. Although 100 task environments were simulated, for clarity only a single comparison is presented between Task 1, the baseline task, and Task 2, a more difficult task. The unity line is shown in black and regression lines are shown in red.
Examining the different types of task demands, we observed that an additive workload leads to all systems being affected to the same degree. This results in regression slopes near 1, which indicates that less capable systems would be unable to perform tasks that were too difficult (Fig. 4C). By contrast, a multiplicative workload leads to the best-performing systems being most affected by increased task demands. This results in regression slopes <1, but all systems are able to perform the more difficult task (Fig. 4D,E).
Assessing the different models, we observed that structural bottlenecks lead to systems being affected by increased difficulty up until 0 Hz. This results in y-intercepts near or below 0 Hz (Fig. 4C,D). The existence of a shared pool of resources allows the worst-performing systems to mitigate task demands, which increases the average performance across all systems. For example, systems in the Flexible Capacity model that had a high processing rate for Module A largely used their central resources in Task 1 to improve their Module B capacity and maximize performance. When Task 2 reduced Module A’s performance, these systems were able to redistribute between modules, mitigating these new demands. By contrast, systems with a high processing rate for Module B had largely used their central resources to improve Module A’s capacity in Task 1 and additional resources were therefore not available during Task 2 to counter the reduced performance of Module A. The overall effect of central resources was a positive y-intercept and the linear regression crossing the unity line in Quadrant I of the Cartesian plane (Fig. 4E). Note that the predictions made by the Additive Workload and Flexible Capacity models are only valid to the right of the unity line in Quadrant I: neither model predicts output that is either negative or increased for the more difficult task.
All three models predict that systems will produce lower steady-state rates for more difficult tasks, but they make diverging predictions about how this occurs:
The Multiplicative Workload and Flexible Capacity models predict increasing task difficulty will affect the best-performing participants more than worse-performing participants, the slope for the linear regression will be less than 1.
By contrast, the Additive Workload model predicts that all participants are equally affected by increased task difficulty, the slope for the linear regression is equal to 1.
The Flexible Capacity model predicts that there is a positive level of performance at which participants will be unaffected by this additional task difficulty, the y-intercept for the linear regression is positive.
The Additive Workload and Multiplicative Workload models predict that equal performance between tasks will only occur at 0 Hz. This demonstrates an inherent advantage to developing processing capacity in the form of shared resources because it reduces the effect of variable task difficulties on individual system performance.
Only the Flexible Capacity model predicts that participants’ drop in performance with increased task demands will fall more heavily on more skilled participants and that some participants will actually be unaffected by increased task demands. We will show over the following sections that only the Flexible Capacity model is consistent with human performance in our rapid motor behavior tasks.
Experiment 1: Object-Hit and Object-Hit-and-Avoid
For OH and OHA, our two-module system roughly represents the perceptual and decision processes with Module A and the motor processes with Module B. Although the baseline task OH does place demands on all processes, the bottleneck is likely the motor system’s ability to hit targets. Compared with OH, OHA adds additional pressures to identify, select and hit specific targets and avoid distractors. This shifts the bottleneck toward the perception/decision module, since objects’ shapes must be reliably perceived and an arbitrary decision rule applied to assess target/distractor status.
We established a ROPE for differences in steady-state rates of (–0.30, 0.30 Hz) to account for the between-trial variability for participants performing OH (see Materials and Methods, Statistical analysis). Steady-state rate differences between OH and OHA that fall within this region are considered to be practically equivalent to 0 Hz. This intertrial variability is independent of performance level, allowing us to use the same ROPE for all participants.
Performance decreases with task difficulty
Scatter-plotting participants’ steady-state rates in OH and OHA highlights that the latter is the more difficult task (Fig. 5A). The mean difference in steady-state rates between the tasks is 0.733 Hz, which is 2.4 times further from 0 Hz than the boundary of the ROPE (Fig. 5B). Taken over the course of a whole trial, ∼140 s, this represents nearly 103 fewer targets hit by a participant. Although OHA does have fewer targets than OH, this cannot account for the entire drop in performance.
Comparing performance between OH and OHA. A, Participants are scatter-plotted according to their steady-state rates in OH and OHA. The unity line is shown in black and is extended by the region of practical equivalence in gray. Regression for participants’ performance is shown in red. B, Probability distribution functions are plotted to represent the region of practical equivalence for OH (green), the distribution of steady-state rates seen in healthy control participants for OH (blue), and the posterior distribution for differences in steady-state rates between OH and OHA (red). The mean of the posterior distribution is highlighted by the red vertical line.
To determine whether participants’ steady-state rate differences are significantly different from 0 Hz, we modelled these differences as being drawn from a normal distribution with mean μ and SD σ, then used a Markov chain Monte Carlo (MCMC) method to estimate these parameters (Fig. 5B; for details, see Materials and Methods, Statistical analysis). The 95% HDI for μ is entirely below the ROPE for the OH task, indicating that the mean difference in participants’ steady-state rates between OH and OHA is practically different from 0 Hz. Notably, most participants perform OH before OHA so any population-level intertask learning effect would be in favor of performance in OHA.
The size of performance drop scales with ability
We split participants into quartiles based on their steady-state rate in the OH task and performed the same Bayesian data analysis for each quartile. We found participants in the lower quartiles are least likely to show a significant drop in performance while participants in the top-performing quartile are most likely to show a large drop in performance (Fig. 6A); in fact, the mean difference in steady-state rates for participants in the top-performing quartile is outside the 95% confidence interval for healthy control participants in the OH task.
Drops in performance between OH and OHA are especially significant for better performers. A, Participants are grouped by their performance quartile in OH and the analysis from Figure 5B is repeated. The mean of the posterior distribution is highlighted by the red vertical line for each quartile and is compared with the mean of the posterior distribution for all participants, shown with the black vertical line. B, Distributions of value estimates are shown for the intercept and slope parameters from the Bayesian linear regression relating participants’ steady-state rates for OH and OHA. Specifically highlighted are the mode value of the distribution and the distribution’s 95% high-density interval.
We used Bayesian linear regression to test our models’ predictions (see Materials and Methods, Statistical analysis). We estimated the t-distribution’s scale and normality parameters directly to confirm that a linear model reasonably linked participants’ steady-state rates. The t-distribution’s centrality parameter was estimated as a linear function of the participant’s steady-state rate in the OH task according to Equation 7:
The estimates for the linear regression parameters are shown in Figure 6B.
We estimated the value for the linear regression slope parameter, β1, as 0.545 (95% HDI 0.508–0.586). This is clearly far from 1: not only is the 95% HDI for this estimate entirely below the ROPE, (0.95, 1.05), but so is the entire distribution of estimates. In the context of the tasks, a regression slope of 0.545 indicates that if a participant improved their performance in OH by 10 extra targets, we would only expect them to improve in OHA by five or six extra targets. This estimate range for β1 provides convincing evidence that skilled participants are more affected by an increase in task difficulty than less skilled participants. This is consistent with the Multiplicative Workload and Flexible Capacity models, but contradicts the Additive Workload model.
Some participants are unaffected by additional task difficulty
The observation that participants who perform best on OH experience the largest drop in performance for OHA is especially striking because we see that the worst participants on baseline tasks often perform within the ROPE when shifting to OHA—that is many of the worst participants perform just as well on the harder task as they would be expected to perform on a second trial of the baseline OH task (Figs. 5A, 6A). To test this observation quantitatively, we revisit our Bayesian linear regression’s intercept parameter, β0, and determine whether it is larger than 0 Hz; our ROPE around 0 Hz was (–0.12 Hz, 0.12 Hz). The estimate for β0 is centered on 0.162 Hz (95% HDI 0.075–0.230 Hz) with 81.4% of the estimate’s probability mass greater than the ROPE. This provides strong evidence that there is a performance level at which participants are not affected by an increase in difficulty in the task, some level of ability exists independent of task difficulty. This result is consistent with the Flexible Capacity model, but contradicts the Additive Workload and Multiplicative Workload models.
Experiment 2: Turbo Object-Hit and Turbo Object-Hit-and-Avoid
In this experiment, we considered participants who performed the original tasks’ “turbo” variants (Mang et al., 2018). We saw a systematic shift downwards in steady-state rates from TOH to TOHA (Fig. 7; Table 4).
Experimental results
Comparing performance between tasks. Participants are scatter-plotted according to their steady-state rates in three cross-task comparisons: (A) TOH and TOHA and (B) OH and TOH. The unity line for each comparison is shown in black and extended by the region of practical equivalence in gray. Regression for participants’ performance is shown in red.
A similar quartile-based effect is seen for this pair of tasks (Fig. 7A). Quantitatively, the 95% HDI for the slope parameter is entirely less than the ROPE around 1 and the 95% HDI for the y-intercept parameter is entirely greater than the ROPE around 0 Hz (Table 4). These results are in line with Experiment 1 and are consistent with the Flexible Capacity model.
Experiment 3: Object-Hit and Turbo Object-Hit
Experiments 1 and 2 both assessed pairs of tasks where the additional difficulty was a result of the distractor objects, thus the dimension of increased task difficulty was the same for both experiments. In Experiment 3, we assessed our models against the performance of participants who performed both OH and TOH. In this pair of tasks, the increased difficulty occurs because the participant must use substantially smaller paddles to hit faster moving targets. This allows us to investigate whether the evidence found for limited capacity, pipeline processing generalizes beyond cases where an arbitrary decision rule is added.
We observed that steady-state rates were consistently lower in TOH than in OH (Fig. 7; Table 4). Comparing SSRs between these two tasks produces the same quartile-based effect (Fig. 7B) and the 95% HDI for the slope parameter is entirely less than the ROPE around 1, consistent with multiplicative task difficulty. Although the 95% HDI for the y-intercept parameter overlaps with the value of 0 Hz, >90% of the estimate’s probability mass is greater than the ROPE (Table 4). This lends moderate support for the Flexible Capacity model.
Discussion
We show that human performance in ongoing, rapid motor behaviors is consistent with a limited-capacity, pipelined processing framework. We found that task demands increased multiplicatively, with the greatest impact on those with better baseline performance. Our results further suggest that neural resources are flexibly allocated between processes to maximize performance.
Principles of pipelined processing
Our systems featured two modules connected in serial, but their ability to process input in parallel is a massive advantage over strictly serial processing: output increases by a factor of n for an n-module system with similar capacity modules. Our framework demonstrates that once processing limits are reached, task and system characteristics conspire to produce bottlenecks which result in the steady-state behavior that is the marker of an overwhelmed system. We distinguish our systems’ pipelined processing from parallel-distributed processing because of the serial nature of these processes: perception must precede deliberation, which in turn must precede deliberate action, even if these processes do overlap and interact. These serial aspects open the door for any one aspect to become the limiting factor on the system’s output rate by becoming a bottleneck.
We investigated additive and multiplicative task demands, e.g., adding a framing phase versus increasing each target’s difficulty, with our models showing that multiplicative demands affect the best performing systems the most in absolute terms. This suggests that performance drops say more about a task’s demands than the system performing it. Concretely, if one participant suffers a larger performance drop than another, we should not necessarily conclude that the former participant found the new task harder than the latter participant. Instead, the former participant might have had a larger portion of their performance that was vulnerable to redistributing the shared resources in response to changed task demands.
We also contrasted structural bottlenecks against shared resources. A system with structural bottlenecks is at the mercy of task demands and will lose ground to increased demands until 0 Hz. The ability to shift resources between processes allows the system to safeguard performance against increased task demands instead.
Human behavior matches that of our model
Human performance across behavioral tasks matched the Flexible Capacity model’s characteristics in a number of ways.
First, we saw evidence of parallel processing as in parallel models for goal-directed actions (Pezzulo and Ognibene, 2012). Tasks such as OH reward parallel processing because of repeated stimuli and sequential decision-making. Consider the following reaction-time chain: 150 ms to make a go/no-go decision using an arbitrary rule based on perceptual stimuli (Thorpe et al., 1996), 600 ms to accumulate evidence between two alternatives (Ratcliff and McKoon, 2008), and 1000 ms to make a simple 35 cm arm movement (Morasso, 1981). If these processes occurred serially, then the sum of 1.75 s per object would limit steady state rates in OHA to ∼0.57 Hz. Most participants comfortably exceed this ceiling (Fig. 5) with the top performers approaching a steady-state rate of 2 Hz. This improved performance represents a clear evolutionary incentive for different brain functions to process in parallel, as has been proposed for the human sensorimotor system (Cisek and Kalaska, 2010; Cisek, 2019). In addition to the benefit of speed, parallel processing also allows information from current actions to shape an ongoing decision-making process (Lepora and Pezzulo, 2015).
Second, we saw that additional demands in behavioral tasks have a multiplicative nature, leading to the best participants seeing the largest drop in performance on harder tasks. Although we might expect that the most skilled participants could minimize the effect of additional demands, our results demonstrate that the most skilled participants are most affected precisely because they are processing the most targets to begin with. The worst performers are completely unaffected by the increased difficulty of targets they were not hitting anyways.
Third, we saw that the least skilled participants were often able to reproduce their steady-state rates in more difficult tasks and showed no practical drop in performance. This did not come from underlying skill but is consistent with allocating shared resources between processes according to task demands. We do not believe that this effect can be explained by participants regressing to the mean, since even if the outlier performers did regress, we would still expect the whole sample to include relatively extreme performance levels on the more difficult task. This suggests that there is a base level of motor performance that is very resilient to outside task demands.
Comparing OH tasks with other tasks
Perceptual, decision-making, and motor control processes have historically been studied using reaction-time tasks, including inter-response interval tasks such as key presses after sequential stimuli (Kahneman, 1973). In the motor decision literature, “decide-then-act” tasks such as motor lotteries (Trommershäuser et al., 2008; Wu et al., 2009; Jarvstad et al., 2013; Neyedli and Welsh, 2013, 2015; Neyedli and LeBlanc, 2017), foraging tasks (Diamond et al., 2017), and intercept tasks (Barany et al., 2020; Fooken and Spering, 2020) have been used to characterize decisions about movements. These tasks have used external time limits to provoke reaction-time like processing to measure the brain’s capacity in motor decisions, but provide minimal opportunity for the brain to exploit its ability for parallel processing.
Rapid motor behavior tasks allow the same motor decision capabilities to be explored, but in the context of ongoing, interactive behavior that results in interference between processes. A participant’s steady-state rate acts as an ongoing and sustained reaction time, which is useful for “decide-while-acting” tasks where humans have been shown to prioritize the global reward rate (Bogacz et al., 2010; Balci et al., 2011). As a practical matter, rapid motor behavior tasks also include a speed-up benefit over other experimental tasks. As a comparison, 300 separate objects can be presented to the participant and either acted on or not within 3 min; by contrast, 300 presentations in a motor lottery task, allowing for 5 s of reset time in between trials, would require 30 min.
Implications for attention
A key feature of our Flexible Capacity model is the central pool of resources that can be flexibly allocated to whichever module is most stressed by the task’s demands. In the brain, this pool of resources is commonly termed attention. Although notoriously difficult to define, one description of attention from the literature is that it is the brain’s ability to selectively process multi-channel sensorimotor input when continuous, interactive behavior is required to achieve our goals (Hommel et al., 2019). This ability is goal-related, e.g., people will not notice the unexpected gorilla in a scene when instructed to count the number of basketball passes they observe (Simons and Chabris, 1999). Importantly, the degree to which an individual can pay attention acts as a capacity limit on their ability to act. Not all neural processes require attention, Schneider and Shiffrin distinguish between automatic and controlled processes (Schneider and Shiffrin, 1977; Shiffrin and Schneider, 1977) and Kahneman distinguishes between System 1 and System 2 (Kahneman, 2011). But processes that do require attention are subject to capacity limits and are prevented from occurring freely in parallel. These are the processes which are subject to constraints in our model.
Kahneman proposed that attention is generated through mental effort and shared across tasks to meet demands and allow parallel processing. Once the maximum attention capacity is generated, the brain’s neural processes interfere with each other’s demand for resources. This leads to some goals being delayed or dropped to permit others to be achieved (Kahneman, 1973). Our results support the existence of a shared pool of neural resources, though we do not speculate about how this pool is made available to neural processes. In all three experiments there is significant support for our linear regression’s y-intercept term being larger than 0. This is predicted by the Flexible Capacity model and indicative of shared resources being re-deployed according to changed task demands. By contrast, the Multiplicative Workload model cannot explain this since structural bottlenecks cannot, by definition, adapt. Our results extend Kahneman’s conception of neural processes competing with one another for resources beyond the “decide-then-act” paradigm by providing evidence for the same type of interference effects during “decide-while-acting” tasks.
We propose that one of the functions of attention during motor tasks is to compensate for goal-related demands. Since attention is a finite resource, this enforces a selectivity between processes that directly explains the serial processing effects that occur in limited capacity systems (Kahneman, 1973). This selectivity in assigning attention between processes causes information bottlenecks to occur between perception and action. The types of rapid motor behavior subtasks requiring attention include: actively searching for decision-relevant information (Orquin and Mueller Loose, 2013), storing perceived stimuli into visual short-term memory (Künstler et al., 2018), planning movement toward different targets (Gallivan et al., 2011), and deciding between alternative motor actions (Haith et al., 2015), especially under uncertainty (Brandstätter and Körner, 2014) or when precise selection is required (Franconeri et al., 2007). Associating behavioral bottlenecks with attention demands is consistent with a number of stimulus response frameworks including: the selection of inputs and selection of outputs (Treisman, 1969), the stimulus set and response set (Broadbent, 1971), and figural emphasis and response selection (Kahneman, 1973).
Considering that these bottlenecks occur when processes must compete for limited attention, it is important to note that controlled processes can be learned to the point where they become automatic processes and thereby reduce their demand for resources. Such learning reduces the instantaneous demands on shared resources during tasks and allows this limited capacity to be directed toward processing important stimuli and controlling increasingly complex responses (Shiffrin and Schneider, 1977). This kind of learning and the dedication of controlled and automatic processes to specific kinds of tasks may also support hierarchical control for motor actions. For example, there is evidence that skilled typists can type roughly twice as fast with two hands as they can with a single finger, but for student typists this is reversed (Gentner, 1983). A possible explanation for this type of effect is that each hand is controlled by separate hierarchical control processes which alternate between the controlled process of sequence planning and the automatic process of sequence execution (Sternberg et al., 1990). Our model is consistent with this explanation, since we can model a shift toward automatic processes as an increase in that module’s capacity. Although we have not previously found evidence for the strategic alternation of hand movements in the OH family of tasks (Moulton et al., 2022) there is clear scope for our model to be extended to account for the bimanual nature of many rapid motor behaviors tasks.
In interpreting the results of Experiment 3, we might expect that additional stresses on a bottleneck process could not be remedied by reallocating shared resources since they are already devoted to the bottleneck process during the easier task. Indeed, the y-intercept term β0 for Experiment 3’s linear regression is the only parameter whose 95% HDI overlapped the associated ROPE. The estimated y-intercept of 0.418 Hz does suggest some effect, however, and this would be possible if the initial distribution of shared resources was split between processes, leaving some available to be reassigned once the bottleneck process is further stressed. Related to this possibility is the question “what proportion of the brain’s processing capacity is a result of the brain’s different regions versus as a result of a central pool of neural resources?” This is an important parameter in shared resource models (Navon and Miller, 2002) and could be the subject of future research. Another question posed by shared resource models is the degree to which resources can be shared across all processes. Theories that posit the existence of multiple resources propose that different pools of resources are available to processes based on the processing stage, encoding, and modality of the process (Wickens, 2008). Task difficulty in these theories reflects both the degree to which individual processes exhaust their own resources and the amount of competition that exists for common resources (Navon and Gopher, 1978). Future work could aim to characterize the source and degree of difficulty in rapid motor behavior tasks using performance operating characteristic curves (Navon and Gopher, 1978, 1979; Wickens, 1981).
Future applications
Our Flexible Capacity model assumed that all systems have a fixed number of shared neural resources, regardless of individual module capacities. Task comparisons could establish the proportion between structural and shared neural resources for rapid motor behavior tasks as well as the interparticipant variability for this proportion. Our model also assumed that shared neural resources can be split across processes arbitrarily and dynamically in response to changing task demands. The validity of this assumption could be tested by using tasks that measure the effects of split attention and shifts of attention on participants’ steady-state rates. Our model could also be extended to incorporate well-known cognitive systems, such as working memory, with a view to examining how these specific systems interfere with other neural processes.
Our model may also be useful for interpreting results in clinical settings. Specifically, stroke may impact motor function and/or cognitive function so that changes in performance between OH and OHA may be much more variable: slow performance in OH could lead to no change in OHA (motor impairments only), or dramatic impairment in OHA (motor and cognitive/perceptual impairments). The general prediction is that any neurologic disease or injury where attention helps the individual counter an impairment will leave that individual susceptible to performance-impacting bottlenecks if behavioral demands are increased, such as a person with difficulties walking but now having to also navigate in a crowded room.
Our model also suggests that athletic coaches and rehabilitation therapists should determine an individual’s bottleneck process to allow targeted training that will produce outsized rewards on improved performance. This is the pay-off of learning motor skills, e.g., the autonomous stage of motor learning (Schmidt and Wrisberg, 2004). Since motor skills involve perception, decision-making, and motor control processes (Wolpert et al., 2011), reducing a particular process’s requirement for attention frees up shared resources to assist another process. This is the initial challenge of learning to drive where attentional demands must be used to perform the basic requisite motor actions (i.e., switch on the turn indicator, press the brake and turn right) at the expense of attending to perceptual and decision processes. This is analogous to the benefit seen in cognitive decision-making when shifting from deliberative to procedural decision-making (Redish, 2013). Similarly, this principle is exemplified by the fact that airplane control instruments, such as joysticks and pedals, are easy to use, ensuring the pilot can focus on perceiving instruments, listening to air-traffic control, and/or directing the co-pilot.
Acknowledgments
Acknowledgments: We acknowledge that Queen’s University is situated on traditional Anishinaabe and Haudenosaunee Territory. We thank Dr. Carolyn Emery, principal investigator for the research Grant “Safe to Play: A longitudinal research program to establish best practice in the prevention, early diagnosis and management of sport-related concussion in youth ice hockey players,” for making data available to us for analysis.
Footnotes
S.H.S. is the co-founder and Chief Science Officer of Kinarm, which commercializes the robotic technology used in the present study. All other authors declare no competing financial interests.
This work was supported by funding from the Natural Sciences and Engineering Research Council of Canada, the Canadian Institutes of Health Research, Own the Podium Canada, the Canadian Academy of Sport and Exercise Medicine Research Fund, the Canadian Sport Institute Calgary, the Ontario Research Foundation, the Queen's University Faculty of Engineering and Applied Science, and the Walter C. Sumner Foundation.
This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International license, which permits unrestricted use, distribution and reproduction in any medium provided that the original work is properly attributed.
References
In this issue
