Tutorial
A tutorial on adaptive design optimization

https://doi.org/10.1016/j.jmp.2013.05.005Get rights and content

Highlights

  • Introduces the reader to adaptive design optimization (ADO).

  • ADO is a Bayesian statistical framework for optimizing experimental designs.

  • Provides the conceptual, theoretical, and computational foundations of ADO.

  • Serves as a practical guide to applying ADO to simple cognitive models.

Abstract

Experimentation is ubiquitous in the field of psychology and fundamental to the advancement of its science, and one of the biggest challenges for researchers is designing experiments that can conclusively discriminate the theoretical hypotheses or models under investigation. The recognition of this challenge has led to the development of sophisticated statistical methods that aid in the design of experiments and that are within the reach of everyday experimental scientists. This tutorial paper introduces the reader to an implementable experimentation methodology, dubbed Adaptive Design Optimization, that can help scientists to conduct “smart” experiments that are maximally informative and highly efficient, which in turn should accelerate scientific discovery in psychology and beyond.

Introduction

Imagine an experiment in which each and every stimulus was custom tailored to be maximally informative about the question of interest, so that there were no wasted trials, participants, or redundant data points. Further, what if the choice of design variables in the experiment (e.g., stimulus properties and combinations, testing schedule, etc.) could evolve in real time as data were collected, to take advantage of information about the response the moment it is acquired (and possibly alter the course of the experiment) rather than waiting until the experiment is over and then deciding to conduct a follow-up?

The ability to fine-tune an experiment on the fly makes it possible to identify and capitalize on individual differences as the experiment progresses, presenting each participant with stimuli that match a particular response pattern or ability level. More concretely, in a decision making experiment, each participant can be given choice options tailored to her or his response preferences, rather than giving every participant the same, pre-selected list of choice options. As another example, in an fMRI experiment investigating the neural basis of decision making, one could instantly analyze and evaluate each image that was collected and adjust the next stimulus accordingly, potentially reducing the number of scans while maximizing the usefulness of each scan.

The implementation of such idealized experiments sits in stark contrast to the current practice in much of psychology of using a single design, chosen at the outset, throughout the course of the experiment. Typically, stimulus creation and selection are guided by heuristic norms. Strategies to improve the informativeness of an experiment, such as creating all possible combinations of levels of the independent variables (e.g., three levels of task difficulty combined with five levels of stimulus duration), actually work against efficiency because it is rare for all combinations to be equally informative. Making matters worse, equal numbers of participants are usually allotted to each combination of treatments for statistical convenience, even the treatments that may not be informative. Noisy data are often combatted in a brute-force way by simply collecting more of them (this is the essence of a power analysis). The continued use of these practices is not the most efficient use of time, money, and participants to collect quality data.

A better, more efficient way to run an experiment would be to dynamically alter the design in response to observed data. The optimization of experimental designs has a long history in statistics dating back to the 1950s (e.g.,  Atkinson and Donev, 1992, Atkinson and Federov, 1975, Berry, 2006, Box and Hill, 1967, Lindley, 1956). Psychometricians have been doing this for decades in computerized adaptive testing (e.g.,  Weiss & Kingsbury, 1984), and psychophysicists have developed their own adaptive tools (e.g.,  Cobo-Lewis, 1997, Kontsevich and Tyler, 1999). The major hurdle in applying adaptive methodologies more broadly has been computational: Quantitative tools for identifying the optimal experimental design when testing formal models of cognition have not been available. However, recent advances in statistical computing (Doucet et al., 2001, Robert and Casella, 2004) have laid the groundwork for a paradigmatic shift in the science of data collection. The resulting new methodology, dubbed adaptive design optimization  (ADO,  Cavagnaro, Myung, Pitt, & Kujala, 2010), has the potential to more broadly benefit experimentation in cognitive science and beyond.

In this tutorial, we introduce the reader to adaptive design optimization. The tutorial is intended to serve as a practical guide to apply the technique to simple cognitive models. As such, it assumes a rudimentary level of familiarity with cognitive modeling, such as how to implement quantitative models in a programming or graphical language, how to use maximum likelihood estimation to determine parameter values, and how to apply model selection methods to discriminate models. Readers with little familiarity with these techniques might find Section  3 difficult to follow, but should otherwise be able to understand most of the other sections. We begin by reviewing approaches to improve inference in cognitive modeling. Next we describe the technical details of adaptive design optimization, covering the computational and implementation details. Finally, we present an example application of the methodology for designing experiments to discriminate simple models of memory retention. Readers interested in more technical treatments of the material should consult (Cavagnaro et al., 2010, Myung and Pitt, 2009).

Section snippets

Not all experimental designs are created equal

To illustrate the importance of optimizing experimental designs, suppose that a researcher is interested in empirically discriminating between formal models of memory retention. The topic of retention has been studied for over a century. Years of research have shown that a person’s ability to remember information just learned drops quickly for a short time after learning and then levels off as more and more time elapses. The simplicity of this data pattern has led to the introduction of

The nuts and bolts of adaptive design optimization

In this section we describe the theoretical and computational aspects of ADO in greater detail. The section is intended for readers who are interested in applying the technique to their own cognitive modeling. Our goal is to provide readers with the basic essentials of implementing ADO in their own experiments. Fig. 3 shows a schematic diagram of the ADO framework that involves a series of steps. In what follows we discuss each step in turn.

Illustrative example

To further illustrate how ADO works, we will demonstrate its implementation in a simulation experiment intended to discriminate between power and exponential models of retention. The effectiveness of ADO for discriminating between these models has been demonstrated in simulation (Cavagnaro et al., 2010) and in experiments with human participants (Cavagnaro et al., 2011, Cavagnaro et al., 2009). The intention of this demonstration is provide an easy to understand companion to the technical and

Limitations

While ADO has the potential to significantly improve the efficiency of data collection in psychological sciences, it is important that the reader is aware of the assumptions and limitation of the methodology. First of all, not all design variables in an experiment can be optimized in ADO. They must be quantifiable in such a way that the likelihood function depends explicitly on the values of the design variables being optimized (Myung & Pitt, 2009, p. 511). Consequently, ADO is not applicable

Conclusions

In this article, we provided a tutorial exposition of adaptive design optimization (ADO). ADO allows users to intelligently choose experimental stimuli on each trial of an experiment in order to maximize the expected information gain provided by each outcome. We began the tutorial by contrasting ADO against the traditional, non-adaptive heuristic approach to experimental design, then presented the nuts and bolts of the practical implementation of ADO, and finally, illustrated an application of

Acknowledgments

This research is supported by National Institute of Health Grant R01-MH093838 to J.I.M. and M.A.P. The C++ code for the illustrative example is available upon request from authors.

References (64)

  • A. Atkinson et al.

    Optimal design: experiments for discriminating between several models

    Biometrika

    (1975)
  • M.A. Beaumont et al.

    Adaptive approximate Bayesian computation

    Biometrika

    (2009)
  • R.E. Bellman

    Dynamic Programming (reprint edition)

    (2003)
  • D.A. Berry

    Bayesian clinical trials

    Nature Reviews

    (2006)
  • G. Box et al.

    Discrimination among mechanistic models

    Technometrics

    (1967)
  • K.P. Burnham et al.

    Model Selection and Multi-Model Inference: A Practical Information—Theoretic Approach

    (2010)
  • H.P. Carlin et al.

    Bayes and Empirical Bayes Methods for Data Analysis

    (2000)
  • D. Cavagnaro et al.

    Optimal decision stimuli for risky choice experiments: an adaptive approach

    Management Science

    (2013)
  • D.R. Cavagnaro et al.

    Adaptive design optimization: a mutual information based approach to model discrimination in cognitive science

    Neural Computation

    (2010)
  • D. Cavagnaro et al.

    Discriminating among probability weighting functions using adaptive design optimization

    Journal of Risk and Uncertainty

    (2013)
  • D. Cavagnaro et al.

    Model discrimination through adaptive experimentation

    Psychonomic Bulletin & Review

    (2011)
  • D.R. Cavagnaro et al.

    Adaptive design optimization in experiments with people

    Advances in Neural Information Processing Systems

    (2009)
  • K. Chaloner et al.

    Bayesian experimental design: a review

    Statistical Science

    (1995)
  • A.B. Cobo-Lewis

    An adaptive psychophysical method for subject classification

    Perception & Psychophysics

    (1997)
  • D. Cohn et al.

    Improving generalization with active learning

    Machine Learning

    (1994)
  • D. Cohn et al.

    Active learning with statistical models

    Journal of Artificial Intelligence Research

    (1996)
  • T. Cover et al.

    Elements of Information Theory

    (1991)
  • A. Doucet et al.

    Sequential Monte Carlo Methods in Practice

    (2001)
  • A. Gelman et al.

    Bayesian Data Analysis

    (2004)
  • J. Gill

    Bayesian Methods: A Social and Behavioral Sciences

    (2007)
  • P.D. Grünwald

    A tutorial introduction to the minimum description length principle

  • R.K. Hambleton et al.

    Fundamentals of Item Response Theory

    (1991)
  • Cited by (99)

    • Challenges and Solutions to the Measurement of Neurocognitive Mechanisms in Developmental Settings

      2023, Biological Psychiatry: Cognitive Neuroscience and Neuroimaging
    • Improving the Reliability of Cognitive Task Measures: A Narrative Review

      2023, Biological Psychiatry: Cognitive Neuroscience and Neuroimaging
    View all citing articles on Scopus
    View full text