No evidential value in samples of transcranial direct current stimulation (tDCS) studies of cognition and working memory in healthy populations

Abstract

A substantial number of studies have been published over the last decade, claiming that transcranial direct current stimulation (tDCS) can influence performance on cognitive tasks. However, there is some skepticism regarding the efficacy of tDCS, and evidence from meta-analyses are mixed. One major weakness of these meta-analyses is that they only examine outcomes in published studies. Given biases towards publishing positive results in the scientific literature, there may be a substantial “file-drawer” of unpublished negative results in the tDCS literature. Furthermore, multiple researcher degrees of freedom can also inflate published p-values. Recently, Simonsohn, Nelson and Simmons (2014) created a novel meta-analytic tool that examines the distribution of significant p-values in a literature, and compares it to expected distributions with different effect sizes. Using this tool, one can assess whether the selected studies have evidential value. Therefore, we examined a random selection of studies that used tDCS to alter performance on cognitive tasks, and tDCS studies on working memory in a recently published meta-analysis (Mancuso et al., 2016). Using a p-curve analysis, we found no evidence that the tDCS studies had evidential value (33% power or greater), with the estimate of statistical power of these studies being approximately 14% for the cognitive studies, and 5% (what would be expected from randomly generated data) for the working memory studies. It is likely that previous tDCS studies are substantially underpowered, and we provide suggestions for future research to increase the evidential value of future tDCS studies.

Introduction

Transcranial direct current stimulation (tDCS) is an affordable, non-invasive technique used to electrically stimulate the brain. Typically, two electrode pads (a positively charged anode and a negatively charged cathode) are placed on the participant. A relatively weak current (typically 1–2 mA) then runs from the cathode to the anode. This current is thought to change the resting membrane potential of neurons, resulting in hyperpolarization (less activity) under the cathode, and hypopolarization (more activity) under the anode (Bindman et al., 1962, Nitsche and Paulus, 2000), along with long-term potentiation/depression-like plasticity after (respectively) anodal/cathodal stimulation (Stagg & Nitsche, 2011). This technique has been used to study changes in motor cortex excitability and motor learning (Nitsche et al., 2003). In addition to research on motor processes, tDCS has more recently been applied to a number of other domains, including altering cognitive (Sparing, Dafotakis, Meister, Thirugnanasambandam, & Fink, 2008) and emotional (Boggio, Zaghi, & Fregni, 2009) function in typical populations, as a therapy for stroke sufferers (Fregni, Boggio, Mansur, et al., 2005) and individuals with mental illness (Boggio et al., 2008), and to augment athletic training in high-level athletes (Banissy & Muggleton, 2013). In addition, it is inexpensive (with commercially available devices costing less than $200) and safe (Bikson et al., 2016). If such a simple device can be used to improve performance in all of these domains, its application could revolutionize brain science and rehabilitation. Therefore, it is of critical importance that claims regarding its effectiveness be examined and scrutinized.

There has been a substantial increase in the number of published tDCS studies, including studies of tDCS and cognitive processes, over the last five years.¹ Intuitively, the sheer number of manuscripts claiming an effect of tDCS on cognitive processes would suggest that this method clearly modulates behavior. However, there are multiple reasons why the number of published papers in a field is not always indicative of evidential value.² First, null results are typically not submitted for publication (the “file-drawer problem”, Rosenthal, 1979), as editors are more likely to accept positive versus null results (Franco, Malhotra, & Simonovits, 2014). Although there are several studies showing a significant effect of tDCS on cognitive processes, there could be a larger number of unpublished studies that found no effect.

Second, decisions made during data analysis can falsely inflate significance. For example, researchers often have a number of decisions to make when collecting and analyzing data, including deciding on how many participants to test, whether (and how) to remove outliers, data transformations (e.g., using raw vs percentage scores, whether to normalize, etc.), which dependent variables should be reported or analyzed, whether to include covariates, whether to use median splits, type of statistical analysis to use, etc. Although it is best practice to decide on the analysis pipeline before data collection, these decisions can be made during data analysis and lead to potential biases (Gelman and Loken, 2013, Gelman and Loken, 2014, Kunda, 1990). Furthermore, researchers may generate hypotheses after, not before, testing (HARKing: Hypothesizing After the Results are Known – see Kerr, 1998). For example, researchers may initially hypothesize that a specific manipulation influences task performance. Not finding the predicted effect, the researcher can probe the data to examine if dividing the population into subsets (e.g., sex differences, median splits on a different variable) results in a significant effect of the manipulation. HARKing, using multiple analysis pipelines, and other practices (such as adding participants until a significant outcome is reached) all significantly increase the odds of a false positive finding (see Simmons, Nelson, & Simonsohn, 2011).

Third, underpowered studies are at risk of, not only false negatives (Type II errors) but also false positives (Type I errors) and overestimation of true effect sizes (Type M errors, see Gelman & Carlin, 2014). Small, underpowered studies can only detect large effects. If the true effect size is small or non-existent, only studies that overestimate the true effect size via randomness will cross significance thresholds (the “winner's curse”, see Button et al., 2013, Ioannidis, 2008). Therefore, a number of factors can lead to a substantial literature with limited evidential value.

Given these concerns, an important question is how to assess evidential value in the literature. Simonsohn, Nelson, and Simmons (2014b) have developed a method for testing the evidential value of a literature by examining reported p-values. Using this method, one first finds the distribution of significant (p < .05) p-values in a selection of published studies, ignoring any p-values that are not statistically significant. Next, one compares this distribution of p-values from the selected literature to distributions that would be expected given different effect sizes. For example, the distribution of p-values from a series of studies with no effect is expected to be flat, such that the same number of p-values should be observed between .12 and .13 or .74 and .75. Importantly, this is also true for significant p-values. If there is no true effect (d = 0), then there should be the same number of p-values from .01 to .02 as there are from .04 to .05. In the presence of a real effect, this p-value distribution should be right skewed, such that there are more observed p-values between .00 and .01 than between .04 and .05. On the other hand, given certain questionable research practices, researchers may stop collecting data or do exploratory analyses once they have crossed the critical p < .05 boundary. This practice, at times called p-hacking, would result in a distribution of p-values with left skew (more p-values closer to .05 than .00).

Previous papers have examined the effects of tDCS on various aspects of cognition using traditional meta-analysis techniques, with varying results. In the working memory domain, Brunoni and Vanderhasselt (2014) found that tDCS led to improvements in reaction time, but not accuracy. Hill, Fitzgerald, and Hoy (2016) found a small effect of anodal tDCS on offline (but not online) reaction time, while Mancuso, Ilieva, Hamilton, & Farah, 2016 found that left dorsolateral prefrontal cortex (DLPFC) tDCS improved performance when paired with training, with no other meta-analyses being significant. Dedoncker, Brunoni, Baeken, and Vanderhasselt (2016) reported that anodal, but not cathodal, DLPFC stimulation altered performance on cognitive tasks, whereas Horvath, Forte and Carter (Horvath, Forte, & Carter, 2015) found no evidence that tDCS influenced performance on cognitive tasks (though see Price & Hamilton, 2015 for discussion). However, Price, McAdams, Grossman, and Hamilton (2015) selected the language studies from Horvath, Forte & Carter, and reported that tDCS can influence performance on language tasks – though this finding has been questioned as well (see Westwood, Olson, Miall, Nappo, & Romani, 2017).

One issue with traditional meta-analytic techniques is that they can only be done on available data. Given publication biases for reporting significant results throughout the scientific literature, meta-analyses that pull from the published literature will be made primarily (or solely) of significant results, and are not able to take into account unpublished null findings. The most commonly used method to correct for publication bias is Trim and Fill (Duval and Tweedie, 2000a, Duval and Tweedie, 2000b), in which one examines a funnel plot for asymmetry, “trims” the studies with the largest effect sizes until symmetry is obtained, and then replaces the trimmed studies maintaining a symmetric funnel plot. However, Trim and Fill has been found to be inadequate at detecting publication bias. Simonsohn, Nelson, and Simmons (2014a) simulated datasets with varying true effect sizes (from d = .0 to .8), and then examined the estimated effect sizes from Trim and Fill compared to the p-curve analysis. They found that Trim and Fill vastly overestimated the effect size, such that Trim and Fill meta-analyses reported an effect size of about .7 from datasets with a true effect size of zero. In contrast, p-curve analyses provided an accurate estimate of the true effect size.

Given its advantages, we used a p-curve analysis to examine the evidential value of tDCS studies of cognition. This method has been used both to demonstrate that certain sets of studies have evidential value (e.g., studies of syntactic priming, see Mahowald, James, Futrell, & Gibson, 2016) and others do not (e.g., studies on “far transfer” in working memory with active controls and “power posing”, see Melby-Lervåg et al., 2016, Simmons and Simonsohn, 2016). Therefore, we conducted two sets of meta-analyses to examine the effect of tDCS on cognitive processes. In our first set, we randomly selected 30 studies that could be included in the p-curve analysis that examined the effect of tDCS on cognition. Then, to account for potential criticisms of the first meta-analysis (study heterogeneity, possibilities of a biased sample), we conducted a second set of meta-analyses which included all studies examining the effects of anodal stimulation on working memory in a recently published meta-analysis (Mancuso et al., 2016). In summary, both meta-analyses did not find evidence that tDCS influences behavior on cognitive and working memory tasks.