Most sciences search for lawful data patterns or regularities to serve as the building blocks of theories (e.g., Bunge, 1967; Carnap, 1995; Popper, 2002). Generally, such data patterns must not be singular findings (i.e., chance findings) but instead be replicable by other researchers under similar conditions in order to be scientifically meaningful (Popper, 2002, p. 23). With this fundamental scientific premise as background, it is understandable that many researchers have become concerned that a surprisingly large number of published results cannot be replicated in independent studies and hence appear to represent chance findings or so-called false positive results (Baker and Penny, 2016; Ioannidis, 2005b; Pashler and Harris, 2012; Simmons et al., 2011; Zwaan et al., 2018). For example, only less than 30% of results in social psychology and about 50% in cognitive psychology appear to be reproducible (Open Science Collaboration, 2015). Similarly, the replication rate of 21 systematically selected experimental studies in the social sciences published between 2010 and 2015 in Nature and Science was estimated to be only about 62% (Camerer et al., 2018). Low replication rates have also been reported in medical research (Begley and Ellis, 2012; Ioannidis, 2005a; Prinz et al., 2011): for example, researchers at the biotechnology firm Amgen tried to confirm findings in 53 landmark studies in preclinical cancer research, but were able to do so for only six cases (Begley and Ellis, 2012). The Reproducibility Project: Cancer Biology was set up to further explore the reproducibility of preclinical cancer research (Errington et al., 2014).

Statistical analysis of the replication scenario

The analyses in this manuscript address replication scenarios in which researchers conduct direct replications of studies that reported a statistically significant positive outcome. An example is the Open Science Replication Project (Open Science Collaboration, 2015), in which many independent research teams conducted high-powered studies attempting to directly replicate published results. Figure 1 depicts these scenarios together with all statistically relevant parameters that must be taken into account when computing the rate of replicating significant results (Miller, 2009; Miller and Ulrich, 2016; Miller and Schwarz, 2011). First, each original study tests either a true effect (i.e., $H_1$ is true) or a null effect (i.e., $H_0$ is true), with base rate probabilities ${\displaystyle \pi }$ and $1-\pi$, respectively, and these probabilities—sometimes called “pre-study probabilities” (Ioannidis, 2005b) or “prior odds” (Wilson and Wixted, 2018)—may vary across research fields (Wilson and Wixted, 2018). If the original study tests a true effect, its statistical power is $1-{\beta }_1$ and the Type 2 error probability is equal to ${\beta }_1$. Thus, the compound probability of examining a true effect and rejecting the null hypothesis is $\pi \cdot (1-{\beta }_1)$; this outcome is called a “true positive.” In contrast, if the original study tests a null effect, its Type 1 error probability is ${\alpha }_1$. Thus, the probability of testing a null effect and falsely rejecting $H_0$ is $(1-\pi )\cdot {\alpha }_1$; this outcome reflects a “false positive.” Note that, in keeping with accepted procedures for null hypothesis testing, we categorize studies as rejecting the null hypothesis or not based on an all-or-none comparison of computed p-values relative to an ${\displaystyle \alpha }$ level cutoff. Such a discrete categorization is, for example, how most journals currently evaluate statistical results in publication decisions and how replication success or failure has mainly been operationalized in empirical studies of replication rates (Camerer et al., 2018; Open Science Collaboration, 2015).

Figure 1

Download asset Open asset

Only true positives and false positives enter into replication projects. The statistical power $1-{\beta }_2$ and Type 1 error probability ${\alpha }_2$ of the replication studies might differ from those of the original study, especially because replication studies are usually designed to have much higher power than the original studies. Thus, the compound probability of examining a true effect that yields a significant effect in the original and in the replication study is $\pi \cdot (1-{\beta }_1)\cdot (1-{\beta }_2)$, whereas the compound probability of examining a null effect and finding significance in both the initial study and the replication study is $(1-\pi )\cdot {\alpha }_1\cdot {\alpha }_2$. From the above compound probabilities, the rate of replication of initially significant results, $RR$, can be computed as

(1) $RR=\frac{\pi \cdot (1-{\beta }_1)\cdot (1-{\beta }_2)+(1-\pi )\cdot {\alpha }_1\cdot {\alpha }_2}{\pi \cdot (1-{\beta }_1)+(1-\pi )\cdot {\alpha }_1}.$

Figure 2 illustrates this equation by showing how $RR$ depends on ${\displaystyle \pi }$, ${\alpha }_1$, and ${\beta }_1$ when the nominal alpha level and the statistical power of the replication studies are ${\alpha }_2=0.05$ and $1-{\beta }_2=0.90$. It can be seen in this figure that $RR$ increases gradually with ${\displaystyle \pi }$ from a minimum of ${\alpha }_2=0.05$ to a maximum of $1-{\beta }_2=0.90$. For $\pi =0$, the proportion of significant results can only represent false positives, so $RR$ necessarily equals ${\alpha }_2$. For $\pi =1$, in contrast, $RR$ merely reflects the power of the replication study. As is also illustrated in this figure, $RR$ grows faster when the power $1-{\beta }_1$ of the original studies is relatively large and their nominal alpha level ${\alpha }_1$ is relatively small. Note that $RR$ must gradually increase with ${\displaystyle \pi }$ from ${\alpha }_2$ to $1-{\beta }_2$ even if the power in the original study were 100%. It is also instructive to note that worst-case p-hacking would imply ${\alpha }_1\to 1$ and ${\beta }_1\to 0$. In this case it follows from Equation 1 that $RR$ approaches the line which runs from ${\alpha }_2$ at $\pi =0$ to $1-{\beta }_2$ at $\pi =1$.

Figure 2

Download asset Open asset

If p-hacking is performed in the original study, this would increase the Type 1 error rate above the nominal significance level ${\alpha }_1$ (usually 5%) to, for example, 10% or even higher. Thus, when a researcher examines a null effect, p-hacking increases the proportion of false positives. The extent of this increase depends on the details of the p-hacking strategy that is used, as we examine in detail below for different strategies. However, and crucially for the analyses that will follow, when a true effect is present, p-hacking also increases the nominal power $1-{\beta }_1$, for example, from 0.20 to 0.40 (i.e., power inflation, as mentioned above). With respect to the overall replication rate $RR$, this increase in power tends to compensate for the increased Type 1 error probability, making it difficult to determine intuitively how p-hacking would affect the replication rate $RR$. Fortunately, however, Equation 1 can be used to assess this issue quantitatively.

Besides assessing the effect of these factors on replicability, we will also report computations of the rate of false positives $FPR$, which is the proportion of false positive results among all significant results within a research area (sometimes also called false discovery rate or false positive report probability)

(2) $FPR=\frac{(1-\pi )\cdot {\alpha }_1}{(1-\pi )\cdot {\alpha }_1+\pi \cdot (1-{\beta }_1)}.$

In discussions about replicability—particularly replicability of published research findings—researchers often focus on this proportion (Button et al., 2013; Pashler and Harris, 2012) under the assumption that true positives are replicable but false positives are not. Therefore, it seems useful to include this rate in the analyses.

In the following, we model each of the four common p-hacking strategies that were described above. For each strategy, the inflated Type 1 error probability and the statistical power can be computed. These values are then inserted into Equation 1, which allows one to evaluate the effects of base rates and p-hacking on the replication rate, for both true and null effects. In addition, we examined the effects on $RR$ of different levels of ${\alpha }_1$ and statistical power, because—as mentioned above—several researchers have recently suggested lowering the ${\displaystyle \alpha }$ level or increasing power in order to increase the replicability of scientific results (Benjamin et al., 2018; Button et al., 2013). This allows one to judge how these suggested measures would combat low replicability and to compare their effects with those of p-hacking and base rate.

Selective reporting of significant studies

It has been often suspected that researchers tend to selectively report studies that yield positive results, that is, results that are in accordance with the researcher’s hypothesis (e.g., John et al., 2012; Rosenthal, 1979; Simmons et al., 2011; Zwaan et al., 2018). As noted earlier, this tendency will increase the number of reported false positives if researchers publish only the significant outcomes. This section models this p-hacking strategy and examines how it would influence the replication rate.

As a specific example, suppose that a researcher runs a series of experiments, each of which uses a slight variation of the same basic paradigm. This researcher terminates the series when a significant result emerges in support of the researcher’s hypothesis, and in this case the researcher tries to publish that result. However, if no significant result is obtained after conducting ${\displaystyle k}$ experiments, the researcher abandons the project and concludes that the hypothesis is false. Thus, this researcher has ${\displaystyle k}$ studies providing opportunities to test the hypothesis, and it would be misleading about the overall ${\displaystyle \alpha }$ level to publish only the significant outcome but not mention the non-significant attempts (Francis, 2014).

To model this scenario more concretely, assume that the researcher computes a z-value for the outcome of each experiment and considers the outcome to be statistically significant if any z-value exceeds a pre-specified criterion c (e.g., the critical z value of 1.96). In general, the probability of rejecting $H_0$ can be computed for $k\ge 1$ with

(3) ${\displaystyle Pr(\mathrm{R}\mathrm{e}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}{H}_0)=1-\prod _{i=1}^{k}Pr(Z_i\le c),}$

because the outcomes of the ${\displaystyle k}$ experiments are statistically independent if a new sample is recruited each time.

Figure 3 and Figure 3—figure supplement 1 depict the probability of rejecting $H_0$ for two- and one-sample tests, respectively, as a function of $k\in (1,2,4,6,8)$, $\alpha \in (5\%,0.5\%)$, and effect size $d\in (0.0,0.2,0.5,0.8)$. (Appendix 1 contains a detailed description of both tests.) In these examples, the group size is assumed to be $n=20$ (i.e., total $n=40$ for a two-sample test), a value that is typical for psychological research (Marszalek et al., 2011, Table 3), though there is evidence that sample sizes have increased recently in the field of social-personality psychology (Fraley and Vazire, 2014; Sassenberg and Ditrich, 2019). The lines for $d=0$ depict the effective Type 1 error probability. Of course, this probability is equal to ${\displaystyle \alpha }$ for $k=1$, but it increases with ${\displaystyle k}$ because of the greater number of opportunities for getting a significant result by chance when more studies are conducted. This increased Type 1 error probability is problematic because it tends to decrease replication rates (Benjamin et al., 2018). In the worst of these cases, the inflated Type 1 error rate attains a value of about 0.34 with $\alpha =5\%$ and $k=8$. As one expects, decreasing the nominal ${\displaystyle \alpha }$ level from 5% to 0.5% substantially diminishes the Type 1 error probability and thus correspondingly diminishes the probability of obtaining a false positive (Benjamin et al., 2018). Even for $k=8$ the Type 1 error rate would only be about 0.04 with this smaller nominal ${\displaystyle \alpha }$ level. It must be stressed, however, that a larger sample would be required for $\alpha =0.5\%$ than for $\alpha =5\%$ to achieve the same level of statistical power in both cases (Benjamin et al., 2018).

Figure 3 with 1 supplement see all

Download asset Open asset

The lines for $d>0$ reveal the statistical power to reject $H_0$ when it is false. When researchers follow good scientific practice, the statistical power associated with each value of ${\displaystyle d}$ can be seen at $k=1$. As is well known, power generally increases with ${\displaystyle d}$, and it is larger with $\alpha =5\%$ than with $\alpha =0.5\%$. For the present purposes, however, the most important aspect of the $d>0$ lines is the strong increase of statistical power with ${\displaystyle k}$, which can be seen in both panels, especially when the single-experiment power is well below one. Since replication rates increase with power (Button et al., 2013; Button and Munafò, 2017), this power inflation will tend to compensate for the increased Type 1 error rate with respect to the overall influence of selective reporting on replication rate. It is therefore necessary to use a quantitative model to assess the net effect of this practice on the replication rate.

Using the above probabilities of rejecting $H_0$, the proportion of false positives associated with this p-hacking scenario can be computed from Equation 2. Figure 4 and Figure 4—figure supplement 1 highlight the false positive rate as a function of ${\displaystyle d}$, ${\displaystyle \alpha }$, and ${\displaystyle k}$. Dashed lines show the rates for researchers engaged in p-hacking. For comparison, the solid lines depict the rates for researchers who follow good scientific practice by just running a single experiment and reaching a conclusion based on its outcome (i.e., $k=1$). The rates for these researchers were also computed with Equation 2 by inserting the nominal value of ${\displaystyle \alpha }$ for ${\alpha }_1$ and the single-experiment power for $1-{\beta }_1$.

Figure 4 with 1 supplement see all

Download asset Open asset

Several effects can be observed in Figure 4 and Figure 4—figure supplement 1: (a) As one expects, the false positive rate decreases from one to zero with increasing ${\displaystyle \pi }$, because the proportion of true effects among all significant effects becomes larger when ${\displaystyle \pi }$ increases (e.g., Ioannidis, 2005b; Wilson and Wixted, 2018). (b) Not surprisingly, the false positive rate becomes smaller when power increases due to larger ${\displaystyle d}$ (Ioannidis, 2005b). (c) Most interestingly and surprisingly, the increase in false positives produced by p-hacking is more pronounced with larger ${\displaystyle d}$, where statistical power is higher. This is presumably because p-hacking cannot increase statistical power much when it is already high (i.e., when ${\displaystyle d}$ is large), so there is little power inflation to compensate for the increased Type 1 error rate. Nevertheless, the effect of p-hacking is far from dramatic for $k=2$, although it can be quite prominent for larger values of ${\displaystyle k}$, especially with small base rates.

Figure 5 and Figure 5—figure supplement 2 depict replication rates computed using the same parameters as in the previous figures. In addition, Figure 5—figure supplement 1 and Figure 5—figure supplement 3 augment these figures and specifically focus on decrease in RR (i.e., “shrinkage”) caused by p-hacking. Three features of these computations are especially noticeable. (a) Successful replication depends strongly on the base rate. As one might expect, all rates converge to the statistical power $1-{\beta }_2=0.90$ of the replication study, because when all significant effects are real, the replication rate simply reflects the statistical power of the replication study, whether p-hacking was involved in the first study or not. (b) The effect of p-hacking is modest for high base rates, for the smaller ${\displaystyle \alpha }$ level, and interestingly also for smaller effect sizes and hence for low statistical power. (c) As emphasized by Benjamin et al., 2018, the replication rate is considerably larger for $\alpha =0.5\%$ than for $\alpha =5\%$, especially for small base rates.

Figure 5 with 3 supplements see all

Download asset Open asset

In summary, the above analysis casts doubt on the idea that this p-hacking strategy is a major contributor to low replicability, even though it seems to be one of the most frequent QRPs (e.g., John et al., 2012). Instead, it seems that using this strategy would have little effect on replicability except in research scenarios where true effects were rare but there was high power to detect them when they were present. The strongest trends suggest that a low base rate of true effects is the major cause of low replicability (Wilson and Wixted, 2018), since changes in base rate can cause replication rates to range across nearly the full 0–1 range.

Failing to report all dependent measures

Failing to report all of a study’s dependent measures seems to be another common QRP (Fiedler and Schwarz, 2016; John et al., 2012). In this section, we analyze how this practice would affect the rate of replicating statistically significant results. In order to model this scenario, we assume that a researcher conducts a study to test a certain hypothesis using control and experimental conditions. After data collection, however, the researcher only reports the outcomes of those dependent measures whose tests surpass the statistical significance threshold and thereby confirm the proposed hypothesis. As examples, multiple dependent measures are usually measured and statistically evaluated in neurosciences and medical research, raising concerns about Type 1 error rates in those fields (e.g., Hutton and Williamson, 2002; Vul et al., 2009).

We again employed z-tests to model this scenario. Let $Z_1,\mathrm{\dots },Z_k$ be the outcomes for all ${\displaystyle k}$ dependent measures of a single study, with each $Z$-value representing the result of the control/experimental comparison for a single measure. Therefore, the probability of obtaining at least one significant result is equal to

${\displaystyle Pr(\mathrm{R}\mathrm{e}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}{H}_0)=1-P(Z_1\le c,\dots ,Z_k\le c)}$

with c being the critical cutoff value (see Appendix 1 for computational details). Because such measures are usually correlated across participants, our model incorporates correlations among the $Z_i$ values.

Figure 6 illustrates the effects on Type 1 error probability (i.e., lines with $d=0$) and statistical power (i.e., lines with $d>0$) associated with this type of p-hacking. For this illustration, the pair-wise correlations of the different dependent measures were set to 0.2 and the sample size (per group) was set to 20, which are seemingly typical values in psychological research (Bosco et al., 2015; Marszalek et al., 2011). As expected, both the Type 1 error rate and power increase with the number of dependent measures, approximately as was found with selective reporting.

Figure 6 with 1 supplement see all

Download asset Open asset

Figures 7 and 8, and Figure 8—figure supplement 1 show the rate of false positives, rate of replications, and the shrinkage of the replication rate, respectively, resulting from this type of p-hacking. These results are quite similar to those seen with the selective reporting scenario (see Figures 4 and 5, and Figure 5—figure supplement 1). In particular, both false positive rates and replication rates show strong expected effects of base rate and ${\displaystyle \alpha }$ level, as well as a clear influence of effect size, d. The effects of p-hacking are again rather modest, however, especially when the effect size is small (i.e., $d=0.2$) so that increased power is especially helpful.

Figure 7 with 1 supplement see all

Download asset Open asset

Figure 8 with 3 supplements see all

Download asset Open asset

It should be noted that the extent of both Type 1 error rate inflation and power enhancement depend on the correlations among the different dependent measures. A correlation of zero would yield results identical to those of the scenario with selective reporting in the previous section, because in this case the outcomes for multiple dependent measures are independent just like the outcomes of multiple independent studies. In contrast, larger correlations (e.g., larger than the 0.2 used in Figures 7 and 8 and Figure 8—figure supplement 1) weaken the effects of this p-hacking strategy, because the measures become increasingly redundant as the intercorrelations increase, and this lowers the possibility of capitalizing on chance. In other words, increasing the intercorrelations would decrease the inflation of both Type 1 error rate and power. Moreover, increased intercorrelations would decrease the false positive rate and increase the replication rate, that is, moving the dashed lines in Figures 7 and 8 toward the solid reference lines (see Figure 6—figure supplement 1, Figure 7—figure supplement 1, Figure 8—figure supplement 2, and Figure 8—figure supplement 3 for a parallel analysis with intercorrelations of 0.8).

Data peeking

Another frequently-used QRP is data peeking (Fiedler and Schwarz, 2016; John et al., 2012). This practice occurs when a researcher collects additional data after finding that the results of initially collected data have not yielded statistical significance. A researcher may even peek at the results several times and increase the sample with additional observations each time a nonsignificant result is obtained. Data collection is finally terminated only if the study yields no significant result after ${\displaystyle k}$ peeks. It is known that this practice increases the Type 1 error rate (Armitage et al., 1969; Francis, 2012a; McCarroll et al., 1992; Simmons et al., 2011; Strube, 2006). For example, Monte-Carlo simulations by Simmons et al., 2011 revealed that this strategy can increase the error rate up to 14.3% with a first peek at $n=10$ and four subsequent peeks (each time increasing the sample by 10 observations). However, this practice increases not only the Type 1 error rate but also the effective statistical power to reject a false $H_0$ (Strube, 2006), so a quantitative analysis is needed to determine its effect on replication rate.

An analysis similar to that of the preceding sections was conducted to examine how data peeking affects Type 1 error rates, power levels, false positive rates, and replication rates. Appendix 1 contains the computational details of this analysis, which follows an extension of Armitage's procedure (Armitage et al., 1969). In brief, the probability of rejecting $H_0$ with a maximum of ${\displaystyle k}$ peeks at successive sample sizes $n_1<n_2<\mathrm{\cdots }<n_k$ is again given by the multivariate normal distribution for z-tests

${\displaystyle Pr(\mathrm{R}\mathrm{e}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}{H}_0)=1-P(Z_1\le c,\dots ,Z_k\le c).}$

The correlations among the different $Z_i$ values are determined by the amount of shared data used in computing them (e.g., all observations used in computing $Z_1$ are also included in the computation of $Z_2$).

Figure 9 depicts the probability of rejecting $H_0$ for various effect sizes and two-sample tests. The abscissa represents the maximal number of peeks ${\displaystyle k}$ at which a researcher would give up recruiting additional participants. For this example, it is assumed that data peeking occurs after 10, 15, 20, 25, 30, 35, 40, or 45 observations per group. Thus, a researcher with a maximum of $k=2$ peeks will check statistical significance the first time at $n_1=10$ and if the first peek does not reveal a significant result, the data will be examined a second and final time at $n_2=15$. For $k=3$, data will be examined a first time at $n_1=10$ and—depending on the outcome of the first peek—a second time at $n_2=15$; if the second peek also does not reveal a significant result, a final peek occurs at $n_3=20$.

Figure 9 with 1 supplement see all

Download asset Open asset

Figure 9 shows quantitatively how the probability of rejecting $H_0$ increases with the maximum number of peeks. In particular, the increase can be quite strong in situations with only moderate power (e.g., $\alpha =0.5\%$ and $d=0.8$) due to the extra chances of detecting the true effect. In contrast to the multiple dependent measures with intercorrelations of 0.2 as discussed in the previous section, the Type 1 error rate inflation is smaller in the present case, because $Z_1,\mathrm{\dots },Z_k$ are more strongly correlated under this scenario (cf. the correlation matrix in Appendix 1).

Given the probabilities of rejecting $H_0$, the replication rate and false positive rate are again computed using Equations 1 and 2, respectively. The results with respect to the false positive rate (Figure 9—figure supplement 1) and the replication rate (Figure 10 and Figure 10—figure supplement 1) are quite similar to those of the preceding scenarios. We compare this p-hacking strategy with researchers who conform to good scientific practice and thus examine the data only once at a preplanned n. In order to enable a conservative comparison with p-hackers, we used a preplanned n corresponding to the maximum number of observations a p-hacker would try when using the indicated number of peeks (i.e., this preplanned group size would be $n=15$ for the comparison with $k=2$, $n=20$ for the comparison with $k=3$, etc.). As can be seen, the pattern of results is quite comparable to the previous scenarios. Overall, the data peeking strategy again seems to have little effect on replication rate except in research scenarios where true effects are infrequent and there is high power to detect them when they do occur, just as with selective reporting.

Figure 10 with 1 supplement see all

Download asset Open asset

Selective outlier removal

Another QRP identified by John et al., 2012 is to analyze the same overall data set several times, each time excluding “outlier” data points identified by different criteria. The researcher may be tempted to conclude that a real effect has been found if any analysis yields a significant result, but this practice inflates the Type 1 error rate, because each of the analyses provides a further opportunity to obtain a significant result by chance. On the positive side, though, this practice again increases power, because each of the analyses also provides a further opportunity for detecting a real effect.

Because the effects of this type of p-hacking are not computable, we conducted Monte-Carlo simulations to see how multiple attempts at outlier removal would affect the Type 1 error rate, power, rate of false positives, and replication rate. Specifically, we examined the common practice of excluding scores more than a given number of standard deviations from the sample mean. We simulated researchers who carried out a sequence of at most five separate analyses on a single data set. The first three analyses included only scores within 3, 2.5, and 2 standard deviations of the mean, respectively, because these limits are most commonly employed in psychological research (Bakker and Wicherts, 2014). The fourth analysis used the Tukey, 1977 “fences” method by including all scores within the range $[l=Q_1-1.5\cdot (Q_3-Q_1),u=Q_3+1.5\cdot (Q_3-Q_1)]$, where $Q_1$ and $Q_3$ are the 25 and 75% percentile points of the data set. The fifth analysis used a nonparametric test, which could potentially be used as an analysis in an attempt to minimize the influence of outliers even further.

We simulated experiments for both one- and two-sample tests, but only report the latter because the two simulations produced extremely similar results. There was a sample size of $n=20$ per group using standard normally distributed scores and true effect sizes of $d=0$, 0.2, 0.5, and 0.8. Researchers were modelled as using either $\alpha =0.5\%$ or 5%, one-tailed. The nonparametric test was the Mann-Whitney $U$ test, and this test was used only if none of the previous analyses had produced significant results. We simulated 10,000 experiments with outliers by adding a random noise value to 5% of the data values, where these noise values came from a normal distribution with $\mu =0$ and $\sigma =10$. This simulation method has often been adopted to model contamination effects of outliers (e.g., Bakker and Wicherts, 2014; Zimmerman, 1998).

Figure 11 shows the probabilities of rejecting $H_0$. As with the other p-hacking methods, this probability increases with the number of analyses conducted, increasing the probability of a Type 1 error when $d=0$ and increasing power when $d>0$. Figures 12 and 13 show the false positive and replication rates; Figure 13—figure supplement 1 depicts the shrinkage of the replication rate. Interestingly, in some cases these measures even indicate slightly better results (i.e., lower false positive rates and higher replication rates) when researchers perform multiple analyses to remove the effects of possible outliers than when they do not. Most importantly, however, the present scenario also reveals that the major impact on the replication rate seems to come from the base rate.

Figure 11 with 1 supplement see all

Download asset Open asset

Figure 12 with 1 supplement see all

Download asset Open asset

Figure 13 with 3 supplements see all

Download asset Open asset

The present simulations assume that researchers try to remove outliers (i.e., apply a three-sigma rule) before they perform a t-test. Alternatively, however, researchers might first conduct a t-test on all data without excluding any extreme data points. If this test did not reveal statistical significance, they would then eliminate extreme data points before conducting one or more further t-tests. Under this alternative scenario, our simulations indicate that multiple analyses can produce notably better replication rates than the single analysis with all data points, apparently because the exclusion of outliers noticeably improves power relative to the analysis without exclusions. Moreover, the standard deviation of our outlier distribution was small compared to simulations of similar outlier scenarios (e.g., Bakker and Wicherts, 2014; Zimmerman, 1998). Our conclusion, of course, is that researchers should carefully examine their data for possible outliers before conducting any statistical tests, not that they should perform multiple tests with different outlier screening criteria—thereby inflating their Type 1 error rates—in order to maximize power.

Naturally, the story is different when no outliers are present in the data set. Making multiple attempts to remove outliers in this case would actually always increase the false positive rate and lower the replication rate (see Figure 11—figure supplement 1, Figure 12—figure supplement 1, Figure 13—figure supplement 2, and , Figure 13—figure supplement 3 for a parallel simulation with no outliers). In fact, extreme data points in data sets without outliers appear to be especially diagnostic for testing the equality of locations between populations, as the Tukey pocket test demonstrates (Tukey, 1959), so throwing away extreme observations that are not outliers reduces the information in the data set.

The analysis of p-hacking in the main article is based on one- and two-sample z-tests. Although researchers usually employ one- and two-sample t-tests rather than z-tests, we studied the latter tests because of their greater mathematical tractability. This should not have a big impact on the results, because z-tests closely resemble the results of the corresponding t-tests for realistic sample sizes.

There are no empirical data because mathematical modelling was employed to assess the impact of various factors on the replication of significant results.

1. 1. Armitage P
   2. McPherson CK
   3. Rowe BC

   (1969) Repeated significance tests on accumulating data

   Journal of the Royal Statistical Society. Series A 132:235–244.

   https://doi.org/10.2307/2343787

   • Google Scholar
2. 1. Asendorpf JB
   2. Conner M
   3. De Fruyt F
   4. De Houwer J
   5. Denissen JJA
   6. Fiedler K
   7. Fiedler S
   8. Funder DC
   9. Kliegl R
   10. Nosek BA
   11. Perugini M
   12. Roberts BW
   13. Schmitt M
   14. van Aken MAG
   15. Weber H
   16. Wicherts JM

   (2013) Recommendations for increasing replicability in psychology

   European Journal of Personality 27:108–119.

   https://doi.org/10.1002/per.1919

   • Google Scholar
3. 1. Baker M
   2. Penny D

   (2016) Is there a reproducibility crisis?

   Nature 533:452–454.

   https://doi.org/10.1038/533452a

   • Google Scholar
4. 1. Bakker M
   2. van Dijk A
   3. Wicherts JM

   (2012) The rules of the game called psychological science

   Perspectives on Psychological Science 7:543–554.

   https://doi.org/10.1177/1745691612459060

   • PubMed
   • Google Scholar
5. 1. Bakker M
   2. Wicherts JM

   (2014) Outlier removal, sum scores, and the inflation of the type I error rate in independent samples t tests: the power of alternatives and recommendations

   Psychological Methods 19:409–427.

   https://doi.org/10.1037/met0000014

   • PubMed
   • Google Scholar
6. 1. Barrett LF

   (2020)

   Forward into the past

   APA Observer 33:5–7.

   • Google Scholar
7. 1. Begley CG
   2. Ellis LM

   (2012) Raise standards for preclinical cancer research

   Nature 483:531–533.

   https://doi.org/10.1038/483531a

   • Google Scholar
8. 1. Benjamin DJ
   2. Berger JO
   3. Johannesson M
   4. Nosek BA
   5. Wagenmakers EJ
   6. Berk R
   7. Bollen KA
   8. Brembs B
   9. Brown L
   10. Camerer C
   11. Cesarini D
   12. Chambers CD
   13. Clyde M
   14. Cook TD
   15. De Boeck P
   16. Dienes Z
   17. Dreber A
   18. Easwaran K
   19. Efferson C
   20. Fehr E
   21. Fidler F
   22. Field AP
   23. Forster M
   24. George EI
   25. Gonzalez R
   26. Goodman S
   27. Green E
   28. Green DP
   29. Greenwald AG
   30. Hadfield JD
   31. Hedges LV
   32. Held L
   33. Hua Ho T
   34. Hoijtink H
   35. Hruschka DJ
   36. Imai K
   37. Imbens G
   38. Ioannidis JPA
   39. Jeon M
   40. Jones JH
   41. Kirchler M
   42. Laibson D
   43. List J
   44. Little R
   45. Lupia A
   46. Machery E
   47. Maxwell SE
   48. McCarthy M
   49. Moore DA
   50. Morgan SL
   51. Munafó M
   52. Nakagawa S
   53. Nyhan B
   54. Parker TH
   55. Pericchi L
   56. Perugini M
   57. Rouder J
   58. Rousseau J
   59. Savalei V
   60. Schönbrodt FD
   61. Sellke T
   62. Sinclair B
   63. Tingley D
   64. Van Zandt T
   65. Vazire S
   66. Watts DJ
   67. Winship C
   68. Wolpert RL
   69. Xie Y
   70. Young C
   71. Zinman J
   72. Johnson VE

   (2018) Redefine statistical significance

   Nature Human Behaviour 2:6–10.

   https://doi.org/10.1038/s41562-017-0189-z

   • PubMed
   • Google Scholar
9. 1. Bosco FA
   2. Aguinis H
   3. Singh K
   4. Field JG
   5. Pierce CA

   (2015) Correlational effect size benchmarks

   Journal of Applied Psychology 100:431–449.

   https://doi.org/10.1037/a0038047

   • PubMed
   • Google Scholar
10. Book

    1. Bunge M

    (1967) Scientific Research II: The Search for Truth

    Springer.

    https://doi.org/10.1007/978-3-642-48138-3

    • Google Scholar
11. 1. Button KS
    2. Ioannidis JP
    3. Mokrysz C
    4. Nosek BA
    5. Flint J
    6. Robinson ES
    7. Munafò MR

    (2013) Power failure: why small sample size undermines the reliability of neuroscience

    Nature Reviews Neuroscience 14:365–376.

    https://doi.org/10.1038/nrn3475

    • PubMed
    • Google Scholar
12. Book

    1. Button KS
    2. Munafò MR

    (2017) Powering Reproducible Research

    In: Lilienfeld S. O, Waldman I. D, editors. Psychological Science Under Scrutiny: Recent Challenges and Proposed Solutions. Wiley Online Library. pp. 22–23.

    https://doi.org/10.1002/9781119095910.ch2

    • Google Scholar
13. 1. Camerer CF
    2. Dreber A
    3. Holzmeister F
    4. Ho TH
    5. Huber J
    6. Johannesson M
    7. Kirchler M
    8. Nave G
    9. Nosek BA
    10. Pfeiffer T
    11. Altmejd A
    12. Buttrick N
    13. Chan T
    14. Chen Y
    15. Forsell E
    16. Gampa A
    17. Heikensten E
    18. Hummer L
    19. Imai T
    20. Isaksson S
    21. Manfredi D
    22. Rose J
    23. Wagenmakers EJ
    24. Wu H

    (2018) Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

    Nature Human Behaviour 2:637–644.

    https://doi.org/10.1038/s41562-018-0399-z

    • PubMed
    • Google Scholar
14. Book

    1. Carnap R

    (1995)

    An Introduction to the Philosophy of Science

    General Publishing Company.

    • Google Scholar
15. 1. Dreber A
    2. Pfeiffer T
    3. Almenberg J
    4. Isaksson S
    5. Wilson B
    6. Chen Y
    7. Nosek BA
    8. Johannesson M

    (2015) Using prediction markets to estimate the reproducibility of scientific research

    PNAS 112:15343–15347.

    https://doi.org/10.1073/pnas.1516179112

    • PubMed
    • Google Scholar
16. 1. Errington TM
    2. Iorns E
    3. Gunn W
    4. Tan FE
    5. Lomax J
    6. Nosek BA

    (2014) An open investigation of the reproducibility of cancer biology research

    eLife 3:e04333.

    https://doi.org/10.7554/eLife.04333

    • PubMed
    • Google Scholar
17. 1. Etz A
    2. Vandekerckhove J

    (2016) A Bayesian perspective on the Reproducibility Project: Psychology

    PLOS ONE 11:e0149794.

    https://doi.org/10.1371/journal.pone.0149794

    • PubMed
    • Google Scholar
18. 1. Fanelli D

    (2009) How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data

    PLOS ONE 4:e5738.

    https://doi.org/10.1371/journal.pone.0005738

    • PubMed
    • Google Scholar
19. 1. Fiedler K
    2. Kutzner F
    3. Krueger JI

    (2012) The long way from α-error control to validity proper: problems with a short-sighted false-positive debate

    Perspectives on Psychological Science 7:661–669.

    https://doi.org/10.1177/1745691612462587

    • PubMed
    • Google Scholar
20. 1. Fiedler K
    2. Schwarz N

    (2016) Questionable research practices revisited

    Social Psychological and Personality Science 7:45–52.

    https://doi.org/10.1177/1948550615612150

    • Google Scholar
21. 1. Fraley RC
    2. Vazire S

    (2014) The N-pact factor: evaluating the quality of empirical journals with respect to sample size and statistical power

    PLOS ONE 9:e109019.

    https://doi.org/10.1371/journal.pone.0109019

    • PubMed
    • Google Scholar
22. 1. Francis G

    (2012a) Publication bias and the failure of replication in experimental psychology

    Psychonomic Bulletin & Review 19:975–991.

    https://doi.org/10.3758/s13423-012-0322-y

    • PubMed
    • Google Scholar
23. 1. Francis G

    (2012b) Too good to be true: Publication bias in two prominent studies from experimental psychology

    Psychonomic Bulletin & Review 19:151–156.

    https://doi.org/10.3758/s13423-012-0227-9

    • PubMed
    • Google Scholar
24. 1. Francis G

    (2014) The frequency of excess success for articles in Psychological Science

    Psychonomic Bulletin & Review 21:1180–1187.

    https://doi.org/10.3758/s13423-014-0601-x

    • PubMed
    • Google Scholar
25. 1. Francis G
    2. Tanzman J
    3. Matthews WJ

    (2014) Excess success for psychology articles in the journal Science

    PLOS ONE 9:e114255.

    https://doi.org/10.1371/journal.pone.0114255

    • PubMed
    • Google Scholar
26. 1. Frick RW

    (1998) A better stopping rule for conventional statistical tests

    Behavior Research Methods, Instruments, & Computers 30:690–697.

    https://doi.org/10.3758/BF03209488

    • Google Scholar
27. 1. Genz A

    (1992) Numerical computation of multivariate normal probabilities

    Journal of Computational and Graphical Statistics 1:141–149.

    https://doi.org/10.1007/978-3-319-33507-0_13

    • Google Scholar
28. 1. Genz A
    2. Bretz F

    (1999) Numerical computation of multivariate t -probabilities with application to power calculation of multiple contrasts

    Journal of Statistical Computation and Simulation 63:103–117.

    https://doi.org/10.1080/00949659908811962

    • Google Scholar
29. 1. Genz A
    2. Bretz F

    (2002) Comparison of methods for the computation of multivariate t probabilities

    Journal of Computational and Graphical Statistics 11:950–971.

    https://doi.org/10.1198/106186002394

    • Google Scholar
30. 1. Gross C

    (2016) Scientific misconduct

    Annual Review of Psychology 67:693–711.

    https://doi.org/10.1146/annurev-psych-122414-033437

    • PubMed
    • Google Scholar
31. 1. Hedges LV

    (1984) Estimation of effect size under nonrandom sampling: the effects of censoring studies yielding statistically insignificant mean differences

    Journal of Educational Statistics 9:61–85.

    https://doi.org/10.3102/10769986009001061

    • Google Scholar
32. 1. Hutton JL
    2. Williamson PR

    (2002) Bias in meta-analysis due to outcome variable selection within studies

    Journal of the Royal Statistical Society: Series C 49:359–370.

    https://doi.org/10.1111/1467-9876.00197

    • Google Scholar
33. 1. Ioannidis JP

    (2005a) Contradicted and initially stronger effects in highly cited clinical research

    JAMA 294:218–228.

    https://doi.org/10.1001/jama.294.2.218

    • PubMed
    • Google Scholar
34. 1. Ioannidis JP

    (2005b) Why most published research findings are false

    PLOS Medicine 2:e124.

    https://doi.org/10.1371/journal.pmed.0020124

    • PubMed
    • Google Scholar
35. 1. Ioannidis JPA
    2. Trikalinos TA

    (2007) An exploratory test for an excess of significant findings

    Clinical Trials: Journal of the Society for Clinical Trials 4:245–253.

    https://doi.org/10.1177/1740774507079441

    • Google Scholar
36. 1. John LK
    2. Loewenstein G
    3. Prelec D

    (2012) Measuring the prevalence of questionable research practices with incentives for truth telling

    Psychological Science 23:524–532.

    https://doi.org/10.1177/0956797611430953

    • PubMed
    • Google Scholar
37. 1. Johnson VE
    2. Payne RD
    3. Wang T
    4. Asher A
    5. Mandal S

    (2017) On the reproducibility of psychological science

    Journal of the American Statistical Association 112:1–10.

    https://doi.org/10.1080/01621459.2016.1240079

    • PubMed
    • Google Scholar
38. 1. Lane DM
    2. Dunlap WP

    (1978) Estimating effect size: bias resulting from the significance criterion in editorial decisions

    British Journal of Mathematical and Statistical Psychology 31:107–112.

    https://doi.org/10.1111/j.2044-8317.1978.tb00578.x

    • Google Scholar
39. 1. Leek JT
    2. Peng RD

    (2015) Statistics: p values are just the tip of the iceberg

    Nature 520:612.

    https://doi.org/10.1038/520612a

    • PubMed
    • Google Scholar
40. 1. Lewandowsky S
    2. Oberauer K

    (2020) Low replicability can support robust and efficient science

    Nature Communications 11:1–12.

    https://doi.org/10.1038/s41467-019-14203-0

    • PubMed
    • Google Scholar
41. 1. Lilienfeld SO

    (2017) Psychology's replication crisis and the grant culture: Righting the ship

    Perspectives on Psychological Science 12:660–664.

    https://doi.org/10.1177/1745691616687745

    • PubMed
    • Google Scholar
42. 1. Marszalek JM
    2. Barber C
    3. Kohlhart J
    4. Holmes CB

    (2011) Sample size in psychological research over the past 30 years

    Perceptual and Motor Skills 112:331–348.

    https://doi.org/10.2466/03.11.PMS.112.2.331-348

    • PubMed
    • Google Scholar
43. 1. McCarroll D
    2. Crays N
    3. Dunlap WP

    (1992) Sequential ANOVAs and type I error rates

    Educational and Psychological Measurement 52:387–393.

    https://doi.org/10.1177/0013164492052002014

    • Google Scholar
44. 1. Miller J

    (2009) What is the probability of replicating a statistically significant effect?

    Psychonomic Bulletin & Review 16:617–640.

    https://doi.org/10.3758/PBR.16.4.617

    • Google Scholar
45. 1. Miller J
    2. Schwarz W

    (2011) Aggregate and individual replication probability within an explicit model of the research process

    Psychological Methods 16:337–360.

    https://doi.org/10.1037/a0023347

    • PubMed
    • Google Scholar
46. 1. Miller J
    2. Ulrich R

    (2016) Optimizing research payoff

    Perspectives on Psychological Science 11:664–691.

    https://doi.org/10.1177/1745691616649170

    • PubMed
    • Google Scholar
47. 1. Miller J
    2. Ulrich R

    (2019) The quest for an optimal alpha

    PLOS ONE 14:e0208631.

    https://doi.org/10.1371/journal.pone.0208631

    • Google Scholar
48. Book

    1. Mood AK
    2. Graybill FA
    3. Boes DC

    (1974)

    Introduction to the Theory of Statistics (3rd Edition)

    McGraw-Hill.

    • Google Scholar
49. 1. Nissen SB
    2. Magidson T
    3. Gross K
    4. Bergstrom CT

    (2016) Publication bias and the canonization of false facts

    eLife 5:e21451.

    https://doi.org/10.7554/eLife.21451

    • PubMed
    • Google Scholar
50. 1. Nosek BA
    2. Alter G
    3. Banks GC
    4. Borsboom D
    5. Bowman SD
    6. Breckler SJ
    7. Buck S
    8. Chambers CD
    9. Chin G
    10. Christensen G
    11. Contestabile M
    12. Dafoe A
    13. Eich E
    14. Freese J
    15. Glennerster R
    16. Goroff D
    17. Green DP
    18. Hesse B
    19. Humphreys M
    20. Ishiyama J
    21. Karlan D
    22. Kraut A
    23. Lupia A
    24. Mabry P
    25. Madon TA
    26. Malhotra N
    27. Mayo-Wilson E
    28. McNutt M
    29. Miguel E
    30. Paluck EL
    31. Simonsohn U
    32. Soderberg C
    33. Spellman BA
    34. Turitto J
    35. VandenBos G
    36. Vazire S
    37. Wagenmakers EJ
    38. Wilson R
    39. Yarkoni T

    (2015) Promoting an open research culture

    Science 348:1422–1425.

    https://doi.org/10.1126/science.aab2374

    • PubMed
    • Google Scholar
51. 1. Nosek BA
    2. Ebersole CR
    3. DeHaven AC
    4. Mellor DT

    (2018) The preregistration revolution

    PNAS 115:2600–2606.

    https://doi.org/10.1073/pnas.1708274114

    • PubMed
    • Google Scholar
52. 1. Oberauer K
    2. Lewandowsky S

    (2019) Addressing the theory crisis in psychology

    Psychonomic Bulletin & Review 26:1596–1618.

    https://doi.org/10.3758/s13423-019-01645-2

    • PubMed
    • Google Scholar
53. 1. Open Science Collaboration

    (2015) Estimating the reproducibility of psychological science

    Science 349:aac4716.

    https://doi.org/10.1126/science.aac4716

    • PubMed
    • Google Scholar
54. 1. Pashler H
    2. Harris CR

    (2012) Is the replicability crisis overblown? Three arguments examined

    Perspectives on Psychological Science 7:531–536.

    https://doi.org/10.1177/1745691612463401

    • PubMed
    • Google Scholar
55. Book

    1. Popper K

    (2002)

    The Logic of Scientific Discovery

    Routledge Classics.

    • Google Scholar
56. 1. Prinz F
    2. Schlange T
    3. Asadullah K

    (2011) Believe it or not: how much can we rely on published data on potential drug targets?

    Nature Reviews Drug Discovery 10:712–713.

    https://doi.org/10.1038/nrd3439-c1

    • PubMed
    • Google Scholar
57. Book

    1. Proschan MA
    2. Lan GKK
    3. Wittes JT

    (2006) Statistical Monitoring of Clinical Trials: A Unified Approach

    Springer.

    https://doi.org/10.1007/978-0-387-44970-8

    • Google Scholar
58. 1. Rosenthal R

    (1979) The file drawer problem and tolerance for null results

    Psychological Bulletin 86:638–641.

    https://doi.org/10.1037/0033-2909.86.3.638

    • Google Scholar
59. 1. Sassenberg K
    2. Ditrich L

    (2019) Research in social psychology changed between 2011 and 2016: larger sample sizes, more self-report measures, and more online studies

    Advances in Methods and Practices in Psychological Science 2:107–114.

    https://doi.org/10.1177/2515245919838781

    • Google Scholar
60. 1. Schmidt FL
    2. Oh I-S

    (2016) The crisis of confidence in research findings in psychology: Is lack of replication the real problem? Or is it something else?

    Archives of Scientific Psychology 4:32–37.

    https://doi.org/10.1037/arc0000029

    • Google Scholar
61. 1. Simmons JP
    2. Nelson LD
    3. Simonsohn U

    (2011) False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant

    Psychological Science 22:1359–1366.

    https://doi.org/10.1177/0956797611417632

    • PubMed
    • Google Scholar
62. 1. Simonsohn U

    (2014) Posterior-hacking: Selective reporting invalidates Bayesian results also

    SSRN Electronic Journal 1:2374040.

    https://doi.org/10.2139/ssrn.2374040

    • Google Scholar
63. 1. Simonsohn U
    2. Nelson LD
    3. Simmons JP

    (2014a) p-Curve: A key to the file-drawer

    Journal of Experimental Psychology: General 143:534–547.

    https://doi.org/10.1037/a0033242

    • Google Scholar
64. 1. Simonsohn U
    2. Nelson LD
    3. Simmons JP

    (2014b) p-Curve and effect size: Correcting for publication bias using only significant results

    Perspectives on Psychological Science 9:666–681.

    https://doi.org/10.1177/1745691614553988

    • PubMed
    • Google Scholar
65. 1. Simonsohn U
    2. Simmons JP
    3. Nelson LD

    (2015) Better p-curves: making p-curve analysis more robust to errors, fraud, and ambitious p-hacking, a reply to Ulrich and Miller (2015)

    Journal of Experimental Psychology: General 144:1146–1152.

    https://doi.org/10.1037/xge0000104

    • Google Scholar
66. 1. Stanley TD
    2. Carter EC
    3. Doucouliagos H

    (2018) What meta-analyses reveal about the replicability of psychological research

    Psychological Bulletin 144:1325–1346.

    https://doi.org/10.1037/bul0000169

    • PubMed
    • Google Scholar
67. 1. Stroebe W
    2. Postmes T
    3. Spears R

    (2012) Scientific misconduct and the myth of self-correction in science

    Perspectives on Psychological Science 7:670–688.

    https://doi.org/10.1177/1745691612460687

    • PubMed
    • Google Scholar
68. 1. Strube MJ

    (2006) SNOOP: A program for demonstrating the consequences of premature and repeated null hypothesis testing

    Behavior Research Methods 38:24–27.

    https://doi.org/10.3758/BF03192746

    • PubMed
    • Google Scholar
69. 1. Tukey JW

    (1959) A quick, compact, two-sample test to Duckworth’s specifications

    Technometrics : A Journal of Statistics for the Physical, Chemical, and Engineering Sciences 1:31–48.

    https://doi.org/10.2307/1266308

    • Google Scholar
70. Book

    1. Tukey JW

    (1977)

    Exploratory Data Analysis 

    Addison-Wesley Publishing Company.

    • Google Scholar
71. 1. Ulrich R
    2. Miller J
    3. Erdfelder E

    (2018) Effect size estimation from t-statistics in the presence of publication bias

    Zeitschrift für Psychologie 226:56–80.

    https://doi.org/10.1027/2151-2604/a000319

    • Google Scholar
72. 1. Ulrich R
    2. Miller J

    (1994) Effects of truncation on reaction time analysis

    Journal of Experimental Psychology: General 123:34–80.

    https://doi.org/10.1037/0096-3445.123.1.34

    • PubMed
    • Google Scholar
73. 1. Vul E
    2. Harris C
    3. Winkielman P
    4. Pashler H

    (2009) Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition

    Perspectives on Psychological Science 4:274–290.

    https://doi.org/10.1111/j.1745-6924.2009.01125.x

    • PubMed
    • Google Scholar
74. 1. Wilson BM
    2. Wixted JT

    (2018) The prior odds of testing a true effect in cognitive and social psychology

    Advances in Methods and Practices in Psychological Science 1:186–197.

    https://doi.org/10.1177/2515245918767122

    • Google Scholar
75. 1. Witt JK

    (2019) Insights into criteria for statistical significance from signal detection analysis

    Meta-Psychology 3:871.

    https://doi.org/10.15626/MP.2018.871

    • Google Scholar
76. 1. Zimmerman DW

    (1998) Invalidation of parametric and nonparametric statistical tests by concurrent violation of two assumptions

    The Journal of Experimental Education 67:55–68.

    https://doi.org/10.1080/00220979809598344

    • Google Scholar
77. 1. Zwaan RA
    2. Etz A
    3. Lucas RE
    4. Donnellan MB

    (2018) Making replication mainstream

    Behavioral and Brain Sciences 41:01–61.

    https://doi.org/10.1017/S0140525X17001972

    • Google Scholar

Thank you for submitting your article "Meta Research: Replication of Significant Results - Modeling the Effects of p-Hacking" to eLife. Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by the eLife Features Editor (Peter Rodgers). The following individuals involved in review of your submission have agreed to reveal their identity: William Hedley Thompson (Reviewer #2); Gregory Francis (Reviewer #3).

Summary:

The authors present an argument and simulations illustrating the impact of the replication rate of four different questionable research practices. I found the argument both interesting and convincing. The article also raises important points about interpreting replicability that are very much misunderstood by many practicing scientists. However, improvements could be made to clarify certain assumptions and aspects of the argument.

Essential revisions:

1) Equation (1) succinctly related replication rate (RR) to power and alpha. In some sense, almost everything follows from this equation. The details about how different QRPs affect alpha and beta are secondary (but useful) analyses. A further discussion of equation (1) might be worthwhile regardless of how alpha and beta take their values (through QRPs or otherwise).

2) It was not clear to me how the value for β₂ was selected. Based on the figure captions, I think it was just set to be β₂=0.9, so 90% power. But the problem introduced by QRPs is that they tend to inflate standardized effect sizes. So, a replication of a QRP-influenced study might estimate power based on the effect size reported in the original study. Doing so will often lead to gross underestimation of power for the replication study (even though the replicator thinks they have 90% power, they might actually have 50% power). Also common, the replicator uses the same sample size as the original study, which again tends to lead to low power if the original study used QRPs. It is through these mechanisms that QRPs for an original study contribute to a low replicability rate (provided the replication study uses good practices).

3) As I read it, the headline result seemed to be that p-hacking doesn't have a large impact on replication, and therefore the explanation for surprisingly low replication must lie elsewhere. Unfortunately, the support for this claim hinges on the degree of p-hacking that one envisions, and it seems to me that the degree of p-hacking envisioned here is rather mild. My own experience suggests that most p-hacking flows from investigators' ignorance about what constitutes p-hacking, and in such cases, investigators can easily p-hack to a much greater degree than the simulations here suggest. I suspect that p-hacking investigators can easily conduct dozens if not hundreds of tests for every result ultimately published, which is a much more severe form of p-hacking than the simulations here envision. In these cases, p-hacking very well may be an important cause of non-reproducible results, thus overturning the major finding of this paper.

To be sure, the manuscript is abundantly aware that the degree to which p-hacking generates non-reproducible results depends on the degree of p-hacking, and both the results and text make that clear. So, I don't think the manuscript is 'wrong'. But, I fear that if one really wants to know how much p-hacking contributes to non-reproducible results, one has to know the extent to which p-hacked studies are indeed p-hacked.

-From the Features Editor: The article needs to explore levels of p-hacking higher than those explored in the current version and, if necessary, to revise the discussion and conclusions in the light of what these new analyses find.

Also, please consider discussing and citing the following paper:

Simonsohn et al 2015, http://dx.doi.org/10.1037/xge0000104

4) I also found the last simulation - meant to investigate the effect of removing outliers - unconvincing. As I understand it, the simulation generated hypothetical data sets that were the contaminated by outliers with a standard deviation ten-fold greater than the actual data generating process. It seems to me that this is a poor model for capturing outliers, because the outliers in this case are so anomalous that they corrupt the performance of the good-practice analysis. Thus we wind up with the head-scratching result that selective outlier removal improves FDR and RR. I don't think this set-up captures the actual hazards of selective outlier removal, and wouldn't put much stock in its results. To summarize, the simulation set-up needs to be more realistic and/or better justified.

5) A possible discussion point regarding the assumptions of the RR value. An interesting assumption in the RR value is that null/non-significant results are not replicated. And lowering the alpha threshold for statistical significance will increase the number of false negatives. So a possible outcome of focusing on improving RR with alpha thresholds that more false negatives go undetected and not replicated?

6) The authors use a group size of 20 (so total n=40) but sample size appears to be a key variable that will impact some of these measures (e.g. outlier exclusion). The researchers motivate their value by citing Marszalek, Barber, Kohlhart, & Holmes, 2011, Table 3.

First, I think the authors may be referring to table 1 to get the value of 40 (I am unable to locate the value in Table 3)?

Second, others (e.g. Fraley & Vazire 2014, 10.1371/journal.pone.0109019), found the average sample size (in social-personality) psychology research to be higher (here: 104). How dependent are the results and conclusions on the limited sample size? (especially for outlier exclusion). Also how dependent are the outlier exclusion results if more/less of the data points our outliers (currently 5% of data).

7) In Figure 3,4,6,9,11,12 and the associated text, there is no quantification about how much the "p-hacking" approach is worse and unprecise language is used, e.g. "is modest for high base rates" and the reader has to deduce the differences from the many-paneled figures. I think adding some summary numbers to the text (or an additional figure) to show the differences between methods would be useful (e.g. state RR difference when the base rate is 0.2 and 0.5 (with d=0.5, k=5) or maybe the total difference between the curves). This would be especially helpful when contrasting the differences between the "p-hacking" and "good practice" differences for the different thresholds where the reader has to deduce two differences from the graph and then compare those deduced differences in their heads.

8) One quite surprising result here was that selective outlier removal seems to increase the RR and perhaps needs a little more discussion. At the moment, a reader could read the paper and conclude that performing selective outlier removal is something that should be done to improve the RR. Is this the authors' position? If not, perhaps this should be explicitly stated.

9) The supplemental material (and a few places in the main text) suggest that QRPs might actually be favorable for scientific investigations because they increase the replicability rate. The text describes the situation properly, but I fear some readers will get the wrong impression. The favorable aspects very much depend on what a scientist wants (to avoid) out of their analyses. The supplemental material makes some claims about inflation and setting of Type I error rates and power that seem to contradict the Neyman-Pearson lemma. If not, then the multiple-studies researcher must using a larger sample size, so there are costs involved. This might be worth discussing.

10) In the Discussion the text suggests it will be difficult to increase replicability in fields with low base rates. To the contrary, I think it is easy: just increase the base rate. Scientists should do a better job picking hypotheses to test. They should not waste time testing hypotheses that would be surprising or counterintuitive. The text then goes onto discuss about how campaigns to reduce p-hacking may be ineffective. I get the point, but a field with a low base rate of hypotheses should have a low replication rate. Increasing replicability is not (or, should not) be the goal of scientific investigations.

11) The authors are familiar with some of my work on this topic (they cite several of my papers). There, the problem is not a low replication rate, but a "too high" replication rate. The problem is that if both original and replication scientists are using QRPs, then the replication rate is too high, compared to what would be expected with "good practice" analyses/experiments. In my view, this is the more serious problem with current practice, because it implies that the Type I error rate is higher than "good practice". This suggests that scientists are not doing what they intended to do. This different viewpoint struck me while reading the introduction of the paper. There it is noted that some people suggest that QRPs lead to low replication rates. But this claim never really made sense (at least not without more discussion) because QRPs increase the probability of rejecting the null; so QRPs increase the replication rate. Indeed, if the simulations were revised so that both the original and replication scientists used QRPs, there would be quite an increase in the replication rate, even when the true effect is 0.

[We repeat the points in the decision letter in italic, and give our responses in Roman.]

Essential revisions:

1) Equation (1) succinctly related replication rate (RR) to power and alpha. In some sense, almost everything follows from this equation. The details about how different QRPs affect alpha and beta are secondary (but useful) analyses. A further discussion of equation (1) might be worthwhile regardless of how alpha and beta take their values (through QRPs or otherwise).

We agree and hence we have further discussed Equation 1 by illustrating its major features in a figure (Figure 2).

2) It was not clear to me how the value for β₂ was selected. Based on the figure captions, I think it was just set to be β₂=0.9, so 90% power. But the problem introduced by QRPs is that they tend to inflate standardized effect sizes. So, a replication of a QRP-influenced study might estimate power based on the effect size reported in the original study. Doing so will often lead to gross underestimation of power for the replication study (even though the replicator thinks they have 90% power, they might actually have 50% power). Also common, the replicator uses the same sample size as the original study, which again tends to lead to low power if the original study used QRPs. It is through these mechanisms that QRPs for an original study contribute to a low replicability rate (provided the replication study uses good practices).

We agree that estimated effect sizes are overestimated by replicators in such scenarios, especially when non-significant results are put into the file drawer. The power of 90% was picked as the best-scenario value close to the average replication power (i.e. 92%) claimed by the OSF project, although we acknowledge that their actual power levels may have been lower for the reasons mentioned by the reviewer. In any case, we have rerun our computations with the above suggested power of 50% and checked if our conclusions are still valid under this condition. For example, Author response image 1 shows the rate of replication for a replication power of 50% instead of 90% (as in Figure 5 of the manuscript). It can be seen in this new figure that p-hacking would even be slightly less harmful to replication rates with low-powered replication studies than with high-powered replications. Thus low-powered replication studies would not change the conclusions of our paper. We now mention this point in the Discussion.

Author response image 1

Download asset Open asset

3) As I read it, the headline result seemed to be that p-hacking doesn't have a large impact on replication, and therefore the explanation for surprisingly low replication must lie elsewhere. Unfortunately, the support for this claim hinges on the degree of p-hacking that one envisions, and it seems to me that the degree of p-hacking envisioned here is rather mild. My own experience suggests that most p-hacking flows from investigators' ignorance about what constitutes p-hacking, and in such cases, investigators can easily p-hack to a much greater degree than the simulations here suggest. I suspect that p-hacking investigators can easily conduct dozens if not hundreds of tests for every result ultimately published, which is a much more severe form of p-hacking than the simulations here envision. In these cases, p-hacking very well may be an important cause of non-reproducible results, thus overturning the major finding of this paper.

To be sure, the manuscript is abundantly aware that the degree to which p-hacking generates non-reproducible results depends on the degree of p-hacking, and both the results and text make that clear. So, I don't think the manuscript is 'wrong'. But, I fear that if one really wants to know how much p-hacking contributes to non-reproducible results, one has to know the extent to which p-hacked studies are indeed p-hacked.

-From the Features Editor: The article needs to explore levels of p-hacking higher than those explored in the current version and, if necessary, to revise the discussion and conclusions in the light of what these new analyses find.

Also, please consider discussing and citing the following paper:

Simonsohn et al 2015, http://dx.doi.org/10.1037/xge0000104

We cannot rule out the possibility that some investigators p-hack to a more considerable degree than we have assumed in our computations. The extent of p-hacking remains a controversial issue, with some arguing and providing evidence that ambitious p-hacking is too complicated and thus not plausible (Simonsohn et al., 2015, p. 1149), and even that the frequency of any p-hacking has probably been overestimated (Fiedler & Schwarz, 2016). Unfortunately, the exact extent of p-hacking is difficult to determine and might strongly depend on the field of research. For example, in areas with small effect sizes, p-hacking might be more extensive than in fields with medium or large effect sizes. But even without knowing the true p-hacking rates, our analyses are still valuable, because they clearly show that evidence of massive p-hacking is needed before one can conclude that p-hacking is a major contributor to the replication crisis. Nevertheless, for most p-hacking methods we now consider more extensive p-hacking to address this point (k to a maximum of 8). Because we cannot see how one could perform so many different selective outlier removal attempts, however, we did not consider more extensive p-hacking with this strategy. We also included this point in the General Discussion.

4) I also found the last simulation - meant to investigate the effect of removing outliers - unconvincing. As I understand it, the simulation generated hypothetical data sets that were the contaminated by outliers with a standard deviation ten-fold greater than the actual data generating process. It seems to me that this is a poor model for capturing outliers, because the outliers in this case are so anomalous that they corrupt the performance of the good-practice analysis. Thus we wind up with the head-scratching result that selective outlier removal improves FDR and RR. I don't think this set-up captures the actual hazards of selective outlier removal, and wouldn't put much stock in its results. To summarize, the simulation set-up needs to be more realistic and/or better justified.

The simulation scenario that we used to contaminate normally distributed scores has previously been employed for assessing the efficacy of outlier elimination methods (e.g., Bakker & Wicherts, 2014; Zimmerman, 1998). For example, Bakker & Wicherts and also Zimmermann sampled normally distributed scores from N(d,1), d=0, 0.2, 0.5, 0.8 with probability 0.95 and contaminated these scores with outliers that were drawn with probability 0.05 from a normal distribution with a standard deviation of 20—even larger than our standard deviation of 10 that the reviewer regards as unrealistically large. Nonetheless, in order to check how the results would change with a less extreme contamination distribution as suggested by the reviewer, we changed the standard deviation of the outlier distribution to 5. We found pretty similar results, that is, the primary factor limiting the replication rate remains the base rate (see Author response image 2 which can be compared to Figure 16 in the main text with a standard deviation of 10). Besides, the Z-score method for identifying outliers is commonly used in psychology as the meta-analysis by Bakker & Wicherts has revealed; their meta-analysis also shows that absolute thresholds values for the Z scores of 2.0, 2.5 and 3.0 are common in psychological research. Thus, our simulations capture standard methods of data analysis. In the revision, we have stressed this point.

Author response image 2

Download asset Open asset

5) A possible discussion point regarding the assumptions of the RR value. An interesting assumption in the RR value is that null/non-significant results are not replicated. And lowering the alpha threshold for statistical significance will increase the number of false negatives. So a possible outcome of focusing on improving RR with alpha thresholds that more false negatives go undetected and not replicated?

Like others modeling replication rates, we assume that researchers only try to replicate significant results. As the reviewer notes, there will be fewer of these when alpha is lowered—whether a true effect is present or H₀ is true—and this effect of alpha can clearly be seen in the probability of rejecting H₀ (e.g., Figure 3). Naturally, our computations of RR take this effect into account, so the effects of alpha on RR can easily be seen (e.g., Figure 5). As the reviewer rightly notes, however, reducing alpha has an additional effect that is not evident in RR: namely, fewer replication studies will be needed, because fewer positive effects will be found. We have now mentioned this fact in the General Discussion, and we thank the reviewer for the suggestion.

6) The authors use a group size of 20 (so total n=40) but sample size appears to be a key variable that will impact some of these measures (e.g. outlier exclusion). The researchers motivate their value by citing Marszalek, Barber, Kohlhart, & Holmes, 2011, Table 3.

First, I think the authors may be referring to table 1 to get the value of 40 (I am unable to locate the value in Table 3)?

Second, others (e.g. Fraley & Vazire 2014, 10.1371/journal.pone.0109019), found the average sample size (in social-personality) psychology research to be higher (here: 104). How dependent are the results and conclusions on the limited sample size? (especially for outlier exclusion). Also how dependent are the outlier exclusion results if more/less of the data points our outliers (currently 5% of data).

Yes, we inferred this value from Table 3 (as indicated in the previous version) but not from Table 1, as the Reviewer seems to believe. Table 3 gives the group sample size.

First, the mean of the medians in Table 3 of Marszlek et al. (2011) is 18.9. The corresponding group size in the Open Science Replication Project (considering studies with significant t-tests) is 27.5 and thus not considerably off our value of 20. It is likely that sample sizes in published articles became larger or were already larger in the field of social psychology and personality (see Sassenberg & Ditrich, 2019, Advances in Methods and Practices in Psychological Sciences). Note that sample and effect size determine the statistical power of the original study. Thus if we would employ a larger group size, this would merely increase the statistical power to an unrealistically high level --- the estimated median power level has been 36% for psychological studies and is even lower in the neurosciences. In order to keep the power of the original studies at a realistic level, we would have to reduce the effect sizes in our computations. We refrained from doing this because the effect sizes of 0.2, 0.5, and 0.8 seem appropriate theoretical choices for demonstrating the effect of p-hacking on the replication rate with small, medium, and large effects. Nevertheless, we have addressed the issue of group size in the General Discussion.

Second, Fraley and Vazire (2014) have reported an average sample size of 104 in their meta-analysis of articles published in social-personality psychology. Unfortunately, this size refers to the total sample size and not to group size. For example, consider a 2 x 2 between factorial design, then each group would be comprised of 21 subjects. Nevertheless, their article contained additional information that we found worth mentioning – for example, they estimated the average power in this area as 50% with an average effect size of d = 0.43, and that the false positive rate is (at least ) 28% for a base rate of 20%. These values fit well with our analyses. The revised paper includes Fraley and Vazire.

Third, we have rerun the simulations on outlier exclusion with group sizes of 50 (i.e., sample sizes of 100) using the new scenario (see also our response to Comment 8 below). Author response image 3 shows the result (which should be compared to Figure 16 in the main text with a group size of 20). It is clear that there are no major changes in the effects of p-hacking, alpha, or base rate, even though overall replicability has increased because of the greater power associated with larger samples.

Author response image 3

Download asset Open asset

7) In Figure 3,4,6,9,11,12 and the associated text, there is no quantification about how much the "p-hacking" approach is worse and unprecise language is used, e.g. "is modest for high base rates" and the reader has to deduce the differences from the many-paneled figures. I think adding some summary numbers to the text (or an additional figure) to show the differences between methods would be useful (e.g. state RR difference when the base rate is 0.2 and 0.5 (with d=0.5, k=5) or maybe the total difference between the curves). This would be especially helpful when contrasting the differences between the "p-hacking" and "good practice" differences for the different thresholds where the reader has to deduce two differences from the graph and then compare those deduced differences in their heads.

We like the suggestion to include an additional figure that quantifies the shrinkage of the replication rate for various level of p-hacking. In the revised paper we have added such figures along with some text that describes the resulting shrinkage.

8) One quite surprising result here was that selective outlier removal seems to increase the RR and perhaps needs a little more discussion. At the moment, a reader could read the paper and conclude that performing selective outlier removal is something that should be done to improve the RR. Is this the authors' position? If not, perhaps this should be explicitly stated.

This is certainly not our position, although our simulations indicate that selective outlier removal can improve the replication rate and lower the false positive rate. In response to this comment, we have reconsidered our outlier removal simulation. In the previous version of the manuscript, the simulated researchers always started with “no removal” and then tried more and more removal. However, “no removal” is not the best method if there are outliers. It seems perhaps a more conventional practice to remove outliers before applying a statistical test (see the meta-analysis by Bakker & Wicherts, 2014 that we cite now). Under this alternative scenario, subsequent “removals” might especially be prone to Type I error inflation and thus lower the replication rate. We have rerun our simulations under this scenario. The results indicate that multiple removals would lower the replication rate. Author response table 1 shows how we have changed the sequence of outlier removal attempts for the simulations in the revised paper. We have also modified the text in the manuscript accordingly.

Author response table 1.

We have also explicitly stated that outlier checks should be made before statistical testing and that multiple testing with different outlier criteria is unacceptable because of the increased Type 1 error rate, regardless of RR.

9) The supplemental material (and a few places in the main text) suggest that QRPs might actually be favorable for scientific investigations because they increase the replicability rate. The text describes the situation properly, but I fear some readers will get the wrong impression. The favorable aspects very much depend on what a scientist wants (to avoid) out of their analyses. The supplemental material makes some claims about inflation and setting of Type I error rates and power that seem to contradict the Neyman-Pearson lemma. If not, then the multiple-studies researcher must using a larger sample size, so there are costs involved. This might be worth discussing.

First, in the revision we have emphasized further that we do not suggest that QRPs might actually be favorable (e.g., see the Conclusions). Even in the section on outlier removal, we have now stressed that researchers should carefully examine their data before conducting any statistical tests in order to avoid inflating the Type 1 error rate. Second, the analysis in Appendix subsection “Type 1 error rate versus power” suggests that p-hacking can produce a larger power compared to good practice for the same level of Type I error. However, as the reviewer has correctly noted and as we have stated there, much larger sample sizes are the price of the potential superiority of this QRP. So the benefit actually comes from the larger samples involved with this QRP, not the QRP per se.

10) In the Discussion the text suggests it will be difficult to increase replicability in fields with low base rates. To the contrary, I think it is easy: just increase the base rate. Scientists should do a better job picking hypotheses to test. They should not waste time testing hypotheses that would be surprising or counterintuitive. The text then goes onto discuss about how campaigns to reduce p-hacking may be ineffective. I get the point, but a field with a low base rate of hypotheses should have a low replication rate. Increasing replicability is not (or, should not) be the goal of scientific investigations.

We have sympathy with this comment and agree that researchers should prefer to test hypotheses deduced from a plausible theory. However, we also see practical constraints that make it difficult to increase base rates, contrary to the reviewer’s suggestion that this would be easy. For example, consider research in clinical pharmacology aiming at discovering better medicines, such as the search for an effective vaccine against an infectious disease. Although we are not pharmacologists, we can imagine that the search for such a vaccine can be very haphazard. Pharmacological research often tests many ineffective drugs before an effective one is discovered. In such areas, the base rate could necessarily be low and only increased by a better theoretical understanding of the disease and how drugs interact with it. When such understanding is difficult to achieve, some black-box approach and the associated low base rate may be the only option. This more philosophical issue is beyond the aim of our paper, which focuses on the question of why significant results often do not replicate, but we now comment on the potential difficulty of increasing base rate in the Discussion.

11) The authors are familiar with some of my work on this topic (they cite several of my papers). There, the problem is not a low replication rate, but a "too high" replication rate. The problem is that if both original and replication scientists are using QRPs, then the replication rate is too high, compared to what would be expected with "good practice" analyses/experiments. In my view, this is the more serious problem with current practice, because it implies that the Type I error rate is higher than "good practice". This suggests that scientists are not doing what they intended to do. This different viewpoint struck me while reading the introduction of the paper. There it is noted that some people suggest that QRPs lead to low replication rates. But this claim never really made sense (at least not without more discussion) because QRPs increase the probability of rejecting the null; so QRPs increase the replication rate. Indeed, if the simulations were revised so that both the original and replication scientists used QRPs, there would be quite an increase in the replication rate, even when the true effect is 0.

We agree that an excess of positive results due to QRPs—in both original studies and replication attempts—is another potential problem that could influence the replication rate. As the reviewer notes, the replication rate will be unrealistically high if QRPs are used to induce a significant result in the replication data. This situation, however, is different from the situation in which an unbiased researcher tries to replicate the results of an original study without using QRPs, and this is the situation producing the empirically low replication rates that have alarmed many researchers. We have now made clear that our analysis focuses on this latter replication situation.

Author details

1. Rolf Ulrich

   Rolf Ulrich is in the Department of Psychology, University of Tübingen, Tübingen, Germany

   Contribution

   Conceptualization, Software, Formal analysis, Writing - original draft

   For correspondence

   ulrich@uni-tuebingen.de

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-8443-2705

2. Jeff Miller

   Jeff Miller is in the Department of Psychology, University of Otago, Dunedin, New Zealand

   Contribution

   Conceptualization, Software, Formal analysis, Writing - original draft

   For correspondence

   miller@psy.otago.ac.nz

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2718-3153

• 8,565

  views

• 487

  downloads

• 17

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.