Article Figures & Data
Figures
- Figure 1.
Funnel plots including published studies (left panel) and our own replication studies (right panel) in which MDZ was used as amnestic agent. Each point represents an observed effect size Hedges’ g against its SE. Visual inspection of the plot on the right panel shows that our replication studies are symmetrically scattered around the effect estimate of 0.04, indicating that the estimated effect size is close to zero and suggesting that no trend in one particular direction was observed across studies. In contrast, the plot of published studies (left panel) clearly shows asymmetry, and the reported effect sizes seem to depend strongly on the research group in which the studies were performed (represented by the different symbols in the left plot). Egger’s test confirmed plot asymmetry (p < 0.0001), even when considering the moderating influence of research group. One should be careful to attach value to the estimated effect size shown in the left funnel plot, given the evidence for publication bias and because the nesting of studies within research groups is not accounted for. The funnel plots were based on the meta-analytic models without moderators. Symbols represent the research group in which each study was performed (left panel) or the lab space that was used (right panel). Note that three of our exact replication studies (right panel) were performed in the same lab space as some of the original, published studies (left panel).
- Figure 2.
The contour-enhanced funnel plot of published studies (left panel) suggests publication bias. Published studies (left panel) are missing in the white area and the region to the left of the white area (where non-significant results would be plotted). This pattern adds credibility to the possibility that funnel plot asymmetry is caused by publication bias based on statistical significance (Peters et al., 2008). As a comparison, our own (mainly non-significant) replication studies are plotted in the right graph. Symbols represent the research group in which each study was performed (left panel) or the lab space that was used (right panel). The white area and the region to the left of the white area contain non-significant one-sided p values (white region: p values between 0.05 and 0.95; dark gray-shaded region: p values between 0.95 and 0.975; medium gray-shaded region: p values between 0.975 and 0.995, light gray-shaded region outside of the funnel: p values > 0.995); areas to the right of the white area represent statistically significant one-sided p values (dark gray-shaded region: p values between 0.025 and 0.05; medium gray-shaded region: p values between 0.005 and 0.025, light gray-shaded region outside of the funnel: p values below 0.005).
- Figure 3.
Funnel plot including published studies suggests biased effect sizes. The asymmetrical funnel shape observed here was statistically confirmed by Egger’s regression (p < 0.001) and is suggestive of biased study outcomes because of selection of significant results for publication.
- Figure 4.
Contour-enhanced funnel plot including published studies suggests publication bias. The white area and the region on its left side contain studies with statistically non-significant amnestic effects based on one-tailed tests (drug < control; p ≥ 0.05). The plot suggests that non-significant studies are missing in the literature (i.e., publication bias). Remarkably, effect sizes are most densely plotted at the border of statistical significance, which might also imply biased effect sizes.
- Figure 5.
Contour-enhanced funnel plot still suggests publication bias when MDZ studies are excluded. The white area and the area on its left side contain studies with statistically non-significant amnestic effects based on one-tailed tests (drug < control; p ≥ 0.05). The asymmetrical funnel shape observed here was statistically confirmed by Egger’s regression (p < 0.001; or with Research Group and Drug as moderators: p = 0.022; exploratory analysis) and is suggestive of biased study outcomes because of selection of significant results for publication.
- Figure 6.
Replies to our request for unpublished data (clustered per research group).
- Figure 7.
Contour-enhanced funnel plot of published (filled circles) and unpublished (empty circles) studies. In contrast to published results (95 drug-vehicle comparisons), studies that remained unpublished (12 drug-vehicle comparisons) showed smaller, and mostly non-significant, amnestic effects. This discrepancy between published and unpublished results is in line with the presence of publication bias that was suggested by the funnel plots. The majority of unpublished “negative” studies of which the existence was revealed could not be included in the current study because of author preferences.
- Figure 8.
Contour-enhanced funnel plot of unpublished studies. A relatively symmetrical pattern is observed and, in line with this observation, Egger’s regression suggests that there is no statistically significant evidence for asymmetry (t(10) = 0.70, p = 0.499).
Tables
- Table 1
Overview of our exact replication attempts using contextual fear conditioning and postreactivation systemic MDZ injection in male Wistar rats
Experiment Correspondence between
original and replication studyAmnestic effect observed?
(MDZ < SAL)Sample size Obtained power Researcher Lab space JA091 x No 12 0.87 JA101 x No 15 0.93 JA111 x x Yes 20 0.98 JA121 x x Yes 19 0.97 NS091 No 12 0.87 NS112 No 12 0.68 NSARG013 x No 16 0.99 In these experiments, the methodology of the original studies was followed as closely as possible. All studies involved contextual fear conditioning, followed one day later by brief (i.e., 2, 3, or 5 min) unreinforced re-exposure to the conditioned context and systemic injection of MDZ (1.5 or 3 mg/kg) or saline (SAL), and retention testing one day later.
↵1Exact replication of Alfei et al. (2015) and Ferrer Monti et al. (2017).
↵2Exact replication of Stern et al. (2012).
↵3Exact replication of Espejo et al. (2016) and Ortiz et al. (2015).
The table indicates whether the experimenter and the lab in which the replication study was performed were the same as in the published, original study. The results show that the amnestic effect could not be replicated when the study was performed in a different lab or by another researcher, despite adherence to the experimental protocol of the original studies. These findings illustrate that the success of treatment may depend on subtle between-study differences, and the underlying causes of these failures to replicate remain unknown. Obtained power for our sample sizes, shown in the last column, was calculated using the smallest effect size (Hedges’ g) that was observed in the original studies (α = 0.05; d = 1.71 in Alfei et al., 2015; 1.31 in Stern et al., 2012; 2.91 in Espejo et al., 2016; see Table 2). A complete description of our replication attempts involving contextual fear can be found in Schroyens et al. (2019a,b). Appendix A (Tables A.1 and A.2) of Schroyens et al. (2019a) contains a detailed overview of experimental parameters for the conceptual and exact replication attempts, respectively. Experiment NSARG01 is described in Schroyens et al. (2019b).
- Table 2
Experiments (until April 2019) from 15 different papers included in our pilot analyses investigating amnestic effects of postreactivation MDZ administration for contextual fear conditioning in adult rats under standard conditions (in chronological order based on publication date)
Publication Research group Exp. Figure Reactivation session
durationMDZ dose
(mg/kg)Ntotal Effect size
(Hedges’ g)Bustos et al. (2006) I 1B 2B 90 s 1 16 1.93* Bustos et al. (2006) I 2A 3B 90 s 1 17 2.40* Bustos et al. (2006) I 2A 3B 90 s 1 17 2.34* Bustos et al. (2006) I 3B 7B 90 s 1 16 4.60* Zhang and Cranney (2008) II 1 1 90 s 2 18 0.96* Zhang and Cranney (2008) II 3 3 90 s 2 16 1.76* Bustos et al. (2009) I NA 1B 1 min 1.5 14 0.74 Bustos et al. (2009) I NA 1C 3 min 1.5 14 3.63* Bustos et al. (2009) I NA 1D 5 min 1.5 20 2.95* Bustos et al. (2009) I NA 2C 10 min 1.5 15 −2.06 Bustos et al. (2009) I NA 3B 3 min 1.5 14 3.13* Bustos et al. (2010) I 1 1B 3 min 1.5 or 3 17 2.52* Bustos et al. (2010) I 1 1C 5 min 1.5 or 3 19 2.41* Stern et al. (2012) III 1 1A 3 min 1.5 20 1.31* Pineyro et al. (2013) IV 1 1C 1 min 3 12 0.04 Pineyro et al. (2013) IV 1 1C 4 min 3 12 3.53* Pineyro et al. (2013) IV 1 1C 5 min 3 12 2.96* Alfei et al. (2015) IV 5 5A 2 min 3 18 1.71* Alfei et al. (2015) IV 6 6B 5 min 3 14 2.59* Ortiz et al. (2015) I 1 1C 3 min 3 19 3.22* Ortiz et al. (2015) I 1 1E 5 min 3 12 4.11* Ortiz et al. (2015) I 2 3C 5 min 3 16 4.10* Ferrer Monti et al. (2016) IV 1 1C 90 s 3 15 –0.05 Ferrer Monti et al. (2016) IV 1 1C 4 min 3 14 2.79* Espejo et al. (2016) I 1 1 5 min 3 15 3.92* Espejo et al. (2016) I 2 2 5 min 3 22 2.91* Saitoh et al. (2017) V NA 2B 3 min 1 24 0.85* Ferrer Monti et al. (2017) IV 2 2B 2 min 3 12 3.58* Espejo et al. (2017) I 1 1 5 min 3 16 3.30* Espejo et al. (2017) I 3 3 5 min 3 18 3.73* Akagi et al. (2018) V NA 2C 3 min 1 30 0.83* Franzen et al. (2019) III NA 1A 1 min 3 18 –0.12 Franzen et al. (2019) III NA 1B 2 min 3 17 1.39* Adapted from Schroyens et al. (2019a). See Inclusion criteria for an overview of which conditions were included. Studies that used contextual fear conditioning and postreactivation systemic MDZ injection in rats were included after a thorough literature search. Note that two additional papers with MDZ studies were identified by our systematic PubMed search and included in the preregistered analyses (De Oliveira Alvares et al., 2013; Couto-Pereira et al., 2019). The effect size (Hedges’ g) for the influence of MDZ on % freezing during the test session was estimated based on means and SEs from reported graphs (MDZ vs vehicle) using the metafor package in R. Details of the intervention, such as duration of the training and reactivation session and drug dose, are indicated as well.
*Amnestic effects reported as significant (at an α level of 0.05). The numbers in the second column refer to the research group,
I, IFEC-CONICET, Departamento de Farmacología, Facultad de Ciencias Químicas, Universidad Nacional de Córdoba, Córdoba, Argentina.
II, School of Psychology, University of New South Wales, Sydney, Australia.
III, Department of Pharmacology, Federal University of Santa Catarina, Florianópolis, Santa Catarina, Brazil.
IV, Laboratorio de Psicología Experimental, Facultad de Psicología, Universidad Nacional de Córdoba, Córdoba, Argentina.
V, Department of Neuropsychopharmacology, National Institute of Mental Health, National left of Neurology and Psychiatry, Tokyo, Japan.
- Table 3
Overview of studies in which pharmacological agents were administered systemically with the aim of inducing reactivation-dependent amnesia for contextual fear memories in rodents, as identified by the systematic review
Target/function Drug Subjects Administration route Dose (/kg) Amnesic effect obtained? Protein synthesis inhibitors DNA and protein synthesis interference Anisomycin Mice i.p. 75 mg + Mice i.p. 150 mg + and +/−1, 3 Mice i.p. 225 mg +/− Mice s.c. 50 mg * Rats i.p. 50 mg + mRNA translocation interference Cycloheximide Rats i.p. 2.2 mg + Receptor antagonists Oxytocin receptor Atosiban Rats i.p. 0.001–1 mg - Adenosine receptor Caffeine Rats i.p. 20 mg +/−1 GABAA-R (partial agonist) Flumazenil Rats i.p. 1 mg + NMDA-R MK-801 Mice i.p. 0.03 mg + Mice i.p. 0.06 mg + Mice i.p. 0.1 mg + and +/− Mice i.p. 0.12 mg + Rats i.p. 0.1 mg + and +/−3 and - Opioid receptor Naloxone Rats i.p. 3 mg +/−3 β-Adrenergic receptor Propranolol Mice i.p. 10 mg - and +/− Rats i.p. 2 mg - Rats i.p. 5 mg + and - Rats i.p. 10 mg + and - Dopamine D1/D5 receptor SCH23390 Rats i.p. 0.1 mg - CB1-R (partial agonist) SR141716A Mice i.p. 1–10 mg - Histamine H3-R (inverse agonist) Thioperamide Mice i.p. 2.5–30 mg - Pitolisant Mice i.p. 1.25–20 mg - Receptor agonists GABAA-R Betulin (BE) Rats oral 2 mg - Betulinic Acid (BA) Rats oral 2 mg - BE + BA Rats oral 2 mg + Ethanol Rats i.p. 0.5/1/1.5 mg +/−4 Souroubea sympetala Rats oral 8/25/75 mg +/−4* α2-Adrenergic receptor Clonidine Mice i.p. 0.3 mg +/−1 Rats i.p. 0.1 mg - Rats i.p. 0.3 mg + μ-Opioid receptor Morphine Rats s.c. 7.5 mg * NOP receptor Ro 65–6570 Mice i.p. 0.1/1 mg +/−4 AT-403 Mice i.p. 0.03/0.1 mg +/−4 Benzodiazepines GABAA receptor agonist Diazepam Rats oral 1/2 mg + GABAA receptor (allosteric modulator) Midazolam Rats s.c. 1 mg + Rats i.p. 1 mg + and - Rats i.p. 1.5 mg + and - and +/−1, 2, 3 Rats i.p. 1/1.5/3 mg +/−4 Rats i.p. 2 mg + Rats i.p. 3 mg + and - and +/−3 Cannabinoids Indirect potentiation of CB1-R-mediated transmission CBD Rats i.p. 1 mg +/− Rats i.p. 3 mg + Rats i.p. 10 mg +/− Rats i.p. 30 mg + Rats oral 50 mg + Activation of cannabinoid system Cannabis plant extracts (after
isolation of THC and CBD)Rats oral 43 mg + CB1-R agonist THC Rats oral 5 mg - Rats i.p. 0.1/0.3/1/10 mg +/−4 Intracellular molecule inhibitors DNA ligases and polymerases Ara-C Mice i.p. 1000 mg - GSK-3 AR-A014418 Mice i.p. 30 mg + NF-κB DDTC Rats i.p. 200 mg + 11β-hydroxylase Metyrapone Rats i.p. 75 mg - PARP-1 Tiq-A Mice i.p. 0.5 mg + Other Hormone Corticosterone Rats i.p. 1/3/10 mg +/−1, 4 Peptide GRP Rats i.p. 10 nmol + Bacterial toxin Lipopolysaccharides Mice i.p. 125 μg + NE-DA reuptake inhibitor Methylphenidate Rats i.p. 3/10 mg - DA reuptake inhibitor Modafinil Mice i.p. 200 mg + AMPA receptor potentiator PEPA Mice i.p. 30 mg - Glutamatergic system blocker Riluzole Rats s.c. 0.1/0.3/1/3 mg +/−4 Additional details for each study, including PubMed ID, strain, duration of the reactivation session (ranging from 30 s to 10 min), time of drug administration, and time between training and reactivation session (ranging from 1 to 36 d), are available at https://osf.io/x2pkq/. Ara-C = 1-β-D-arabinofuranosylcytosine triphosphate; CBD = cannabidiol; DA = dopamine; DDTC = diethyldithiocarbamate; GRP = gastrin releasing peptide; NE = norepinephrine; PEPA = 4-[2-(phenylsulfonylamino)ethylthio]−2,6-difluorophenoxyacetamide; s.c. = subcutaneous; THC = Δ9-tetrahydro-cannabinol; + = at least one study reported a statistically significant amnestic effect; * = amnestic effect was found to be transient; - = at least one study reported a non-significant effect; +/− = at least one study observed that the amnestic effect occurred under some conditions:
↵1 depending on training parameters (e.g., shock intensity).
2 depending on memory age.
↵3 depending on reactivation duration.
↵4 depending on drug dose.
- Table 4
Overview of studies in which pharmacological agents were administered intracranially with the aim of inducing reactivation-dependent amnesia for contextual fear memories in rodents, as identified by the systematic review
Target/function Drug Subjects Administration
routeDose Amnesic effect
obtained?Protein synthesis inhibitors DNA and protein synthesis interference Anisomycin Mice ACC 50 μg - Mice BLA 62.5 μg/side + Mice CA1 60 μg/side +/−3 Mice CA1 62.5 μg/side + and +/−2, 3 Mice dHipp 62.5 μg/side + Mice dHipp 75 μg +/−2 Mice mPFC 62.5 μg/side - Mice i.c.v. 0.1 mg +/−3 Rats ACC 62.5 μg/side + Rats BLA 62.5 μg/side + Rats CA1 80 μg/side * Rats CA1 250 μg/side + Rats MC 62.5 μg/side - RNA Polymerase II DRB Rats CA1 10 ng/side + Receptor antagonists CB1-R (inverse agonist) AM251 Rats amygdala 280 pg + and - NMDA-R D-AP5 Rats dHipp 5 μg/side + β-Adrenergic receptor Propranolol Rats BLA 1.25 μg/side + 5-HT6-R SB-271046 Rats CA1 10 μg/side - 5-HT5A-R SB699551 Rats CA1 10 μg/side +/− mAch-R Scopolamine Rats amygdala 50 μg - Histamine H3-R
(inverse agonist)Thioperamide Rats amygdala 44 pg - Receptor agonists GABAa-R Muscimol Rats IL 4 nmol/side - Rats PL 4 nmol/side + 5-HT7-R AS-19 Rats CA1 5 μg/side - 5-HT6-R WAY-208466 Rats CA1 0.04 μg/side +/− Cannabinoids CB1 and CB2-R agonist Anandamide Rats CA1 0.17 ng/side + CP55,940 Rats CA1 2.5 μg/side + Rats IL 2.5 μg/side + Rats RSC 2.5 μg/side + Intracellular molecule inhibitors PARP-1 3-aminobenzamide Mice dHipp 18 μg/side + Mice mPFC 18 μg/side - PJ34 Mice dHipp 0.2 mM/side + LIM kinase BMS-5 Rats CA1 200 μm/side + PKC Chelerythrine Rats PL 3 nmol/side +/− PKMζ ZIP Rats PL 10 nmol/side +/− MEK U0126 Rats dHipp 2/4 μg/side - IKK Sulfasalazine Rats dHipp 2 μg/side + and - Rats i.c.v. 5/10 mM +/−4 Proteasome β-lac Rats dHipp 32 ng/side - Calpain ALLN Mice CA1 1 μg/side + PD150606 Rats CA1 0.153 ng/side + Rac NSC23766 Rats BLA 5 μg/side - Rats CeA 5 μg/side - Rats CA1 5 μg/side + mTOR Rapamycin Rats dHipp 5 μg/side + N-glycosylation inhibition Swainsonine Mice dHipp 0.5 μg/side + 1-deoxynojirimycin Mice dHipp 16 μg/side + Tunicamycin Mice dHipp 0.5 μg/side + Other Glutamatergic system blocker Riluzole Rats dHipp 2 μm/side + Sodium channel blocker Tetrodotoxin Rats Ent 5 ng/side + Rats amygdala 5 ng/side + Hormone Angiotensin II Rats CA1 0.5 nmol/side * Peptide Nociceptin Mice i.c.v 1/3 nmol +/−4 Cytokine IL-1β Rats CA1 5 ng/side + Additional details for each study, including PubMed ID, strain, duration of the reactivation session (ranging from 1 to 10 min), time of drug administration, and time between training and reactivation session (ranging from 1 to 36 d), are available at https://osf.io/x2pkq/. ACC = anterior cingulate cortex; ALLN = N-Acetyl-Leu-Leu-norleucinal; BLA = basolateral amygdala; CeA = Central amygdala; dHipp = dorsal hippocampus; D-AP5 = D-2-amino-5-phosphonovaleric acid; DRB = 5,6-dichloro-1-b-dribofuranosylbenzimidazole; Ent = entorhinal cortex; IL = infralimbic cortex; i.c.v. = intracerebroventricular; MC = motor cortex; mPFC = medial prefrontal cortex; PL = prelimbic cortex; RSC = retrosplenial cortex; + = at least one study reported a statistically significant amnestic effect; * = amnestic effect was found to be transient; - = at least one study reported a non-significant effect; +/− = at least one study observed that the amnestic effect occurred under some conditions (superscripts see Table 3).
In this issue
