Contact: Maya B. Mathur
For each study, we conducted analyses for three subsets: (1) all replications, regardless of which protocol they used; (2) only replications using the RP:P protocol; and (3) only replications using the Revised protocol. For each subset, we meta-analyzed the replications using methods that do not use any distributional assumptions and that provide valid inference even for small sample sizes (Hedges et al., 2010; Tipton, 2015). In the two cases in which the model failed to converge (noted in the column robu.error
in the results file results_table_full.csv
), we instead used parametric meta-analysis fit via REML (Viechtbauer, 2010). We conducted all analyses on the Fisher's
We used robust methoods to estimate Psignif.agree
). The file results_aggregated_by_subset.csv
aggregates these results by subset (all replications, RP:P, Revised), showing the mean heterogeneity estimate $\widehat{\tau}$, the percentage of studies with $\widehat{\tau}>0$, the mean values of $P{\text{orig}}$, $\widehat{P}{>0}$, $\widehat{P}{>0.10}$, and $\widehat{P}{>0.20}$, and . For replication sets that had a heterogeneity estimate $\widehat{\tau}=0$ exactly or which had only 1 replication, we simply report the $\widehat{P}{>q}$ as 100% or 0% depending on whether the single point estimate was above or below
Results of these analyses for all subsets are in results_table_pretty.csv
and its more verbose version, results_table_full.csv
. Results stratified by subset are in results_table_pretty_combined.csv
, results_table_pretty_rpp.csv
, and results_table_pretty_revised.csv
.
Interestingly, despite our close standardization of protocols across sites, 40% of replication sets within each of the 3 subsets had