Show simple item record

Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

dc.contributor.authorEbersole, Charles
dc.contributor.authorMathur, Maya B.
dc.contributor.authorBaranski, Erica
dc.contributor.authorBart-Plange, Diane-Jo
dc.contributor.authorRopovik, Ivan
dc.date.accessioned2024-11-11T14:10:57Z
dc.date.available2024-11-11T14:10:57Z
dc.date.issued2021
dc.identifier.urihttps://hdl.handle.net/20.500.14178/2687
dc.description.abstractReplication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3-9; median total sample = 1,279.5, range = 276-3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Delta r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols (r = .05) was similar to that of the RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than that of the original studies (r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00-.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19-.50).en
dc.language.isoen
dc.relation.urlhttps://doi.org/10.1177/2515245920958687
dc.rightsLicenci mají autoři a ti chtějí výsledek zveřejnit volně. Mají zveřejněno s informací free access.
dc.titleMany Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicabilityen
dcterms.accessRightsopenAccess
dcterms.licensehttps://journals.sagepub.com/doi/10.1177/2515245920958687
dc.date.updated2024-11-11T14:10:56Z
dc.subject.keywordreplicationen
dc.subject.keywordreproducibilityen
dc.subject.keywordmetascienceen
dc.subject.keywordpeer reviewen
dc.subject.keywordRegistered Reportsen
dc.subject.keywordopen dataen
dc.subject.keywordpreregistereden
dc.identifier.eissn2515-2467
dc.relation.fundingReferenceinfo:eu-repo/grantAgreement/MSM//PRIMUS/20/HUM/009
dc.relation.fundingReferenceinfo:eu-repo/grantAgreement/UK/PROGRES/Q17
dc.date.embargoStartDate2024-11-11
dc.type.obd73
dc.type.versioninfo:eu-repo/semantics/publishedVersion
dc.identifier.doi10.1177/2515245920958687
dc.identifier.utWos000707042400001
dc.identifier.eidScopus2-s2.0-85096661669
dc.identifier.obd601705
dc.identifier.rivRIV/00216208:11410/21:10434047
dc.subject.rivPrimary50000::50300::50301
dc.subject.rivSecondary50000::50100::50101
dc.relation.datasetUrlhttps://osf.io/7a6rd/
dcterms.isPartOf.nameAdvances in Methods and Practices in Psychological Science
dcterms.isPartOf.issn2515-2459
dcterms.isPartOf.journalYear2021
dcterms.isPartOf.journalVolume3
dcterms.isPartOf.journalIssue3
uk.faculty.primaryId117
uk.faculty.primaryNamePedagogická fakultacs
uk.faculty.primaryNameFaculty of Educationen
uk.department.primaryId1584
uk.department.primaryNameÚstav výzkumu a rozvoje vzdělávánícs
uk.department.primaryNameInstitute for Research and Development in Educationen
dc.description.pageRange309-331
dc.type.obdHierarchyCsČLÁNEK V ČASOPISU::článek v časopisu::původní článekcs
dc.type.obdHierarchyEnJOURNAL ARTICLE::journal article::original articleen
dc.type.obdHierarchyCode73::152::206en
uk.displayTitleMany Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicabilityen


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record