Reviewing Registered Reports

Background

A handful of publications offer Registered Reports, a type of empirical article where the techniques and suggested analysis are pre-registered and vetted before the research is carried out. Before data collecting starts, high-quality protocols are then tentatively accepted for publishing. With the ability to conduct exploratory (unregistered) studies and report coincidental findings, this approach is intended to reduce publication bias and research bias in hypothesis-driven research.

There are two stages to the evaluation process for registered reports. Before any data is gathered, reviewers evaluate study ideas at Stage 1 (Study Design). Reviewers take into account the entire study, including the data and interpretation, at Stage 2 (Completed Study).

Recommendations for Reviewers

Only the Introduction, Methods (including any suggested analysis), and Pilot Data will be included in Stage 1 papers (where applicable). Reviewers will be invited to rate articles at Stage 1 based on the following criteria:

• the significance of the research question(s), in publications where this is typically a requirement for admission.
• The offered hypotheses’ reasonability, logicalness, and believability.
• The technique and analysis pipeline’s validity and viability (including statistical power analysis where appropriate).
• to what extent the proposed experimental protocols and analysis pipeline can be precisely replicated in terms of clarity and level of technical information
• Whether the authors have included positive controls and quality checks as well as enough outcome-neutral tests to guarantee that the results are able to evaluate the stated hypotheses.

After Stage 1: Study Design peer review, manuscripts will either be approved, given a chance to be revised, or outright refused. Upon successful completion of the study using the pre-registered methods and analytic procedures, as well as a defensible and evidence-based interpretation of the findings, manuscripts that pass peer review will be given an in principle acceptance (IPA), which indicates that the article will be published subject to these conditions. Be aware that some journals may decide to publish both the final Stage 2 completed study and the Stage 1 Study Design after receiving an IPA, while other journals will only publish the Stage 2 final article. For more information, please refer to the individual Registered Reports rules for your publication.

            The authors will finish the manuscript, including the Results and Discussion sections, after the study is over. These Stage 2: Completed Study manuscripts will be written in a style that is more like to an ordinary article. The reviewers will then receive the article again and be tasked with rating:

• Whether the data can test the hypotheses put forth by the authors by meeting the deemed outcome-neutral conditions (such as quality checks, positive controls).
• If the Introduction, justification, and stated hypotheses are the same as the Stage 1 proposal that was approved (required).
• Whether the authors followed the registered experimental protocols precisely.
• if the authors’ unregistered post hoc analyses are justified, informative, and methodologically sound.
• Whether the data support the writers’ conclusions.

However, authors are not required to do so unless further post hoc tests are required to meet one or more of the Stage 2 review requirements. Stage 2 reviewers may propose that authors disclose additional post hoc tests on their data. Please be aware that editorial decisions will be made based on data-supported conclusions (even if they are negative findings) and adherence to the approved protocols and experimental design in Stage 1 rather than novelty and perceived importance of results.