There are many things to take care of in empirical research. The research design, programming the experiment, piloting, gathering a decent pool of participants, giving them clear instructions, handling complex technical equipment, analysing the data, communicating the findings… Every step of the way, a researcher can make this process run more or less smoothly. But there is one part of a research study that should without a doubt be beyond our control. Which part is that?
It’s the results. They are what they are. In fact, if we tried to exert an influence on the results, we’d be in serious trouble.
And which part of a research study is most important for advancing your career? Well, yes, it’s the results…
Hmm.
This how Chris Chambers opened his talk on Registered Reports.
And that is indeed a clear illustration of the current incentive problem in science: the one thing you’re not supposed to control, influences your career hugely. In this results-driven culture, what’s best for science has less and less overlap with for what’s best for scientists.
Once researchers are placed under relentless pressure to produce exciting results, they will do that by circumventing the scientific process in a variety of creative ways. The scientific process is abundant with vulnerabilities. Science needs high quality research, published regardless of the outcome. Not just lots of unreliable but ‘good’ results.
When it comes to hypothesis testing, the scientific value lies in the question our hypothesis asks, the quality of methods it uses, and never in the result it produces. Fixing this situation requires a philosophical shift in the way we think about our research. If the results tell us nothing about the quality of the science, then editorial decisions should surely be blind to results, shouldn’t they?
Registered reports emerged from this philosophy. A number of journals including Cortex have started to accept registered reports as an alternative to traditional peer-review. The results can only bias us, given the existing incentive structure. Registered reports are a results-blind review, done before results even exist. The emphasis is shifted to the processes that lead to those results.
Creating a Registered Report
What do these registered reports look like? The review process has two stages. The first stage centers around hypotheses, experimental procedures, and the proposed analyses. Acceptance is based on the importance of the question as well as the rigor of the methods. The study also needs to be adequately powered (<90%), and to include relevant controls. The manuscript will be methods-heavy, in fact, it should be sufficiently detailed so that some other researcher could run the same experiment after reading it. No data collection (except possibly a pilot) is done at this point.
Data collection takes place only after In Principle Acceptance (IPA). The study is virtually guaranteed to be published after this stage, regardless of the results.
And then the data get collected and analysed, and a stage 2 manuscript is submitted. A stage 2 manuscript will contain the same introduction and methods section as the stage 1 manuscript. The results are now added, with a clear delineation between registered confirmatory analyses and unregistered exploratory analyses. The discussion, will, of course depend on the results as well.
What is the job of the reviewers in this stage of peer review? Reviewers assess whether the authors followed the initially outlined protocol, whether any potential deviations from the plan were justified, and whether the conclusions are sound, given the data. Statistical significance is irrelevant for publication. Whether the hypothesis was supported by the data is irrelevant. Whether the results are novel is irrelevant. Whether they are subjectively judged to be impactful is irrelevant. If the study is properly conducted, the paper gets published. Oh, and the data need to be uploaded to a public repository as well.
And guess what? Registered reports get cited more than traditionally peer-reviewed articles within the same journal. Could it be because they are more scientifically sound? Head over to Cortex and decide for yourself,
Click on the video below to hear Chris’ compelling talk on why registered reports are a good thing for science, and a good thing for scientists, too:
(This blog post is a preview of the themes in Chris Chambers’ talk at the Oxford Reproducibility School, held on September 27, 2017. I occasionally use my own words to describe the contents, but the presented ideas do not deviate from the talk.)