Talking About Registered Reports

Since my earlier blog post about registered reports, I've found myself having quite a few conversations about them. I thought it might be useful to share some of the most common points people have raised, and my thoughts on them.

What are Registered Reports?

Registered Reports are a type of article where the authors submit their introduction, detailed method section and pilot data ahead of time, and get those reviewed and accepted before they collect the real study data. It's like when you do a PhD dissertation. First you come up with a proposal, and you show it to your committee, and they meet and talk about it and maybe give you feedback, tell you to change things, and then you go to the work. And everyone understands that if you basically follow the plan, you're going to get the PhD. Similarly, with Registered Reports, the journal reviews the introduction and methods and gives you an in-principle acceptance, which says that if you follow the plan that everyone has agreed to, the paper will get published no matter what the results show. There's lots of information about them here, and I also made this slide presentation to explain them at a meeting a few months ago. (Feel free to use and repost the slides, no attribution necessary.)

What about exploratory/discovery research?

Some folks are worried that registered reports glorify hypothesis-testing at the expense of more exploratory research. I certainly agree that science needs a back-and-forth between exploration/discovery (to generate hypotheses) and experiments (to test them). But registered reports don't inhibit exploratory research. Here's why:

1. Registered Reports are totally optional.

Nobody has to submit one if they don't want to; the old article types are still available.

2. Even within a registered report, you can include all the exploratory findings and unexpected discoveries you want.

You just can't rewrite the introduction to pretend that you hypothesized those things in the first place. This is called HARKing-- (Hypothesizing After the Results are Known) and our current culture basically forces us to do it. I love your description of exploratory research, but most journals won't publish research like that. Instead, it has become customary for authors to make up a story to justify the findings, as if they had been hypothesized.

Here's a silly example: Say I observe a bunch of preschoolers, and I notice that the kids all count toy animals in Spanish, but they count toy vehicles in English. I can notice that, and I can measure it, but no one will publish it. In order to publish it, I have to dig through the literature and come up with some story about what could be going on, and write a paper saying, "Maybe kids count animals in Spanish and vehicles in English. This is plausible because of (citation, citation, citation), and it was even observed by (citation). However, it was not mentioned by (citation), and it's really important because (plausible reason). So in this study, we measured whether children count animals in Spanish and vehicles in English..." In other words, I have to imply that I hypothesized this thing based on the literature and then went out and voila! Found it.

Developmental psychology is FULL of studies where somebody found something unexpected and then made up a story to explain it. We all do this because it's the only way to publish our results. When I submit interesting findings without a compelling 'story' to go with them, I get a rejection or a revise-and-resubmit demanding a better story. So I come up with a story. To be clear, I *believe* the stories I come up with. They are the best post-hoc explanations I can generate with for my findings. But they are still *post-hoc,* and I'm sick of having to present them as a priori hypotheses in order to get the findings published. Registered reports allow us to be honest about what was predicted and what wasn't. So the kind of lovely exploratory research you describe doesn't have to be (falsely) presented as hypothesis-testing.

3. Exploratory Reports are also a thing.

A few journals (such as Cortex) offer yet another new article type, called Exploratory Reports. They're like Registered Reports, but for explicitly exploratory studies.

I think it's a great idea for authors to be intentional about what they will analyze, observe and measure. You can't just analyze and observe and measure everything, after all.

If I register my idea, someone might scoop me!

Submitting a registered report does not make your idea public. The only people who read your Stage 1 submission are the reviewers and the editor at the journal. If the journal gives you an in-principle acceptance (IPA), then they will keep your approved Stage 1 manuscript on file (or deposit it somewhere under an embargo, so that no one can see it) until you submit the Stage 2 manuscript, which includes the whole paper. Then the whole thing gets published, just like a traditional article type. So no one except the editor and the reviewers get a sneak peek at your research plan.

If you're afraid that the reviewers will scoop you, keep in mind this is the same situation as when you submit a grant proposal. You have to let reviewers see your plans, and trust that they won't steal your ideas. (Honestly, this scooping thing is a much bigger problem in young investigators' imaginations than in real life.)

I have no use for Registered Reports because I never feel pressured to do any naughty things.

OK, I get it. You never do anything questionable because you exist outside the system of incentives that the rest of us have to work in. But surely not every paper you review is that perfect, right? Even if your own work is unimpeachably correct, you must review a lot of papers that have problems-- I certainly do. And I'm eager to give feedback to those authors early in the process, rather than later. Developmental data collection is time-consuming and expensive. I feel bad when I have to reject a paper that people have spent months or years on . . . but sometimes those papers have serious problems and must be rejected. Sometimes the authors collected data that just can't answer the question they wanted to ask. Sometimes what they're trying to do has already been done. If I could give them feedback before they spent two years conducting an unpublishable study, and give them the chance to improve the research design (or at least stop wasting their time on a lost cause), that would be a much better use of everyone's time and research funds.

I regard change with fear and loathing.

I know, change is scary. But Registered Reports are a good thing, both for individual scientists and for science as a whole. And anyway if you don't like them, you don't have to try them. They're just another option that's available-- they're really nothing to be scared of.