 |
| Figure showing a range of problems (in red) that are addressed by Registered Reports |
Once the reviewers and authors are in agreement about the research plan, the journal gives the authors an in-principle acceptance, which says that the journal agrees to publish the work no matter how the findings come out, as long as the authors follow the agreed-upon plan and draw reasonable conclusions from the data.
This is so obviously a much better way to do science! It has so many benefits for researchers. I love the idea of having reviewers' input on my design before I spend months or years collecting the data, and it would be so reassuring to know, while I did the actual work, that the paper had definitely been accepted for publication.
You know that nightmare scenario, where you spend years doing a project and then when it's finished, you can't publish it because it turns out that (1) someone already did this ten years ago, or (2) there's some mistake in the design, or (3) the findings are just boring or (4) one of the reviewers just doesn't like the findings, so they demand that you go back and do more work, or change the framing of the study so it's more to their liking?
Here's a silly, imaginary example. Say I measure a bunch of preschoolers' ice cream preferences, and I find that most of the two-year-olds and four-year-olds like chocolate ice cream best, but most of the three-year-olds like vanilla. And I think this is interesting because some studies have previously said that kids always like chocolate ice cream best, because there's some chemical in it that tickles their dopamine receptors or whatever. (I know it's silly--stay with me here.)
So I write up a study saying, "The preference for chocolate ice cream is not universal-- three-year-olds don't share it," and I submit it for publication. It is sent to three reviewers, including one or two who have published several papers explaining why children will always like chocolate ice cream best, and how this fact tells us deep things about the brain, and so forth. And those reviewers don't like my findings. They say, "No way-- we're not convinced! We know that children always prefer chocolate ice cream. We have written important papers about it! So there must be some other explanation for your data. You probably didn't collect it right. It was the wrong time of day, probably. And the spoons you used were too cold. And those kids probably just didn't want to eat up all the chocolate ice cream because it was the best, and they wanted some to be left over for the other kids! So all you actually measured was generosity. This paper is no good and we can't believe anything it says." The paper is rejected.
So I go do a series of follow-up experiments to rule out the as many of the reviewers' alternative explanations as I can. I control for the time of day and the coldness of spoons and all that stuff, and I resubmit the new manuscript somewhere else. The new reviewers say, "Well, maybe you've got something here, but we're not sure. Maybe the three-year-olds have heard that chocolate is bad for them. Or maybe they love chocolate so much that they eat it all day long, and then when you come along with your chocolate ice cream, they have tummy aches." So back I go, to collect more data to test those explanations, and I still find that the three-year-olds like vanilla ice cream, and I resubmit the manuscript.
Finally the reviewers say, "Well OK, you've got a series of experiments here showing that three-year-olds eat vanilla ice cream, but you haven't shown that they enjoy it more than chocolate, have you? (Don't you know that chocolate has a special chemical that tickles their dopamine receptors?) So you're going to need to go back and rewrite your introduction to frame the whole thing as a study of eating behavior, not preferences. Because it's still possible that the three-year-olds like chocolate ice cream better."
Of course I think this is idiotic, but I've done all this work and I want the thing published. So I rewrite the introduction to very carefully say that we measured behavior and not preferences, and that it's still totally possible that all kids everywhere like chocolate ice cream best (even though I think this is highly unlikely) because at this point I just want the damn thing off my desk. Finally, the paper gets published.
Contrast this with a registered reports scenario: I propose to some journal that I'm going to measure ice-cream preferences in children, and the reviewers and I agree ahead of time what will be done. How many kids I will test, what ages, how the data will be analyzed, all that stuff. Even if it's those same reviewers, the ones who are really invested in the idea that kids like chocolate ice cream best, they have to raise all their objections before the data are collected. Cold spoons, tummy aches, time-of-day considerations, whatever: Let's agree on the whole plan in advance. And once I get an in-principle acceptance from the journal, I know the journal is going to publish the study no matter how the results come out. So the reviewers can't start bitching and moaning about methods after the fact, or demand that I reframe the question just because they were disappointed or unconvinced by the results.
The fact is, good research is valuable no matter how the results come out. If I spend a year looking for an effect and it's just not there . . . well, that should still be publishable. At least it will save everyone else from wasting a year looking for the same thing. I was talking with colleagues at the Mathematical Cognition and Learning Society conference in Oxford last month and we were saying, "How many failed preschool math interventions have we all run? Some intervention seems like it should work, so you do it, but it doesn't work, so you can't publish it. And meanwhile, all the other labs are trying the same thing, and it's also not working for them. It's a huge waste of time and money!" But if those intervention studies were done as RRs, they would be published-- no matter how the results came out. SO much better for us researchers and for science! (This was the theme of the brief talk I gave about RRs at the same meeting.)
In my lab, we're always saying that we can't wait to finish all the projects that were started before we learned about RRs, so that we can move on to doing new projects as RRs. I figured that was the only reason people weren't doing RRs--either that, or people just didn't know about them. But this month I realized that there's another barrier: The tabloid culture of journals.
I had an eye-opening email exchange with the editor of a major journal in my field. The whole exchange is below. Chris Chambers is a neuroscientist and a leading advocate for registered reports. He makes the case for them in his recent book, which I highly recommend. When I signed on to the Journal of Numerical Cognition as their Associate Editor for Registered Reports, I reached out to Chris for help with the process, and he's been really responsive. At some point I started bcc-ing him on the exchange with the editor, so his emails are included below.
NOTE: I've paraphrased the emails from the editor using my own writing style, so that they can't be identified through any little linguistic quirks. (The emails from me and Chris Chambers are still the originals, except that I've made all pronouns referring to the editor plural, in order to avoid revealing their gender.) It makes the whole exchange a little less authentic, but I really don't want to embarrass or 'out' the editor, or the journal. They were not expecting these emails to be made public.
<><><><><><><><><>
FROM: [Name]
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Dear Professor Sarnecka,
Hello! I'm the Managing Editor of [Journal], and I'm writing to you because we're looking to expand our editorial board. Your recent publications in [Journal], together with your expertise in cognitive development and number, and your more recent interests in the development of judgment and decision making, make you ideally suited to cover the areas where we need help. So I'm writing to ask whether you would be interested in joining us as an Associate Editor (AE). Our AEs handle 15 to 20 submissions per year, plus resubmissions of those papers. AEs are responsible for recruiting reviewers (using the board of reviewers, our reviewer database, and their network of connections), making decisions on the basis of the reviews, and writing action letters to authors. Of course we'd like this to be done in a timely manner whenever possible. You would be compensated $1200 per year for your trouble.
Please let me know if you are interested, and I'd be happy to answer any further questions.
Best wishes,
Best wishes,
[Name]
<><><><><><><><><>
DATE: 16 May 2018
FROM: Barbara Sarnecka
TO: [Name]
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Dear [Name],
DATE: 16 May 2018
FROM: Barbara Sarnecka
TO: [Name]
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Dear [Name],
Hello, and thanks for the invitation! I’ve actually been thinking a lot lately about service to the field, and about how we can improve the quality of science in our field going forward.
One thing I’ve become passionate about is Registered Reports (https://cos.io/rr/). As you probably know, this is an article type where authors submit their introduction, methods and a detailed analysis plan for peer review, prior to doing the actual work. The submissions thus get reviewed in two stages: Once for the plan (introduction, methods, analysis plan, pilot data) and again when the work has been done.
One thing I’ve become passionate about is Registered Reports (https://cos.io/rr/). As you probably know, this is an article type where authors submit their introduction, methods and a detailed analysis plan for peer review, prior to doing the actual work. The submissions thus get reviewed in two stages: Once for the plan (introduction, methods, analysis plan, pilot data) and again when the work has been done.
Registered reports are (in my opinion) clearly a better way to do science. They allow reviewers and editors to give input when it can still do some real good, and they eliminate many forms of bias (publication bias, hindsight bias, p-hacking, harking) that undermine the quality of our science. They are now offered by 105 journals (click the tab for “Participating Journals” at https://cos.io/rr/ to see a list).
This is all preamble to say that although I would not be interested in serving on the editorial board in the regular capacity, I *would* be willing to serve as Editor for Registered Reports if [Journal] agrees to adopt them. I’m serving in that role now for the Journal of Numerical Cognition, where we’re just finishing up the process of setting up Registered Reports as an article type.
In case you’d like to read more about it, here’s an introduction for journal editors . . . https://docs.google.com/document/d/1YAvhtSiTYGmDKfsQsPqccvOo9-YklU1HK7atVLG18C4/edit?usp=sharing
And here’s a list of FAQs. https://docs.google.com/document/d/1VpRebTKltTo6wrJX8alfNqbCzVFjTaHnxQyUWIdVB3I/edit?usp=sharing
Of course I realize that [Journal] may not be interested in offering Registered Reports right now. If that’s the case, then I’ll politely decline your invitation, but I do thank you for the offer.
Warmest wishes,
Barbara Sarnecka
<><><><><><><><><>
DATE: 16 May 2018
FROM: [Name]
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Dear Barbara,
Thanks for your speedy reply. As a matter of fact, we are adopting some of the recommendations from the Center for Open Science. For example, in the future we expect to offer badges for open data and open materials. I'm afraid I'm not fully convinced that registered reports would be right for [Journal]. Part of the problem is that we don't have the resources, but RRs also seem better suited to pure psychology journals; they just would not be appropriate for some of the studies we publish. Another issue is that I don't think RRs would fit some of the kinds questions that we address. (I'm thinking of [famous author's] well-known [famous paper].)
But in any case, I'm sure the issue will come up at our annual board meeting, and I will certainly pass along your enthusiasm.
Thanks again for getting back to me so quickly.
Best wishes,
[Name]
<><><><><><><><><>
DATE: 17 May 2018
FROM: Barbara Sarnecka
TO: [Name]
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Hello [Name],
I'm delighted to hear that [Journal] will soon be launching badges for Open Data and Open Materials-- those seem like excellent steps forward.
If the board decides to explore Registered Reports in the future, I'll be happy to help out however I can.
Just to address the two concerns you raised (1) I don't think that offering RRs requires any additional resources, and (2) RRs don't *replace* traditional articles, they're just another option available to authors. Studies that aren't suitable for the RR format (e.g., modeling studies, or the [famous author] paper) can still be published the same way they always have.
In any case, thanks for the invitation, and I wish you all the very best of luck in the future.
Warm regards,
Barbara W. Sarnecka
<><><><><><><><><>
DATE: 17 May 2018
FROM: Christopher Chambers
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Hi Barbara
Well firstly, hats off for putting RRs in as a condition of editorial board membership (and thank you for standing up for them!)
I’ve been doing the same for several years now and have found it deeply illuminating.
It always worries me when a journal says it lacks the resources to offer RRs. What do they envisage it involving? Limousines and Cristal? (we’ve cut back on them, honest). Sometimes when editors respond like this it turns out they mean intellectual resources, which is doubly worrying because it means they basically don’t have the expertise to assess theory and methods without seeing results. And that is truly concerning about the capabilities of our “top” scientists.
Ironically this very same editor admonished a colleague on the very same day about that colleague’s support of the PRO initiative thusly: "Editors need to change policy because they agree with the policy change (and so by all means they should be lobbied directly and assisted in making such changes) *my emphasis*, not because they are being forced into a corner by militant reviewers.” And yet when given the opportunity to implement an important open science initiative on the same day, with the generous assistance of a willing colleague, [they] turn that offer down. Says a lot.
best, Chris
<><><><><><><><><>
DATE: 17 May 2018
FROM:[Name]
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Dear Barbara,
Thanks for your most recent message.
The additional resources I had in mind were, (1) the work required on the submission site to support the RR workflow, and (2) a dedicated Associate Editor. I agree that after RRs were set up, it might turn out to be the same amount of work overall, assuming that some proportion of the submissions we are getting now would come in as RR submissions instead. (Hopefully not too many and not too few.)
But I would still argue that [Journal] is not a pure psychology journal - we are explicitly multi-disciplinary and interdisciplinary. I think the RR format is better suited to a single-discipline, experimental psychology journal.
Best wishes,
[Name]
<><><><><><><><><>
FROM: Barbara Sarnecka
TO: [Name]
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Dear [Name],
Not to flog a dead horse, you *do* have a dedicated Associate Editor (me) and I’m willing to work with the publishers to set up the submission site, as I am doing at JNC. I do see your point about [Journal] being a special journal. But to the degree that it publishes any articles where authors use data to test hypotheses, that work would be better (more reliable, more trustworthy, more methodologically sound, less biased) if it were produced through a registered-report process than through the traditional process.
Of the 105 journals that have adopted registered reports to date (go to https://cos.io/rr/ and click on ‘Participating Journals’) only a minority are what I would call ‘pure psychology’ journals.
Nevertheless, institutional change is difficult and I’m sure you are doing everything you can, at the pace that is realistic for [Journal] right now. I applaud you for embracing Open Data and Open Materials, and I wish you all the best of luck with the journal.
Take care,
Barbara
<><><><><><><><><>
FROM: Christopher Chambers
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Great response. [Journal] is a [Publisher] journal and [Publisher] not only already offer RRs across many of their journals (including one that I edit for), but they have also created an editor’s pack for them in which the entire manuscript handling process can imported into the journal with a few mouse clicks. It couldn’t be easier.
If [they] go for this, feel free to copy me in explicitly and I’d be happy to connect you and [him/her] with the relevant people at the publisher.
As you say, there is nothing special about [Journal]. RRs don’t need to apply to all kinds of research that it is submitted to the journal to have value. I mean, consider the fact that they are offered by Royal Society Open Science, across 205 different sciences. They are offered for all of them, and those areas for which it is relevant submit them.
<><><><><><><><><>
DATE: 17 May 2018
FROM: Barbara Sarnecka
TO: [name]
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Dear [Name],
Another quick note (although you’re probably ready for this exchange to be finished) . . . [Publisher] already offers RRs at a number of its journals, and I’m told that they’ve even created an editor’s pack, in which the entire RR manuscript handling process can be imported in a few mouse clicks. So when the day comes that you are ready to implement RRs, it won’t be any trouble at all.
Best wishes,
Barbara
<><><><><><><><><>
DATE: 18 May 2018
FROM: [Name]
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Hi Barbara,
Listen, I hear what you are saying. But right now, I really have to think about filling the empty Associate Editor spots. Several of our AEs have recently quit, and but the submissions keep pouring in. I just can't think about any big changes until I have a full team of AEs again. I just don't have the attention to spare.
Best wishes,
[Name]
<><><><><><><><><>
DATE: 18 May 2018
FROM: Barbara Sarnecka
TO: [Name]
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Hi [Name],
I definitely understand. It sounds like you are really slammed with editorial work, and I’m sure you still have a lab to run at the same time.
What about if I agreed to handle 10 regular papers while we were setting up the registered reports? Would that give you a little breathing room? Chris Chambers (who is AE for registered reports at another [Publisher] journal) has offered to connect me with the right people at [Publisher] and advise on the set-up process, keeping your cognitive load to a minimum.
Or perhaps now just isn’t the right time-- I know what it feels like to be overwhelmed. Feel free to check back with me in a year or two, or whenever you feel ready to think about RRs. I can’t promise I’ll still be available, but there’s no harm in emailing.
Until then, best of luck with everything, and I hope the spring in [Editor's city] is nice. We don’t have much seasonal change here in Southern California, and I think spring is so much more joyful when it comes at the end of a long winter.
Best wishes,
Barbara
<><><><><><><><><>
DATE: 19 May 2018
FROM: [Name]
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Thanks. Yes - now is not the right time. But I'll raise it with the board at the annual meeting, and get back to you if folks are interested in doing this.
Best wishes,
[Name]
<><><><><><><><><>
DATE: 19 May 2018
FROM: Barbara Sarnecka
TO: Christopher Chambers
SUBJECT: Re: Invitation to serve as an editor for [Journal]
An interesting exercise, in the end. Change is anxiety-provoking. At least in this case, it clearly seemed to be about anxiety: [Name]'s multiple, shifting excuses for why [they] couldn't do it, culminating in, “I just can’t think about this right now!”
I sympathize. We all seem to take on more than we can handle, and then we panic about everything we’re not getting done. In that state of mind, the idea of dealing with anything new sounds intolerable. I suspect that it’s this mundane fact— the fact that most people with decision-making power feel personally overwhelmed most of the time— that slows change as much as anything else.
Anyway, thanks for all the support. You’ve been so generous with your attention and guidance, and I really appreciate it. I hope that someday I will have the chance to thank you in person.
Take care,
Barbara
<><><><><><><><><>
DATE: 19 May 2018
FROM: Christopher Chambers
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Hi Barbara,
You’re most welcome, it’s been a pleasure to help you get going at JNC and I’m sure the journal will go on to do great things with the format!
If it’s any consolation, your interaction with [Name] is typical of dozens of interactions I’ve had with journal editors about RRs in the last 5+ years…many like to talk the talk on open science issues but when offered the opportunity to actually do something transformative they are too shy (either work shy or just feel that RRs are too radical) and the reasons for declining to offer them continually shift…one minute it’s workload, and you address that, then it’s that the format won’t suit all article types, then you address that, then it shifts to something else. I have long suspected (and even got some editors to admit off the record) that at the core of their objections is an almost instinctive discomfort with selecting what to publish before seeing the results because they judge QUALITY using results, not theory and methods and they fear that the journal will publish more negative results (which it will!). Editors like this are basically hostages to the worst incentives in the field and I’m afraid my interactions with them about RRs have made me quite cynical about their motives and expertise and (perhaps most importantly) their professional courage.
I can also virtually guarantee that you will never hear from [Name] again. As much as I would love to be proved wrong, when editors raise the idea in a half hearted way with their own boards or governing councils, the concept is universally dismissed based on the laziest, most ignorant comments. [Their] governing board will nod along and the idea will be quietly shelved because without an advocate it it just too radical — that’s of course if they raise it all which I suspect they won’t (even if they say otherwise — I wish I had a dollar for every time an editor had fobbed me off with this line, followed by me finding out from other board members that RRs were not mentioned in the meeting). This, incidentally, is why we now publicly record most of the approaches we make to journals and the outcomes (https://osf.io/3wct2/wiki/Journal%20Responses/)
Good luck at JNC and please consider me a sounding board at any time!
very best, Chris
<><><><><><><><><><><><>
So that was it-- my email exchange with the editor. I guess we'll never know why this editor was unwilling to implement Registered Reports. I guess change is scary, and people are busy, and journal editors want 'sexy' results, even if those results are less trustworthy because they weren't preregistered, could be a fluke, are probably at least a little bit HARKed and p-hacked, and so on.
As long as journals exist to make money (and that's certainly why they exist at Elsevier, Wiley, Springer, and Taylor & Francis), they're going to care about getting attention and selling subscriptions, even at the cost of some scientific integrity. If your article will get a lot of attention, the journal isn't going to look too hard or too critically at how you did the work. Registered Reports would take away the journal's ability to accept and reject papers based on their findings, which would be good for science and good for scientists, but very bad for the publishers.
Ah, the publishers . . . what a sweet deal they have. We give them their content and their quality control for free, and they walk away with 35% profits as a proportion of revenue. The president of Elsevier got some crazy pay package last year-- something like twenty million dollars? So I guess no one should be surprised when they prioritize marketing over science.
[For more chat about registered reports, see my later blog post on Talking About Registered Reports].
This is all preamble to say that although I would not be interested in serving on the editorial board in the regular capacity, I *would* be willing to serve as Editor for Registered Reports if [Journal] agrees to adopt them. I’m serving in that role now for the Journal of Numerical Cognition, where we’re just finishing up the process of setting up Registered Reports as an article type.
In case you’d like to read more about it, here’s an introduction for journal editors . . . https://docs.google.com/document/d/1YAvhtSiTYGmDKfsQsPqccvOo9-YklU1HK7atVLG18C4/edit?usp=sharing
And here’s a list of FAQs. https://docs.google.com/document/d/1VpRebTKltTo6wrJX8alfNqbCzVFjTaHnxQyUWIdVB3I/edit?usp=sharing
Of course I realize that [Journal] may not be interested in offering Registered Reports right now. If that’s the case, then I’ll politely decline your invitation, but I do thank you for the offer.
Warmest wishes,
Barbara Sarnecka
<><><><><><><><><>
FROM: [Name]
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Dear Barbara,
Thanks for your speedy reply. As a matter of fact, we are adopting some of the recommendations from the Center for Open Science. For example, in the future we expect to offer badges for open data and open materials. I'm afraid I'm not fully convinced that registered reports would be right for [Journal]. Part of the problem is that we don't have the resources, but RRs also seem better suited to pure psychology journals; they just would not be appropriate for some of the studies we publish. Another issue is that I don't think RRs would fit some of the kinds questions that we address. (I'm thinking of [famous author's] well-known [famous paper].)
But in any case, I'm sure the issue will come up at our annual board meeting, and I will certainly pass along your enthusiasm.
Thanks again for getting back to me so quickly.
Best wishes,
[Name]
<><><><><><><><><>
FROM: Barbara Sarnecka
TO: [Name]
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Hello [Name],
I'm delighted to hear that [Journal] will soon be launching badges for Open Data and Open Materials-- those seem like excellent steps forward.
If the board decides to explore Registered Reports in the future, I'll be happy to help out however I can.
Just to address the two concerns you raised (1) I don't think that offering RRs requires any additional resources, and (2) RRs don't *replace* traditional articles, they're just another option available to authors. Studies that aren't suitable for the RR format (e.g., modeling studies, or the [famous author] paper) can still be published the same way they always have.
In any case, thanks for the invitation, and I wish you all the very best of luck in the future.
Warm regards,
Barbara W. Sarnecka
<><><><><><><><><>
FROM: Christopher Chambers
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Hi Barbara
Well firstly, hats off for putting RRs in as a condition of editorial board membership (and thank you for standing up for them!)
I’ve been doing the same for several years now and have found it deeply illuminating.
It always worries me when a journal says it lacks the resources to offer RRs. What do they envisage it involving? Limousines and Cristal? (we’ve cut back on them, honest). Sometimes when editors respond like this it turns out they mean intellectual resources, which is doubly worrying because it means they basically don’t have the expertise to assess theory and methods without seeing results. And that is truly concerning about the capabilities of our “top” scientists.
Ironically this very same editor admonished a colleague on the very same day about that colleague’s support of the PRO initiative thusly: "Editors need to change policy because they agree with the policy change (and so by all means they should be lobbied directly and assisted in making such changes) *my emphasis*, not because they are being forced into a corner by militant reviewers.” And yet when given the opportunity to implement an important open science initiative on the same day, with the generous assistance of a willing colleague, [they] turn that offer down. Says a lot.
best, Chris
<><><><><><><><><>
FROM:[Name]
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Dear Barbara,
Thanks for your most recent message.
The additional resources I had in mind were, (1) the work required on the submission site to support the RR workflow, and (2) a dedicated Associate Editor. I agree that after RRs were set up, it might turn out to be the same amount of work overall, assuming that some proportion of the submissions we are getting now would come in as RR submissions instead. (Hopefully not too many and not too few.)
But I would still argue that [Journal] is not a pure psychology journal - we are explicitly multi-disciplinary and interdisciplinary. I think the RR format is better suited to a single-discipline, experimental psychology journal.
Best wishes,
[Name]
<><><><><><><><><>
DATE: 17 May 2018
FROM: Barbara Sarnecka
TO: [Name]
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Dear [Name],
Not to flog a dead horse, you *do* have a dedicated Associate Editor (me) and I’m willing to work with the publishers to set up the submission site, as I am doing at JNC. I do see your point about [Journal] being a special journal. But to the degree that it publishes any articles where authors use data to test hypotheses, that work would be better (more reliable, more trustworthy, more methodologically sound, less biased) if it were produced through a registered-report process than through the traditional process.
Of the 105 journals that have adopted registered reports to date (go to https://cos.io/rr/ and click on ‘Participating Journals’) only a minority are what I would call ‘pure psychology’ journals.
Nevertheless, institutional change is difficult and I’m sure you are doing everything you can, at the pace that is realistic for [Journal] right now. I applaud you for embracing Open Data and Open Materials, and I wish you all the best of luck with the journal.
Take care,
Barbara
<><><><><><><><><>
DATE: 17 May 2018
FROM: Christopher Chambers
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Great response. [Journal] is a [Publisher] journal and [Publisher] not only already offer RRs across many of their journals (including one that I edit for), but they have also created an editor’s pack for them in which the entire manuscript handling process can imported into the journal with a few mouse clicks. It couldn’t be easier.
If [they] go for this, feel free to copy me in explicitly and I’d be happy to connect you and [him/her] with the relevant people at the publisher.
As you say, there is nothing special about [Journal]. RRs don’t need to apply to all kinds of research that it is submitted to the journal to have value. I mean, consider the fact that they are offered by Royal Society Open Science, across 205 different sciences. They are offered for all of them, and those areas for which it is relevant submit them.
<><><><><><><><><>
FROM: Barbara Sarnecka
TO: [name]
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Dear [Name],
Another quick note (although you’re probably ready for this exchange to be finished) . . . [Publisher] already offers RRs at a number of its journals, and I’m told that they’ve even created an editor’s pack, in which the entire RR manuscript handling process can be imported in a few mouse clicks. So when the day comes that you are ready to implement RRs, it won’t be any trouble at all.
Best wishes,
Barbara
<><><><><><><><><>
FROM: [Name]
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Hi Barbara,
Listen, I hear what you are saying. But right now, I really have to think about filling the empty Associate Editor spots. Several of our AEs have recently quit, and but the submissions keep pouring in. I just can't think about any big changes until I have a full team of AEs again. I just don't have the attention to spare.
Best wishes,
[Name]
<><><><><><><><><>
FROM: Barbara Sarnecka
TO: [Name]
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Hi [Name],
I definitely understand. It sounds like you are really slammed with editorial work, and I’m sure you still have a lab to run at the same time.
What about if I agreed to handle 10 regular papers while we were setting up the registered reports? Would that give you a little breathing room? Chris Chambers (who is AE for registered reports at another [Publisher] journal) has offered to connect me with the right people at [Publisher] and advise on the set-up process, keeping your cognitive load to a minimum.
Or perhaps now just isn’t the right time-- I know what it feels like to be overwhelmed. Feel free to check back with me in a year or two, or whenever you feel ready to think about RRs. I can’t promise I’ll still be available, but there’s no harm in emailing.
Until then, best of luck with everything, and I hope the spring in [Editor's city] is nice. We don’t have much seasonal change here in Southern California, and I think spring is so much more joyful when it comes at the end of a long winter.
Best wishes,
Barbara
<><><><><><><><><>
FROM: [Name]
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Thanks. Yes - now is not the right time. But I'll raise it with the board at the annual meeting, and get back to you if folks are interested in doing this.
Best wishes,
[Name]
<><><><><><><><><>
FROM: Barbara Sarnecka
TO: Christopher Chambers
SUBJECT: Re: Invitation to serve as an editor for [Journal]
An interesting exercise, in the end. Change is anxiety-provoking. At least in this case, it clearly seemed to be about anxiety: [Name]'s multiple, shifting excuses for why [they] couldn't do it, culminating in, “I just can’t think about this right now!”
I sympathize. We all seem to take on more than we can handle, and then we panic about everything we’re not getting done. In that state of mind, the idea of dealing with anything new sounds intolerable. I suspect that it’s this mundane fact— the fact that most people with decision-making power feel personally overwhelmed most of the time— that slows change as much as anything else.
Anyway, thanks for all the support. You’ve been so generous with your attention and guidance, and I really appreciate it. I hope that someday I will have the chance to thank you in person.
Take care,
Barbara
<><><><><><><><><>
DATE: 19 May 2018
FROM: Christopher Chambers
TO: Barbara Sarnecka
SUBJECT: Re: Invitation to serve as an editor for [Journal]
Hi Barbara,
You’re most welcome, it’s been a pleasure to help you get going at JNC and I’m sure the journal will go on to do great things with the format!
If it’s any consolation, your interaction with [Name] is typical of dozens of interactions I’ve had with journal editors about RRs in the last 5+ years…many like to talk the talk on open science issues but when offered the opportunity to actually do something transformative they are too shy (either work shy or just feel that RRs are too radical) and the reasons for declining to offer them continually shift…one minute it’s workload, and you address that, then it’s that the format won’t suit all article types, then you address that, then it shifts to something else. I have long suspected (and even got some editors to admit off the record) that at the core of their objections is an almost instinctive discomfort with selecting what to publish before seeing the results because they judge QUALITY using results, not theory and methods and they fear that the journal will publish more negative results (which it will!). Editors like this are basically hostages to the worst incentives in the field and I’m afraid my interactions with them about RRs have made me quite cynical about their motives and expertise and (perhaps most importantly) their professional courage.
I can also virtually guarantee that you will never hear from [Name] again. As much as I would love to be proved wrong, when editors raise the idea in a half hearted way with their own boards or governing councils, the concept is universally dismissed based on the laziest, most ignorant comments. [Their] governing board will nod along and the idea will be quietly shelved because without an advocate it it just too radical — that’s of course if they raise it all which I suspect they won’t (even if they say otherwise — I wish I had a dollar for every time an editor had fobbed me off with this line, followed by me finding out from other board members that RRs were not mentioned in the meeting). This, incidentally, is why we now publicly record most of the approaches we make to journals and the outcomes (https://osf.io/3wct2/wiki/Journal%20Responses/)
Good luck at JNC and please consider me a sounding board at any time!
very best, Chris
<><><><><><><><><><><><>
So that was it-- my email exchange with the editor. I guess we'll never know why this editor was unwilling to implement Registered Reports. I guess change is scary, and people are busy, and journal editors want 'sexy' results, even if those results are less trustworthy because they weren't preregistered, could be a fluke, are probably at least a little bit HARKed and p-hacked, and so on.
As long as journals exist to make money (and that's certainly why they exist at Elsevier, Wiley, Springer, and Taylor & Francis), they're going to care about getting attention and selling subscriptions, even at the cost of some scientific integrity. If your article will get a lot of attention, the journal isn't going to look too hard or too critically at how you did the work. Registered Reports would take away the journal's ability to accept and reject papers based on their findings, which would be good for science and good for scientists, but very bad for the publishers.
Ah, the publishers . . . what a sweet deal they have. We give them their content and their quality control for free, and they walk away with 35% profits as a proportion of revenue. The president of Elsevier got some crazy pay package last year-- something like twenty million dollars? So I guess no one should be surprised when they prioritize marketing over science.
[For more chat about registered reports, see my later blog post on Talking About Registered Reports].
