Our latest survey shows that the overwhelming majority of our authors and reviewers support our journal agnostic peer-review process and consider the elimination of multiple cycles of peer review to be the most important benefit of Review Commons. They also regard the preparation of the authors’ detailed response as the rate-limiting step for the public posting of refereed preprints.
EMBO and ASAPbio launched Review Commons in December of 2019 with three very clear goals: to eliminate the wasteful cycle of multiple peer-reviews of the same manuscript by different journals; to reduce the time to the dissemination of peer-reviewed research, and to dissociate the evaluation of a manuscript’s scientific quality from that of its fit to any given journal. Review Commons also promoted a novel publication type, the ‘refereed preprint’, with a peer review process run by the professional scientific editors at EMBO Press and subsequent posting of a revised preprint, the referee reports, and the authors’ response (to learn more about refereed preprints, read our previous post).
Two and a half years into our experiment, we are happy to report that Review Commons has achieved its goals of accelerating the dissemination of peer-reviewed research and reducing (or eliminating) the wasteful cycles of sequential peer-review of the same manuscript. However, speed & efficiency are not the only goals of Review Commons. As Project Leader Thomas Lemberger and EMBO Press Head Bernd Pulverer put it when the platform was created, they also hoped that by “disentangling the review process from the context of a specific journal, Review Commons encourages reviewers to focus on the science rather than journal-specific criteria” (Lemberger & Pulverer EMBO Reports 2019). To probe how the scientific community views these changes to the peer review process, we have completed our second survey of Review Commons authors and reviewers.
Close to eleven hundred (n = 1099) Review Commons authors and reviewers took our survey; 56.6% described themselves as senior or tenured principal investigators (PIs), and an additional 17.5% were early-career PIs. One-fifth of respondents were trainees (5.64% Ph.D. students and 13.74% post-docs) and 6.55% were staff scientists (Fig 1A). Reflecting Review Commons’ origins in the cell and molecular biology community, over half of the respondents identified their main field of expertise as Cell Biology (55.64%) (Fig 1B).
Researchers appreciate the efficiency of our platform: over 90% said that the elimination of multiple cycles of peer-review was an essential (49.02%) or very important (41.95%) feature of Review Commons. Our community of authors and reviewers rated this reduced waste in the peer-review process to be the most significant benefit of Review Commons (Fig 2). Currently, 95% of refereed preprints that are submitted and accepted for publication in journals in the Review Commons consortium do not involve the recruitment of additional reviewers by journal editors.
Review Commons’ peer review process is run by dedicated managing editors and scientific editors at EMBO Press. After a single round of peer review, the editors do not, however, make any decision to accept or reject a manuscript, and neither editors nor reviewers recommend submission to any journal. We asked our authors and reviewers if Review Commons should remain “journal agnostic”. Three-quarters of them strongly agreed (40.42%) or agreed (34.40%) with this statement, and only ten percent disagreed (7.66%) or strongly disagreed (2.28%) (fig 3A). The survey data is consistent with the feedback we receive from our authors in one-on-one discussions: removing the need to assess journal fit encourages reviewers to focus on the quality of the data and analysis, the strength of the conclusions, and the significance of the work for the field.
The journal agnostic process has been one of the best features for us as submitters.
Harmit Malik, Fred Hutchinson Cancer Center
In our 2021 survey, authors also pointed out that the journal agnostic review process improved the tone of referee reports, which Review Commons authors considered more constructive than traditional journal review. Geneticist Harmit Malik, for example, told the EMBO Podcast that “the journal agnostic process has been one of the best features for us as submitters. (Reviewers) really focused on our story and what we needed to do to make it complete rather than this arbitrary bar (of journal fit, editorial note).”
Of course, authors and reviewers who have already engaged with Review Commons are a self-selected group, so we also asked a randomly selected sample of EMBO Press authors who had never interacted with Review Commons to complete our survey. As seen in Fig 3B, this group’s responses (n = 472) were overall very similar to the Review Commons cohort, with only slightly less enthusiasm for journal agnostic peer review (over 70% agree or strongly agree). Interestingly, there is significant support in both surveyed groups for giving authors the option to name journals they would be interested in submitting to (Fig 4) – a tricky proposition that would almost certainly distort the review process. Strikingly, the least popular option was asking the reviewers to suggest journals for submission. There was more support for allowing authors to inform reviewers (and editors) which scientific communities they intend to target. This could represent a good compromise and complements our existing request that reviewers declare their expertise. It may also increase the chance that the referee reports are useful for the authors and the journal editors who will handle the transferred refereed preprint.
Review Commons is committed to a transparent review process and transparent review has been a feature of EMBO Press journals for over a decade. From the outset, Review Commons has encouraged authors to post the referee reports together with their preprints while recognizing that making reviews public before an editorial decision poses potential challenges for authors. Whether authors would accept this early public exposure of expert critique of their work was an open question when the project was launched: “How will researchers respond to the idea of publicly posting their Refereed Preprint before publication?” (Lemberger & Pulverer EMBO Reports 2019).
Our survey asked what is the earliest stage that it is acceptable to post the reviews next to the preprint, assuming it is possible to appeal against an inappropriate review? Less than a fifth (18.63%) of respondents said that reviews should only be posted once the manuscript is accepted by a journal (Fig 5). The largest fraction of respondents (40.64%) said that reviews should be posted once the authors have had a chance to prepare a response (and an adventurous 9.68% believe reviews should be posted as soon as the full set of referee reports are in). Notably, the EMBO Press cohort showed a very similar distribution of responses, albeit slightly more conservative (23.03% would wait for the final journal decision, fig 3b). “We were thrilled to see such a strong signal in the survey. In our view, the author’s response is a critical step and an integral part of the peer-review process,” said Lemberger.
Authors and reviewers support the creation of an appeals process, which Review Commons is now establishing. Thomas Lemberger explains that this will differ from traditional journal appeals: “The appeals process of a journal is based on an editorial decision to accept or reject a manuscript. Review Commons does not take post-review decisions. What we are talking about now is more of a challenge procedure than an appeal, in the case of strong and substantiated concern with inappropriately worded content in a referee report, before the reviews are made public (for more information, please visit our guidelines for authors page).”
EMBO and ASAPBio launched Review Commons two and half years ago as an experiment to accelerate and improve scientific peer review. We believe the results of our surveys show that we are on the right track – though there is still a long way to go. Most authors and reviewers share our views that the scientific peer review process works best when it is divorced from journal acceptance (or rejection) decisions and that peer-reviewed research should be made public before journal publication.
These are pragmatic goals, and we believe our ongoing experiment shows that they are achievable (though significant issues of scale and sustainability remain). Lemberger and Pulverer also asked a more revolutionary question: is the refereed preprint enough? “While at some point the community may decide that the reviewed preprint will suffice, this is unlikely to happen for some time, in particular as the current research assessment system is closely tied to publication venue. If it does, we will aim to support this transition.” (The EMBO Journal 2019). Perhaps we will include the question in a future survey.
