Guest Post by Willis Eschenbach
A Modest Proposal For Improving Peer Review
Abstract.
A proposal is made for the design of a specific type of post-publication peer review.
Background
In 2006, the Journal of the Royal Society of Medicine published a widely cited paper (960 citations) by Richard Smith entitled “Peer review: a flawed process at the heart of science and journals”. In it he noted the following problems with the peer-review system.
• There is no clear definition of “peer-review”, nor any standardization of protocols.
He described this lack of a definition as follows:
“What is clear is that the forms of peer review are protean. Probably the systems of every journal and every grant giving body are different in at least some detail; and some systems are very different. There may even be some journals using the following classic system.
The editor looks at the title of the paper and sends it to two friends whom the editor thinks know something about the subject. If both advise publication the editor sends it to the printers. If both advise against publication the editor rejects the paper. If the reviewers disagree the editor sends it to a third reviewer and does whatever he or she advises.
This pastiche—which is not far from systems I have seen used—is little better than tossing a coin, because the level of agreement between reviewers on whether a paper should be published is little better than you’d expect by chance.”
Other problems with peer review pointed out in his study are:
• It is slow and expensive
• It is inconsistent
• Reviewers often have biases
• It allows reviewers to block publication of ideas they simply disagree with, or to prevent their scientific opponents from publishing their ideas.
• It sometimes functions to inhibit innovation.
These findings by Richard Smith were borne out by a number of other studies. In particular, John Ioannidis’s study of research findings entitled “Why Most Published Research Findings Are False” [2] highlighted the size of the problem. Further issues, along with examples, were identified in work published by the BMJ. [3] And chance has been shown to play a big part in whether something is approved. [4]
As a result, it’s clear that the peer review process that we have isn’t working.
Proposed Additions
Here is how we can improve peer review.
Start out with standard double-blind peer review, where neither the authors nor the reviewers are identified. Then when the journal article is published, the journal would also publishe online a single web page containing a transcript of the complete correspondence between the authors and the reviewers. The reviewers would be identified by their names and credentials.
Below that would be the usual setup for a thread of comments and questions.
The reviewers, the authors, other scientists in the field, and the public would be invited to address the issues. It would be lightly moderated, no ad-hominems, stick to the science.
This would have a number of benefits for all participants, and would solve a number of the problems with traditional peer review:
Benefits For The Reviewers:
• It would allow their specific, detailed views and comments on the subject to be made known. In many cases the reviewers will know more about some aspect of the paper’s subject matter than the author. This would give their views on that aspect of the subject matter and on other related issues a much wider audience.
• It would allow for the publication of the minority views among the peer reviewers.
• It would give the reviewers a chance to go on record that although they recommended publication, they still disagreed with or had issues with certain parts of the study.
• It would reveal the process by which they shaped the paper for publication.
• It would place on permanent record their contributions to and their positions regarding both the paper and that area of study.
• In science, as with many occupations, all publicity is good publicity.
For The Authors
• All of the benefits for the Reviewers apply to the authors as well.
• It would tend to discourage bad behavior by reviewers. Scientists are humans, and sometimes do things for less than the finest motives. Making the process transparent and visible to all will help encourage reviewers to act out of scientific motives.
For Both Authors And Reviewers
• It would encourage both sides to be clear, collegiate, and dispassionate in their interactions with each other during the review process, knowing that their words would eventually be published.
• It would provide a public forum where the discussion that went on during the review could be continued, with both reviewers and authors having a larger time and place to explain and defend their ideas.
For Other Scientists In The Field
• It would provide much deeper insights into both the areas of agreement and the areas of disagreement between the reviewers, the authors, and other scientists studying the subject in question.
• It would allow other scientists to see just exactly why the paper was recommended for publication.
• It would provide other scientists a forum to ask questions of the reviewers, the authors, or other scientists commenting on the subject matter.
• It would offer other scientists a forum where they might be instrumental in resolving any unresolved questions or possible misunderstandings between the authors and reviewers, by contributing their own insights, theories, questions, and understandings.
For The Public
• It would give the general public a look “behind the scenes” at what goes into the process of peer review, by both authors and reviewers.
• As with the scientists, it would give the general public the chance to improve the scientific process.
• Science lives or dies by transparency. At present, the process is very opaque to the general public.
For The Journals
• Inviting new subscribers. People will be more willing to subscribe to a journal that has an understandable process behind what they are reading.
• Access to volunteer reviewers. Scientists are just like anyone else, looking for recognition. This will give them a chance to get their ideas published and to defend their ideas under their own names. And in turn, for many, this will make them more likely to volunteer their time.
• It will give all scientists confidence that your peer review process is solid, and thus your journal is reputable.
For Science In General
We desperately need a substitute for peer review. While this proposal may need fine-tuning, it goes a long way towards solving the problems with the current peer-review process.
• It makes scientific findings and the scientific process accessible and understandable, thus improving the acceptance of scientific findings.
• It offers, on a paper by paper basis, the opportunity for scientific ideas to be tested, updated, refined, or falsified. Over time, the ReviewPlus page for any given paper will provide a record of just where those ideas fit in the current scientific understanding.
• It would provide a place where even sometime later, as new understandings of the subject come to light, they can be made publicly available to either support or falsify the conclusions of the paper. This would have the effect of keeping the status of the various ideas in the paper up to date. Have the claims been falsified over time, or are they still valid?
• It would provide a place where scientists who disagree with all or some part of the study can have their objections and counter-arguments placed on permanent record.
• It would give all of us a clear idea of the complexities, areas of agreement and disagreement, support for and arguments against, and a much more comprehensive view of the state of current science regarding that study.
Implementation
For clarity, let me call the single page where the discussion occurs the “ReviewPlus” page for that specific study. For this to work, the ReviewPlus discussion web page needs to be accessible, interesting, and most of all, scientific. So, on the ReviewPlus page:
• Identify the authors’ and reviewers’ comments with some special symbols like “A:” and “R:“, with no special symbols for everyone else.
• Allow several levels of indenting so conversations can diverge to discuss details.
• Moderate all the comments.
• No anonymous posting. Make people take responsibility for their words. Ensure that they have a valid email address.
• Have few but clear site rules, and enforce them, eg:
No personal attacks
Stick to the scientific subject of this study
Keep it polite.
Something like that. And then enforce them strongly.
• Don’t censor comments in private. Replace them with something that says “[SNIPPED: Please stick to the scientific subject.]” or “[SNIPPED: No personal attacks.]” in bold. This lets other folks know what’s going on. In addition, you can snip just that part of an otherwise interesting comment that breaks site rules, and leave the rest.
• Ban repeat offenders, but again not in private. Simply replace their comment with something that says [BANNED: Regrettably, your repeated transgressions of site rules have earned you a ban. Please email the editors if you wish to be reinstated.]
ReviewPlus Page Proposed Layout
At the top of each individual ReviewPlus page would be what is sometimes called the “money graphic”, the one graphic from the study that best exemplifies and embodies the ideas.
Below that money graphic we’d have:
The title of the study.
The names of the authors
The names of the reviewers, with their relevant credentials
Then, the “lead-in”, viz:
This page hosts a public discussion of the ideas presented in the above study. The discussion involves some or all of the authors, the reviewers, other scientists in the field, and interested lay people. It is a place to strive for understandings, to request clarification of unclear parts of the study, to provide additional links and information that tends to either support or falsify the study, and to ask questions of the authors, the reviewers, and other participants.
That example above is just a very rough first cut at what is a critical part of the page. For newcomers it sets the tone and describes the direction and requested behavior. In some ways it’s the most important part of the page. It should be brief, clear, and interesting.
Below the lead-in would be the Abstract of the Study.
Below that would be a standard set of links to:
The Study (whether paywalled or not)
Supplementary Online Information
Data As Used
Code As Used
Next would come the Review Transcript. The Review Transcript would be the complete and exact record, warts and all, in chronological order, of the questions, the to and fro, the discussions, and the changes and exchanges between the reviewers and the authors during the review process.
Then, after the Review Transcript and above the “Comments” text box, would be the ask, e.g.
We invite you to contribute to the ongoing scientific discussion of the issues arising from this study. If you choose to do so, please follow these few simple site rules. Comments that are over the line will be snipped to maintain decorum.
• To avoid misunderstandings, please quote exactly or link to what you are discussing. This avoids endless misunderstandings.
• No personal attacks. We repeat. No personal attacks. We repeat.
• Stick to the scientific subject of this study.
• Please keep it polite, friendly, and collegiate. We are not adversaries. We are working together towards greater and clearer scientific understanding.
The ReviewPlus page will also need a checkbox or other way to subscribe to further comments on that particular study.
Rejected Papers
Each journal should publish papers that have been through the full peer-review process but that have been rejected, in electronic form only, and allow free public access to them. And each rejected paper should have its accompanying ReviewPlus page.
In a way, this is more important than publishing the accepted papers. Science proceeds by falsification. But we have hidden away the most important falsification in the entire process, the falsification done by the reviewers.
These provisionally falsified claims are very important. If the reviewers’ rejections hold up, it will provide the ideas and logic needed to assess future repetitions of the same claim. If an eminent statistician peer-reviewing my work has convincingly refuted my argument, that should be in the public record.
Then the next time the argument comes up, someone could just say nope, someone tried that, here’s a link t why it doesn’t work.
It would also encourage people to be reviewers, since their eminently scientific work of falsification would not be hidden away forever … and where’s the fun in that?
Discussion
Let me emphasize that Peer Review Plus is not something different from traditional peer review.
It is something in addition to traditional peer review … that’s the “Plus” part.
Peer review will go on in exactly the same way as before, unchanged … the only difference is that at the end of the usual process the complete correspondence between the authors and reviewers, their ideas, objections, compromises, clarifications, and all, will be published on the given study’s ReviewPlus page and people will be invited to contribute.
Peer Review Plus is a bolt-on addition to traditional peer review, not a replacement. And daring to dream big, the fact that it is a bolt-on to peer review, and not a replacement for peer review, will make the global transition from peer review to Peer Review Plus much simpler. It will be able to take place gradually, one journal at a time, without any disruption, changing, or weakening of the current system.
=============
Looking to the future, there should be a standard template for the ReviewPlus web pages, such that any journal can simply download the template, fill in the blanks and the links, and they’re in business. This is worth keeping in mind as the first pages are designed. This standardization is important for several reasons.
First, it will make it easy for any journal to opt in to adding Peer Review Plus to their traditional peer review.
Second, it will let the reader know where to find specific info on any journal’s ReviewPlus page. People should find the format familiar even if they’ve never been to the particular journal’s website.
And third, it will help with searching. If all the fields (authors names, review transcript, links to code, abstract, etc) are the same and in the same order, it will be possible for someone like Google Scholar to search the global set of ReviewPlus pages curated by all journals by field, or title.
References
[1] Smith, Richard. “Peer review: a flawed process at the heart of science and journals.” Journal of the Royal Society of Medicine vol. 99,4 (2006): 178-82. doi:10.1258/jrsm.99.4.178
[2] Ioannidis JPA (2005) “Why Most Published Research Findings Are False.” PLoS Med 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124
[3] Henderson, Mark. “Problems with peer review.” BMJ 340 (2010).
[4] Cole, Stephen, Jonathan R. Cole, and Gary A. Simon. “Chance and consensus in peer review.” Science 214, no. 4523 (1981): 881-886.
Related
