korrents · piece

The rise and fall of peer review

Adam Mastroianni · 13 Dec 2022

Others here said this too

Peer review is one of the things genuinely wrong with academia, not a safeguard that mostly works.

Their wordsPeer review doesn’t work and there’s probably no way to fix it.

Also said by Alexey Guzey

Read the piece on experimental-history.com ↗

8 korrents from this piece ↓

Adam Mastroianni did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

Our reading
Sixty years of peer review produced no evidence that it made science better, and the literature it produced is in a bad state.

Their wordsAll we can say from these big trends is that we have no idea whether peer review helped, it might have hurt, it cost a ton, and the current state of the scientific literature is pretty abysmal.
Our reading
Peer reviewers miss most of the serious errors in the papers they review.

Their wordsScientists have run studies where they deliberately add errors to papers, send them out to reviewers, and simply count how many errors the reviewers catch. Reviewers are pretty awful at this.
Our reading
Fraudulent papers pass peer review routinely, which is direct evidence that peer review does not do the job it is supposed to do.

Their wordsIn fact, we’ve got knock-down, real-world data that peer review doesn’t work: fraudulent papers get published all the time.
Our reading
The demands of peer reviewers push scientists into producing bad research.

Their wordsThe invention of peer review may have even encouraged bad research.
Our reading
Scientists say they take peer review seriously, but what they actually do shows they do not think it matters.

Their wordsIf you look at what scientists actually do, it’s clear they don’t think peer review really matters.
Our reading
Needing a reviewer's approval before an idea counts for anything makes scientists worse at thinking, and costs the field its most unusual ideas.

Their wordsMaking peer review harsher would also exacerbate the worst problem of all: just knowing that your ideas won’t count for anything unless peer reviewers like them makes you worse at thinking.
Our reading
Peer review is one of the things genuinely wrong with academia, not a safeguard that mostly works.

Their wordsPeer review doesn’t work and there’s probably no way to fix it.
Our reading
A vetting system that catches almost nothing is worse than no vetting at all, because its stamp makes people trust what nobody checked.
Mental modelSecond-order thinking

Their wordsThis extremely bad system is worse than nothing because it fools people into thinking they’re safe when they’re not.

Keep reading

The other pieces are the ones whose claims are worded most like this one's. That measures language, not agreement: they may argue the opposite.