korrents

peer review

How science checks itself, and the case that the checking mostly does not work.

FilterEveryone, all time22 from 11 people
Group All Technologists 1Scientists 3Economists & investors 1Writers & journalists 2
Who Everyone AMAdam Mastroianni 7 JRJosé Luis Ricón 3 AHAnton Howes 2 SRStuart Ritchie 2 TTTerence Tao 2 AGAlexey Guzey 1 CBChris Blattman 1 DKDaniel Kahneman 1 NFNat Friedman 1 RKRohit Krishnan 1 TCTyler Cowen 1
When All time Past 1 yearPast 3 yearsPast 5 years
  1. TC

    Tyler Cowen quoted

    Economist at George Mason University

    Our readingFear of strong AI models killing everyone was not based on peer-reviewed research showing high doom odds, nor indicated by market prices.

    Their wordsMany people panicked about the possibility of strong AI models killing us all. That fear was not based on peer-reviewed scientific research showing a high chance of doom, nor was doom indicated in any market prices of the time, including measures of risk.

    ↗A doomsday scenario for American AImarginalrevolution.com

  2. 3 weeks earlier
  3. NF

    Nat Friedman quoted

    Investor and engineer; CEO of GitHub from 2018 to 2021

    Our readingThe replication crisis is not an aberration; many of the things we believe are wrong

    metascience

    Their wordsWe know less than we think The replication crisis is not an aberration Many of the things we believe are wrong We are often not even asking the right questions

    ↗Nat Friedmannat.org

  4. 3 weeks earlier
  5. TT

    Terence Tao quoted

    Our readingAutomatic filters cannot replace community acceptance, and human referees cannot be removed from mathematical publication.

    Their wordsBut passing an automatic filter is not a substitute for community acceptance; I do not believe that human referees can be removed from the publication process.

    ↗Mathematics in the age of AIarxiv.org All 5 from this paper

  6. 2 months earlier
  7. RK

    Rohit Krishnan quoted

    Writes Strange Loop Canon

    Our readingCombining multiple AI models' outputs into one final answer tends to discard most of the good ideas that only one model produced.

    Their wordsthe council does not simply keep the best bits from everyone. It keeps a minority of the good ideas, while peer review seems to give consensus ideas an extra push.

    ↗LLM councils show groupthinkstrangeloopcanon.com 2nd of 3 in this piece

  8. 3 months earlier
  9. TT

    Terence Tao quoted

    Our readingThe structures of science now have to change, because the scarce thing is no longer proposing theories but verifying and evaluating the thousands anyone can now propose.

    Their wordsSo, we're now in a situation where suddenly people can generate thousands of theories for a given scientific problem. And now we have to to verify them, evaluate them and this is something which we we have to to change our structures of science to actually sort this out.

    ↗Terence Tao – How the world’s top mathematician uses AIyoutube.com 2nd of 24 in this recording

  10. 5 months earlier
  11. AH

    Anton Howes quoted

    Historian of innovation

    Our readingEconomics journals have made large datasets and heavy statistics the price of making any causal claim at all, which has squeezed qualitative economic history out of the field.

    Their wordsOver the past few decades, journal editors and peer-reviewers have increasingly insisted that papers must present large datasets that have been treated using complex statistical methods in order to make even the mildest claims about what caused what.

    ↗Age of Invention: Joel Mokyr's Nobelageofinvention.xyz

  12. 2 years earlier
  13. AH

    Anton Howes quoted

    Our readingHistory has a reproducibility problem worse than science’s replication crisis, because hardly anyone ever goes back to the sources to check what a historian claimed they say.

    metascience

    Their wordsBut I’ve become increasingly worried that science’s replication crises might pale in comparison to what happens all the time in history, which is not just a replication crisis but a reproducibility crisis.

    ↗Age of Invention: Does History have a Replication Crisis?ageofinvention.xyz 1st of 4 in this piece

  14. 8 months earlier
  15. SR

    Stuart Ritchie quoted

    Psychologist, formerly a lecturer at King's College London

    Our readingThe replication crisis is not an aberration; many of the things we believe are wrong

    metascience

    Their wordsWe know how easily these practices can distort the scientific record and give us false-positive, nonsense results.

    ↗Science isn't storytellingsciencefictions.org 2nd of 5 in this piece

  16. 2 weeks earlier
  17. AM

    Adam Mastroianni quoted

    Our readingSixty years of peer review produced no evidence that it made science better, and the literature it produced is in a bad state.

    Their wordsAll we can say from these big trends is that we have no idea whether peer review helped, it might have hurt, it cost a ton, and the current state of the scientific literature is pretty abysmal.

    ↗The rise and fall of peer reviewexperimental-history.com 1st of 8 in this piece

  19. AM

    Adam Mastroianni quoted

    Our readingPeer reviewers miss most of the serious errors in the papers they review.

    Their wordsScientists have run studies where they deliberately add errors to papers, send them out to reviewers, and simply count how many errors the reviewers catch. Reviewers are pretty awful at this.

    ↗The rise and fall of peer reviewexperimental-history.com 2nd of 8 in this piece

  21. AM

    Adam Mastroianni quoted

    Our readingFraudulent papers pass peer review routinely, which is direct evidence that peer review does not do the job it is supposed to do.

    Their wordsIn fact, we’ve got knock-down, real-world data that peer review doesn’t work: fraudulent papers get published all the time.

    ↗The rise and fall of peer reviewexperimental-history.com 3rd of 8 in this piece

  23. AM

    Adam Mastroianni quoted

    Our readingThe demands of peer reviewers push scientists into producing bad research.

    Their wordsThe invention of peer review may have even encouraged bad research.

    ↗The rise and fall of peer reviewexperimental-history.com 4th of 8 in this piece

  25. AM

    Adam Mastroianni quoted

    Our readingScientists say they take peer review seriously, but what they actually do shows they do not think it matters.

    Their wordsIf you look at what scientists actually do, it’s clear they don’t think peer review really matters.

    ↗The rise and fall of peer reviewexperimental-history.com 5th of 8 in this piece

  27. AM

    Adam Mastroianni quoted

    Our readingNeeding a reviewer's approval before an idea counts for anything makes scientists worse at thinking, and costs the field its most unusual ideas.

    Their wordsMaking peer review harsher would also exacerbate the worst problem of all: just knowing that your ideas won’t count for anything unless peer reviewers like them makes you worse at thinking.

    ↗The rise and fall of peer reviewexperimental-history.com 6th of 8 in this piece

  29. AM

    Adam Mastroianni quoted

    Our readingPeer review is one of the things genuinely wrong with academia, not a safeguard that mostly works.

    Their wordsPeer review doesn’t work and there’s probably no way to fix it.

    ↗The rise and fall of peer reviewexperimental-history.com 7th of 8 in this piece

  30. 2 months earlier
  31. SR

    Stuart Ritchie quoted

    Our readingPublished effect sizes shrink as more studies are done, and that decline is what the replication crisis is made of.

    metascience

    Their wordsThe decline effect is the idea that scientific findings get smaller over time. That is, initial findings tend to report effect sizes that then crumble away—sometimes to nothing—with subsequent research.

    ↗How growth mindset shranksciencefictions.org 2nd of 3 in this piece

  32. 21 months earlier
  33. JR

    José Luis Ricón quoted

    Engineer and writer at Nintil

    Our readingPre-grant peer review predicts which research will have impact substantially better than chance does.

    Their wordsI have reviewed before the effectiveness of peer review at figuring out "what's good" in the context of grant awards, finding that peer review as currently practiced does substantially better than chance at predicting future impact, especially for the most impactful of the papers; but at the same time is is far from perfect, leaving plenty of variance unexplained.

    ↗Fund people, not projects IV: Scientific egalitarianism and lotteriesnintil.com 1st of 3 in this piece

  35. JR

    José Luis Ricón quoted

    Our readingReviewers can tell the obviously good from the obviously bad, but there is a messy middle where they cannot tell at all.

    Their wordsThe way I see this is that reviewers can identify what's obviously not great and what is obviously great (Perhaps identifying the latter is easier than the former). But there is a "messy middle" where it is unclear what to fund.

    ↗Fund people, not projects IV: Scientific egalitarianism and lotteriesnintil.com 2nd of 3 in this piece

  37. JR

    José Luis Ricón quoted

    Our readingThe cost of grant peer review is dominated by the time researchers spend writing applications, not by the reviewing itself.

    Their wordsThe cost of peer review is largely dominated by applying for grants, not peer reviewing.

    ↗Fund people, not projects IV: Scientific egalitarianism and lotteriesnintil.com 3rd of 3 in this piece

  38. 17 months earlier
  39. AG

    Alexey Guzey quoted

    Writer and founder of New Science

    Our readingPeer review is one of the things genuinely wrong with academia, not a safeguard that mostly works.

    Their wordsSpecific parts of academia that seem to be problematic: rigid, punishing for deviation, career progression; peer review; need to constantly fundraise for professors.

    ↗How Life Sciences Actually Work: Findings of a Year-Long Investigationguzey.com 3rd of 4 in this piece

  40. 5 years earlier
  41. CB

    Chris Blattman quoted

    Economist and political scientist at the University of Chicago

    Our readingComparing randomised trials of cash against NGOs’ own claims for their programmes is not a fair comparison.

    Their wordsThe evidence they cite in favor of cash points to peer reviewed randomized trials. The evidence on better performing programs point to… NGO home pages.

    ↗Are cash transfers overrated?chrisblattman.com 5th of 5 in this piece

  43. DK

    Daniel Kahneman quoted

    Psychologist and Nobel laureate (1934 to 2024)

    Our readingSocial science disciplines use ritual tests of competence that screen work by form rather than substance

    Their wordsEvery discipline of social science, I believe, has some ritual tests of competence, which must be passed before a piece of work is considered worthy of attention. Such tests are necessary to prevent information overload, and they are also important aspects of the tribal life of the disciplines.

    ↗Daniel Kahneman – Biographical, Part 2: Eulogy for Amos Tversky (June 5, 1996)nobelprize.org 3rd of 6 in this piece