korrents
Adam Mastroianni

What Adam Mastroianni thinks about metascience

@adam-mastroianni · 60 positions · 2 changes of mind · On ppll ↗

Experimental psychologist with a PhD from Harvard, and the author of the Experimental History newsletter, where he writes about how research actually gets done and why he thinks the institutions around it are failing.

Adam Mastroianni did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

12 dated positions, 2022 to 2023, in their own words. Our reading of what Adam Mastroianni has said — not written or endorsed by them.

  1. There’s no point in picking some studies that are convenient to replicate, doing ‘em over, and reporting “only 36% of them replicate!”

    ↗Science is a strong-link problemexperimental-history.com 3rd of 3 in this piece

  2. 8 weeks earlier
  3. And this is part of why I think so many people I know come to despise the things that they publish by the time that they get published.

    ↗Adam Mastroianni on Peer Review and the Academic Kitchen (EconTalk)econtalk.org 2nd of 38 in this recording

  4. So, all that work when someone is actually trying to use the thing turns out to be unimportant. This is done mainly for purposes of figuring out who should have high status.

    ↗Adam Mastroianni on Peer Review and the Academic Kitchen (EconTalk)econtalk.org 3rd of 38 in this recording

  5. It's a big undertaking to actually check the results of a paper, which is why it's virtually never done. Although that is, of course, maybe the single most important thing that this process could do, rather than provide some kind of aesthetic judgment.

    ↗Adam Mastroianni on Peer Review and the Academic Kitchen (EconTalk)econtalk.org 4th of 38 in this recording

  6. This is maybe unique to psychology, where we don't really have a paradigm. So, it's really hard to know what matters and what doesn't.

    ↗Adam Mastroianni on Peer Review and the Academic Kitchen (EconTalk)econtalk.org 8th of 38 in this recording

  7. when I read that paper, my initial reaction is always: Could be. This could be true. And, if I really want to know whether it's true or not, I unfortunately have to apply additional effort to it.

    ↗Adam Mastroianni on Peer Review and the Academic Kitchen (EconTalk)econtalk.org 11th of 38 in this recording

  8. I just think that we should have a more diverse ecosystem where people can do research in different ways, communicate in different ways.

    ↗Adam Mastroianni on Peer Review and the Academic Kitchen (EconTalk)econtalk.org 20th of 38 in this recording

  9. 2 months earlier
  10. peer review

    All we can say from these big trends is that we have no idea whether peer review helped, it might have hurt, it cost a ton, and the current state of the scientific literature is pretty abysmal.

    ↗The rise and fall of peer reviewexperimental-history.com 1st of 9 in this piece

  11. Scientists have run studies where they deliberately add errors to papers, send them out to reviewers, and simply count how many errors the reviewers catch. Reviewers are pretty awful at this.

    ↗The rise and fall of peer reviewexperimental-history.com 2nd of 9 in this piece

  12. peer review

    In fact, we’ve got knock-down, real-world data that peer review doesn’t work: fraudulent papers get published all the time.

    ↗The rise and fall of peer reviewexperimental-history.com 3rd of 9 in this piece

  13. peer review

    I used to think about all the ways we could improve peer review. Reviewers should look at the data! Journals should make sure that papers aren’t fraudulent!

    It’s easy to imagine how things could be better—my friend Ethan and I wrote a whole paper on it—but that doesn’t mean it’s easy to make things better.

    ↗The rise and fall of peer reviewexperimental-history.com 6th of 9 in this piece

  14. 3 months earlier
  15. But the point that gets mixed in is “now we should go back and double-check every psychology study.” And that, my dear Kyle, makes no sense to me.

    ↗Psychology might be a big stinkin’ load of hogwash and that’s just fineexperimental-history.com