Adam Mastroianni on Peer Review and the Academic Kitchen (EconTalk)
Adam Mastroianni, Russ Roberts · 1h 02m ·
econtalk.org ↗
38 korrents from this recording
Adam Mastroianni, Russ Roberts did not write this page.
Every claim below is a statement made in this
recording, quoted word for word and linked to the second it was said, so
you can hear it rather than take our word for it. The wording comes from
the transcript published alongside the recording; the sentence above each
quote is our reading of the claim, not their wording.
Their wordsit really isn't until the middle of the 20th century that we start centralizing and developing the system that we assume today has always existed.
Their wordsSo, all that work when someone is actually trying to use the thing turns out to be unimportant. This is done mainly for purposes of figuring out who should have high status.
Their wordsIt's a big undertaking to actually check the results of a paper, which is why it's virtually never done. Although that is, of course, maybe the single most important thing that this process could do, rather than provide some kind of aesthetic judgment.
Their wordsmost of the time papers are not rejected because they're not true. They're rejected because they're not interesting, or they're not profound, or the results are not sufficiently important.
Their wordsThese are people who are supposed to be working on the most pressing problems of humanity, and instead they're spending a lot of time sort of glancing to get papers and going, 'Eh, not interesting. This one is interesting.'
Their wordsIt's obviously true that you can get tired when you exert effort. I don't know what we could find that would make us say that, like, you don't get tired.
Their wordsBut this replication crisis--which again, I think its home base is in psychology, but it's spreading much more widely--is, it's kind of dramatic. It's not a very good system, the idea of peer review.
Their wordswhen I read that paper, my initial reaction is always: Could be. This could be true. And, if I really want to know whether it's true or not, I unfortunately have to apply additional effort to it.
Their wordsthere's a long list of books that claim to give us the latest science on various aspects of life. And, that science is always peer-reviewed journal articles, many of which are false. And yet people are reading them thinking they're true
Their wordsSo, you need to come to terms with the fact that mainly we operate in the darkness. And, I think it's actually very exciting because there's so much left to do and so much left to discover.
Their wordsMore information is better only if you can weigh it properly. Only if you can assess it properly. If you overreact to it, if you are overly confident in the information, the imperfect information you get, you make a different kind of error.
Their wordspeople say, 'Well, what's the alternative? Using your gut?' And, the answer is, 'No. It's understanding that the information you have is not always reliable.'
Their wordsI think actually that should be the modal response to scientific claims--is neglect--because most of them don't matter or we haven't figured out whether they matter or we can't use them yet or at all.
Their wordsI have very little interest in trying to fix academia because it feels impossible, and it feels like you could waste the rest of your life doing it.
Their wordsI care a lot about going to a doctor that isn't going to harm me. And so, I'm willing to give up some of the best doctors if it means I get to prevent some of the worst doctors.
Their wordswhat we want to do, I think, is actually increase the variance of the work that we do, because the bad stuff basically ends up not mattering in the long run and the good stuff changes the world.
Their wordsit wasn't always this way. And, once you realize that, then you could imagine an alternative. It's a really important, I think, tool for thinking generally--much beyond these very narrow issues.
Their wordsthere's a huge incentive to publish. And, that quote changes everything. I mean, it changes what you think about, how you think about it. It pushes you towards smaller thoughts and smaller questions because the return is more--safer. And, most people in academics are not risk-takers.
Their wordsThe marketplace of ideas, I would argue, on social media is not quite as resilient, doesn't work quite as well. The incentives aren't there.
Their wordsBut, I think the beauty of so-called science is that the experts who might vet or opine on scientific reliability have reputational things at stake that are not present when they are anonymous reviewers for obscure journal.
Their wordsBut, I think that there should be something for the people who want to swing for the fences. Or who want everything that they try to be worthwhile, whether it works or not.
Their wordsLike, what sticks in my head the most? And trust that I'm not that weird of a person. And, what's true for me is going to be true for a lot of other people as well.
Their wordsThis is why I write in regular words on my blog rather than writing scientific legalese in journals, because I think I'm better for it and they're better for it, too.
Their wordsAnd, what you're supposed to get is the freedom to say the things that you think are true. And, if you don't have that freedom, this life isn't worth it.
Their wordsI think because in order to succeed at that game, I inevitably have to say something that I think isn't true, or that I don't believe in, or write a paper in a way that I don't think is the best way to write it.
Their wordsI can do so much of a better job explaining them to people and actually representing what I did then when I have to produce what can pass peer review, which is basically a legal contract.
Their wordsWhy wouldn't you want to produce something that an eight-year-old could understand? Maybe that eight-year-old could be the next great scientist.
Their wordsas a Hayekian believer in emergent order, I think if more people felt the way you did, other institutions, norms, and other things would emerge that--from the bottom up