korrents

korrents · audio

Adam Mastroianni on Peer Review and the Academic Kitchen (EconTalk)

Adam Mastroianni, Russ Roberts · 1h 02m · econtalk.org ↗

38 korrents from this recording

1h
Adam Mastroianni, Russ Roberts did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

  1. 0:00:55 · Adam Mastroianni

    it really isn't until the middle of the 20th century that we start centralizing and developing the system that we assume today has always existed.
  2. 4 min later
  3. 0:04:38 · Adam Mastroianni

    And this is part of why I think so many people I know come to despise the things that they publish by the time that they get published.
  4. 0:04:38 · Adam Mastroianni

    So, all that work when someone is actually trying to use the thing turns out to be unimportant. This is done mainly for purposes of figuring out who should have high status.
  5. 0:04:38 · Adam Mastroianni

    It's a big undertaking to actually check the results of a paper, which is why it's virtually never done. Although that is, of course, maybe the single most important thing that this process could do, rather than provide some kind of aesthetic judgment.
  6. 0:04:38 · Russ Roberts

    most of the time papers are not rejected because they're not true. They're rejected because they're not interesting, or they're not profound, or the results are not sufficiently important.
  7. 7 min later
  8. 0:11:41 · Adam Mastroianni

    These are people who are supposed to be working on the most pressing problems of humanity, and instead they're spending a lot of time sort of glancing to get papers and going, 'Eh, not interesting. This one is interesting.'
  9. 8 min later
  10. 0:20:02 · Adam Mastroianni

    It's obviously true that you can get tired when you exert effort. I don't know what we could find that would make us say that, like, you don't get tired.
  11. 0:20:02 · Adam Mastroianni

    This is maybe unique to psychology, where we don't really have a paradigm. So, it's really hard to know what matters and what doesn't.
  12. 5 min later
  13. 0:25:31 · Russ Roberts

    But this replication crisis--which again, I think its home base is in psychology, but it's spreading much more widely--is, it's kind of dramatic. It's not a very good system, the idea of peer review.
  14. 5 min later
  15. 0:30:01 · Adam Mastroianni

    you should aspire to have the appropriate level of trust in a system given its level of performance.
  16. 0:30:01 · Adam Mastroianni

    when I read that paper, my initial reaction is always: Could be. This could be true. And, if I really want to know whether it's true or not, I unfortunately have to apply additional effort to it.
  17. 0:30:01 · Russ Roberts

    there's a long list of books that claim to give us the latest science on various aspects of life. And, that science is always peer-reviewed journal articles, many of which are false. And yet people are reading them thinking they're true
  18. 0:30:01 · Russ Roberts

    I've always found not to pass the sniff test. But it turns out it doesn't pass the replication test.
  19. 4 min later
  20. 0:34:14 · Adam Mastroianni

    And we will look ignorant very soon. And I think the thing that we'll look most ignorant for is thinking that we weren't ignorant.
  21. 0:34:14 · Adam Mastroianni

    So, you need to come to terms with the fact that mainly we operate in the darkness. And, I think it's actually very exciting because there's so much left to do and so much left to discover.
  22. 0:34:14 · Russ Roberts

    More information is better only if you can weigh it properly. Only if you can assess it properly. If you overreact to it, if you are overly confident in the information, the imperfect information you get, you make a different kind of error.
  23. 0:34:14 · Russ Roberts

    people say, 'Well, what's the alternative? Using your gut?' And, the answer is, 'No. It's understanding that the information you have is not always reliable.'
  24. 4 min later
  25. 0:38:37 · Adam Mastroianni

    I think actually that should be the modal response to scientific claims--is neglect--because most of them don't matter or we haven't figured out whether they matter or we can't use them yet or at all.
  26. 0:38:37 · Adam Mastroianni

    What I don't want is feedback at the end of a gun, where the options are, 'Change your paper or else we're not going to publish it.'
  27. 0:38:37 · Adam Mastroianni

    I just think that we should have a more diverse ecosystem where people can do research in different ways, communicate in different ways.
  28. 7 min later
  29. 0:45:47 · Adam Mastroianni

    I have very little interest in trying to fix academia because it feels impossible, and it feels like you could waste the rest of your life doing it.
  30. 0:45:47 · Adam Mastroianni

    I care a lot about going to a doctor that isn't going to harm me. And so, I'm willing to give up some of the best doctors if it means I get to prevent some of the worst doctors.
  31. 0:45:47 · Adam Mastroianni

    what we want to do, I think, is actually increase the variance of the work that we do, because the bad stuff basically ends up not mattering in the long run and the good stuff changes the world.
  32. 0:45:47 · Russ Roberts

    it wasn't always this way. And, once you realize that, then you could imagine an alternative. It's a really important, I think, tool for thinking generally--much beyond these very narrow issues.
  33. 0:45:47 · Russ Roberts

    there's a huge incentive to publish. And, that quote changes everything. I mean, it changes what you think about, how you think about it. It pushes you towards smaller thoughts and smaller questions because the return is more--safer. And, most people in academics are not risk-takers.
  34. 5 min later
  35. 0:50:51 · Adam Mastroianni

    And, it could still be, in the long run, that this is a period of time where we all tried something that didn't work.
  36. 0:50:51 · Russ Roberts

    The marketplace of ideas, I would argue, on social media is not quite as resilient, doesn't work quite as well. The incentives aren't there.
  37. 0:50:51 · Russ Roberts

    But, I think the beauty of so-called science is that the experts who might vet or opine on scientific reliability have reputational things at stake that are not present when they are anonymous reviewers for obscure journal.
  38. 3 min later
  39. 0:53:24 · Adam Mastroianni

    But, I think that there should be something for the people who want to swing for the fences. Or who want everything that they try to be worthwhile, whether it works or not.
  40. 0:53:24 · Adam Mastroianni

    Like, what sticks in my head the most? And trust that I'm not that weird of a person. And, what's true for me is going to be true for a lot of other people as well.
  41. 0:53:24 · Adam Mastroianni

    This is why I write in regular words on my blog rather than writing scientific legalese in journals, because I think I'm better for it and they're better for it, too.
  42. 0:53:24 · Russ Roberts

    Most scholars work on tiny things that are not relatively important. And, that's tragedy. That's worse than the 15,000 hours on reviewing.
  43. 8 min later
  44. 1:01:25 · Adam Mastroianni

    And, what you're supposed to get is the freedom to say the things that you think are true. And, if you don't have that freedom, this life isn't worth it.
  45. 1:01:25 · Adam Mastroianni

    I think because in order to succeed at that game, I inevitably have to say something that I think isn't true, or that I don't believe in, or write a paper in a way that I don't think is the best way to write it.
  46. 1:01:25 · Adam Mastroianni

    I can do so much of a better job explaining them to people and actually representing what I did then when I have to produce what can pass peer review, which is basically a legal contract.
  47. 1:01:25 · Adam Mastroianni

    Why wouldn't you want to produce something that an eight-year-old could understand? Maybe that eight-year-old could be the next great scientist.
  48. 1:01:25 · Russ Roberts

    as a Hayekian believer in emergent order, I think if more people felt the way you did, other institutions, norms, and other things would emerge that--from the bottom up
  49. 1:01:25 · Russ Roberts

    There are things that are different. As I said, there's way too many Ph.D.s, but maybe it'll work.