How science works
3 papers · 2004–2018 · oldest first
Start here
Research about research: whether findings hold up, and what measuring people does to them.
Ours, not the authors'.
Ways of thinking this field uses
Mental models that keep turning up in these arguments. Our pick; each page holds the claims behind it, in the words of whoever said them.
- When a measure becomes a target Reward a number and people, or machines, will find ways to move the number without the thing it measured (Goodhart's law).
- Incentives Look at what people are rewarded for; it explains more behaviour than what they say, your own included.
- Cognitive biases Your mind jumps to conclusions and defends them; look for the evidence against you and use checklists.
- Probabilistic thinking Think in odds, not certainties, and bet heavily only when the odds are clearly yours.
- The map is not the territory Every model, word or plan is a simplified map of reality; useful when its structure matches, misleading when mistaken for the thing itself.
The shape of the field
-
2004
The legacy of Bristol: public disclosure of individual surgeons' results (with 5 co-authors)
David Spiegelhalter · in the archive · 5 korrents
After Bristol, heart surgeons set out to publish each one's raw death rates inside very wide safety bands to show they are safe, not to help patients pick a surgeon.
-
2005
Ioannidis, "Why Most Published Research Findings Are False" (PLoS Medicine)
John Ioannidis · in the archive · 6 korrents
Most published research findings are false because studies are often too small, effects too weak, methods too flexible, and too many teams race for positive results.
-
2018
Daniel Schacter · in the archive · 2 korrents
3 of these 3 papers are in this archive, with quotes you can check — those are the linked ones. The rest are here so the timeline is the field's shape rather than ours: they carry no quote, no link and no claim beyond their title, authors and year, and the one-line notes are ours, about what each paper did rather than what anyone believes.
What you can read here
-
1 The legacy of Bristol: public disclosure of individual surgeons' results (with 5 co-authors)
David Spiegelhalter · 19 Aug 2004 · 5 korrents
After Bristol, heart surgeons set out to publish each one's raw death rates inside very wide safety bands to show they are safe, not to help patients pick a surgeon. This will be the first UK surgical specialty to review named individuals publicly, opening the door for other branches of medicine and debate on surgeon versus whole-hospital responsibility. It argues a position and proposes a method using three years of crude death rates against 99.99 percent limits, building on Bristol recommendations while resisting US-style adjusted comparisons.
-
2 Ioannidis, "Why Most Published Research Findings Are False" (PLoS Medicine)
John Ioannidis · 30 Aug 2005 · 6 korrents
Most published research findings are false because studies are often too small, effects too weak, methods too flexible, and too many teams race for positive results. It shows why so many claims fail later and why findings can simply track a field’s bias instead of real effects. An opinion essay models the odds a claim is true and argues against treating one significant study as proof.
-
Daniel Schacter · 5 Dec 2018 · 2 korrents
Where to go next