← People and their mental models
3 claims Scott Alexander made fit 3 mental models. Most often: Emergence, Probabilistic thinking, When a measure becomes a target. Everything they said here.
Think in odds, not certainties, and bet heavily only when the odds are clearly yours.
Their wordsThis wasn't exactly a mistake, since it's not wrong to give 60-40 odds on a false thing, but it seems worth acknowledging here.
Whole systems do things none of their parts do, and nobody has to design that order for it to appear.
Their wordsLarge language models are “grown, not built”. Researchers run training data through a neural network. Eventually this creates a working AI; nobody really knows how.
Reward a number and people, or machines, will find ways to move the number without the thing it measured (Goodhart's law).
Their wordsWe’re positively reinforcing AI for success on benchmarks, including impossible benchmarks , then negatively reinforcing it for getting caught cheating.