Tap a claim on the ring to put it at the centre.
← A test meant to find the best candidates must make perfect scores…
12 connected korrents · 9 moments from 17 Apr 2012 to 3 Sept 2026.
Everything filed under benchmarks
benchmarks
Everything filed under education
education
Everything filed under software quality
software quality Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: A test meant to find the best candidates must make perfect scores vanishingly rare, so that the top of the distribution can still be told apart. A test meant to find the best candidates must make perfect scores vanishingly rare, so that the top of the distribution can still be told apart. Last stated 2 years ago
15 Apr 2025
BC
Bryan Caplan — holds since 2025-04-15 — tap for who they are
Same subject: Elite programmes stay with tests that cannot separate the best candidates because doing so is a stealthy move in a wider war against merit. — tap to centre the map on it Elite programmes stay with tests that cannot separate the best candidates because doing so is a stealthy move in a wider war against merit. Last stated 2 years ago
15 Apr 2025
BC
Bryan Caplan — holds since 2025-04-15 — tap for who they are
Same subject: Running a model five times and keeping the best answer buys a higher benchmark score without buying a better model. — tap to centre the map on it Running a model five times and keeping the best answer buys a higher benchmark score without buying a better model. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: Making test coverage a target destroys it, because high coverage numbers are easy to reach with worthless tests. — tap to centre the map on it Making test coverage a target destroys it, because high coverage numbers are easy to reach with worthless tests. Last stated 15 years ago
17 Apr 2012
MF
Martin Fowler — holds since 2012-04-17 — tap for who they are
Same subject: An AI can already earn an A on a university economics midterm, years earlier than informed forecasters expected. — tap to centre the map on it An AI can already earn an A on a university economics midterm, years earlier than informed forecasters expected. Last stated 4 years ago
21 Mar 2023
BC
Bryan Caplan — holds since 2023-03-21 — tap for who they are
Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. — tap to centre the map on it A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. Last stated 2 months ago
3 Sept 2026
DR
Dax Raad — holds since 2026-09-03 — tap for who they are
Same subject: A better school looks more like the old-fashioned schoolhouse than like project work; the more school resembles free play, the worse it gets. — tap to centre the map on it A better school looks more like the old-fashioned schoolhouse than like project work; the more school resembles free play, the worse it gets. Last stated 5 months ago
27 May 2026
SY
Scott H. Young — holds since 2026-05-27 — tap for who they are
Same subject: A dense-looking building does not by itself create a livable dense suburb without supporting schools and shops. — tap to centre the map on it A dense-looking building does not by itself create a livable dense suburb without supporting schools and shops. Last stated 8 years ago
2 Jun 2018
FS
Felix Salmon — holds since 2018-06-02 — tap for who they are
Same subject: A design course should start with history and art, grounding students in the world, rather than with tools that may be gone tomorrow. — tap to centre the map on it A design course should start with history and art, grounding students in the world, rather than with tools that may be gone tomorrow. Last stated 2 years ago
21 Feb 2025
TC
Thiago Costa — holds since 2025-02-21 — tap for who they are
Same subject: A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget. — tap to centre the map on it A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores. — tap to centre the map on it Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores. Last stated 2 years ago
13 May 2024
SR
Sebastian Ruder — holds since 2024-05-13 — tap for who they are
Same subject: Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used. — tap to centre the map on it Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
At the centre
A test meant to find the best candidates must make perfect scores vanishingly rare, so that the top of the distribution can still be told apart.
Holds Read this korrent →
Elite programmes stay with tests that cannot separate the best candidates because doing so is a stealthy move in a wider war against merit.
Holds
Running a model five times and keeping the best answer buys a higher benchmark score without buying a better model.
Holds
Making test coverage a target destroys it, because high coverage numbers are easy to reach with worthless tests.
Holds
An AI can already earn an A on a university economics midterm, years earlier than informed forecasters expected.
Holds
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Holds
A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year.
Holds
A better school looks more like the old-fashioned schoolhouse than like project work; the more school resembles free play, the worse it gets.
Holds
A dense-looking building does not by itself create a livable dense suburb without supporting schools and shops.
Holds
A design course should start with history and art, grounding students in the world, rather than with tools that may be gone tomorrow.
Holds
A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget.
Holds
Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.
Holds
Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used.
Holds