On the map

At the centre Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used. Holds NBNoam Brown Read this korrent →

• Same subject: benchmarks, scaling laws A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. Holds NBNoam Brown
• Same subject: benchmarks A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. Holds Dax Raad
• Same subject: benchmarks A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost. Holds TPThuan Pham
• Same subject: benchmarks A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages. Holds THThePrimeagen
• Same subject: benchmarks AI learning to generate good conjectures will never show up as a benchmark being knocked down; it will show up as a shift in how mathematicians talk about the tools. Holds GSGrant Sanderson
• Same subject: benchmarks Benchmarks only rise on problems somebody has already framed and scored, so saturating them does not mean senior engineers have been replaced. Holds DSDan Shipper
• Same subject: benchmarks Cash transfers are the index fund of development: the benchmark every actively managed aid programme should have to beat. Holds CBChris Blattman
• Same subject: benchmarks Every lab knows the benchmark grid is the wrong way to present a model, and publishes it anyway because everybody else does. Holds NBNoam Brown