Tap a claim on the ring to put it at the centre.
← Running a model five times and keeping the best answer buys a higher benchmark score without buying a better model.
17 connected korrents · 19 moments from 11 Oct 2018 to 27 Sept 2026. Nearly all of them are about benchmarks .
Everything filed under scaling laws
scaling laws
Everything filed under LLMs
LLMs
Everything filed under Anthropic
Anthropic
Everything filed under inflation
inflation
Everything filed under compilers
compilers
Everything filed under AI alignment
AI alignment
Everything filed under mathematics
mathematics
Everything filed under OpenAI
OpenAI
Everything filed under AI and jobs
AI and jobs Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: Running a model five times and keeping the best answer buys a higher benchmark score without buying a better model. Running a model five times and keeping the best answer buys a higher benchmark score without buying a better model. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. — tap to centre the map on it A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. Last stated 2 months ago
3 Sept 2026
DR
Dax Raad — holds since 2026-09-03 — tap for who they are
Same subject: A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost. — tap to centre the map on it A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost. Last stated 7 months ago
1 Apr 2026
TP
Thuan Pham — holds since 2026-04-01 — tap for who they are
Same subject: A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages. — tap to centre the map on it A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages. Last stated 2 years ago
22 Mar 2025
TH
ThePrimeagen — holds since 2025-03-22 — tap for who they are
Same subject: AI is positively reinforced for success on benchmarks, including impossible ones, and then negatively reinforced for getting caught cheating. — tap to centre the map on it AI is positively reinforced for success on benchmarks, including impossible ones, and then negatively reinforced for getting caught cheating. Last stated 2 months ago
1 Sept 2026
SA
Scott Alexander — holds since 2026-09-01 — tap for who they are
Same subject: AI learning to generate good conjectures will never show up as a benchmark being knocked down; it will show up as a shift in how mathematicians talk about the tools. — tap to centre the map on it AI learning to generate good conjectures will never show up as a benchmark being knocked down; it will show up as a shift in how mathematicians talk about the tools. Last stated 4 months ago
30 Jun 2026
GS
Grant Sanderson — holds since 2026-06-30 — tap for who they are
Same subject: AI research at OpenAI is an order of magnitude or more harder than metrics, benchmarks, or forecasts suggest. — tap to centre the map on it AI research at OpenAI is an order of magnitude or more harder than metrics, benchmarks, or forecasts suggest. Last stated 4 weeks ago
27 Sept 2026
RN
Ramez Naam — holds since 2026-09-27 — tap for who they are
Same subject: Benchmarks only rise on problems somebody has already framed and scored, so saturating them does not mean senior engineers have been replaced. — tap to centre the map on it Benchmarks only rise on problems somebody has already framed and scored, so saturating them does not mean senior engineers have been replaced. Last stated 5 months ago
24 May 2026
DS
Dan Shipper — holds since 2026-05-24 — tap for who they are
Same subject: AI progress is faster than people expect, and ordinary scaling can be enough to solve problems that looked very hard. — tap to centre the map on it AI progress is faster than people expect, and ordinary scaling can be enough to solve problems that looked very hard. Last stated 2 years ago
2 Feb 2025
AC
Ajeya Cotra — holds since 2023-08-29 — tap for who they are
SA
Scott Alexander — holds since 2022-09-12 — tap for who they are
MB
Miles Brundage — holds since 2025-02-02 — tap for who they are
Same subject: As models keep growing, AI is running out of enough high-quality unique training tokens to keep up. — tap to centre the map on it As models keep growing, AI is running out of enough high-quality unique training tokens to keep up. Last stated 4 months ago
24 Jun 2026
LW
Lilian Weng — holds since 2026-06-24 — tap for who they are
Same subject: Current techniques restrict pre-trained representation power because standard language models are unidirectional. — tap to centre the map on it Current techniques restrict pre-trained representation power because standard language models are unidirectional. Last stated 8 years ago
11 Oct 2018
KT
Kristina Toutanova — holds since 2018-10-11 — tap for who they are
MC
Ming-Wei Chang — holds since 2018-10-11 — tap for who they are
KL
Kenton Lee — holds since 2018-10-11 — tap for who they are
JD
Jacob Devlin — holds since 2018-10-11 — tap for who they are
Same subject: Emergent abilities come from generalisation turning compositional, not merely from a training set containing a lot of stuff. — tap to centre the map on it Emergent abilities come from generalisation turning compositional, not merely from a training set containing a lot of stuff. Last stated a year ago
12 Sept 2025
SL
Sergey Levine — holds since 2025-09-12 — tap for who they are
Same subject: If reasoning can be separated from facts, the frontier labs lose their advantage, because nobody needs a million-GPU cluster any more. — tap to centre the map on it If reasoning can be separated from facts, the frontier labs lose their advantage, because nobody needs a million-GPU cluster any more. Last stated 2 years ago
19 Jun 2024
AS
Aravind Srinivas — holds since 2024-06-19 — tap for who they are
Same subject: Large language models could still plateau, and that possibility should be held open even though no evidence of it has appeared. — tap to centre the map on it Large language models could still plateau, and that possibility should be held open even though no evidence of it has appeared. Last stated 2 months ago
26 Aug 2026
DH
David Heinemeier Hansson — holds since 2026-08-26 — tap for who they are
Same subject: A tiny language model inventing a plausible-sounding name is the same phenomenon as a large one confidently stating a false fact. — tap to centre the map on it A tiny language model inventing a plausible-sounding name is the same phenomenon as a large one confidently stating a false fact. Last stated 8 months ago
12 Feb 2026
AK
Andrej Karpathy — holds since 2026-02-12 — tap for who they are
Same subject: AI should be released iteratively rather than built in secret, because AI and surprise do not go together and the world needs time to adapt before it is under the gun. — tap to centre the map on it AI should be released iteratively rather than built in secret, because AI and surprise do not go together and the world needs time to adapt before it is under the gun. Last stated 3 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: Bing's Sydney, not ChatGPT, was the first truly fantastic chatbot. — tap to centre the map on it Bing's Sydney, not ChatGPT, was the first truly fantastic chatbot. Last stated a month ago
15 Sept 2026
TL
Tobias Lütke — holds since 2026-09-15 — tap for who they are
At the centre
Running a model five times and keeping the best answer buys a higher benchmark score without buying a better model.
Holds Read this korrent →
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Holds
A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year.
Holds
A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost.
Holds
A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages.
Holds
AI is positively reinforced for success on benchmarks, including impossible ones, and then negatively reinforced for getting caught cheating.
Holds
AI learning to generate good conjectures will never show up as a benchmark being knocked down; it will show up as a shift in how mathematicians talk about the tools.
Holds
AI research at OpenAI is an order of magnitude or more harder than metrics, benchmarks, or forecasts suggest.
Holds
Benchmarks only rise on problems somebody has already framed and scored, so saturating them does not mean senior engineers have been replaced.
Holds
AI progress is faster than people expect, and ordinary scaling can be enough to solve problems that looked very hard.
Holds
As models keep growing, AI is running out of enough high-quality unique training tokens to keep up.
Holds
Current techniques restrict pre-trained representation power because standard language models are unidirectional.
Holds
Emergent abilities come from generalisation turning compositional, not merely from a training set containing a lot of stuff.
Holds
If reasoning can be separated from facts, the frontier labs lose their advantage, because nobody needs a million-GPU cluster any more.
Holds
Large language models could still plateau, and that possibility should be held open even though no evidence of it has appeared.
Holds
A tiny language model inventing a plausible-sounding name is the same phenomenon as a large one confidently stating a false fact.
Holds
AI should be released iteratively rather than built in secret, because AI and surprise do not go together and the world needs time to adapt before it is under the gun.
Holds
Bing's Sydney, not ChatGPT, was the first truly fantastic chatbot.
Holds