Tap a claim on the ring to put it at the centre.
← Average performance on a benchmark suite is very different from the upper bound of what an AI agent can do.
17 connected korrents · 18 moments from 3 Oct 2012 to 27 Sept 2026.
Everything filed under benchmarks
benchmarks
Everything filed under measuring intelligence
measuring intelligence
Everything filed under scaling laws
scaling laws
Everything filed under ARC-AGI
ARC-AGI Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: Average performance on a benchmark suite is very different from the upper bound of what an AI agent can do. Average performance on a benchmark suite is very different from the upper bound of what an AI agent can do. Last stated 6 months ago
16 Apr 2026
SK
Sayash Kapoor — holds since 2026-04-16 — tap for who they are
Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. — tap to centre the map on it A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. Last stated 2 months ago
3 Sept 2026
DR
Dax Raad — holds since 2026-09-03 — tap for who they are
Same subject: A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost. — tap to centre the map on it A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost. Last stated 7 months ago
1 Apr 2026
TP
Thuan Pham — holds since 2026-04-01 — tap for who they are
Same subject: A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages. — tap to centre the map on it A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages. Last stated 2 years ago
22 Mar 2025
TH
ThePrimeagen — holds since 2025-03-22 — tap for who they are
Same subject: AI is positively reinforced for success on benchmarks, including impossible ones, and then negatively reinforced for getting caught cheating. — tap to centre the map on it AI is positively reinforced for success on benchmarks, including impossible ones, and then negatively reinforced for getting caught cheating. Last stated 2 months ago
1 Sept 2026
SA
Scott Alexander — holds since 2026-09-01 — tap for who they are
Same subject: AI learning to generate good conjectures will never show up as a benchmark being knocked down; it will show up as a shift in how mathematicians talk about the tools. — tap to centre the map on it AI learning to generate good conjectures will never show up as a benchmark being knocked down; it will show up as a shift in how mathematicians talk about the tools. Last stated 4 months ago
30 Jun 2026
GS
Grant Sanderson — holds since 2026-06-30 — tap for who they are
Same subject: AI research at OpenAI is an order of magnitude or more harder than metrics, benchmarks, or forecasts suggest. — tap to centre the map on it AI research at OpenAI is an order of magnitude or more harder than metrics, benchmarks, or forecasts suggest. Last stated 4 weeks ago
27 Sept 2026
RN
Ramez Naam — holds since 2026-09-27 — tap for who they are
Same subject: An oil embargo without a price cap would reduce global oil supply and push up benchmark prices like Brent. — tap to centre the map on it An oil embargo without a price cap would reduce global oil supply and push up benchmark prices like Brent. Last stated 4 years ago
11 Oct 2022
SJ
Simon Johnson — holds since 2022-10-11 — tap for who they are
Same subject: "Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next. — tap to centre the map on it "Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next. Last stated 11 months ago
25 Nov 2025
IS
Ilya Sutskever — holds since 2025-11-25 — tap for who they are
Same subject: A lab should spend most of its compute on research rather than on building the next model, because research is where the tenfold yearly efficiency gains come from. — tap to centre the map on it A lab should spend most of its compute on research rather than on building the next model, because research is where the tenfold yearly efficiency gains come from. Last stated 7 months ago
13 Mar 2026
DP
Dylan Patel — holds since 2026-03-13 — tap for who they are
Same subject: After pre-training, post-training and test-time scaling, the fourth scaling law is agentic: multiplying AI by spawning agents, and the whole loop scales on one thing, compute. — tap to centre the map on it After pre-training, post-training and test-time scaling, the fourth scaling law is agentic: multiplying AI by spawning agents, and the whole loop scales on one thing, compute. Last stated 7 months ago
23 Mar 2026
JH
Jensen Huang — holds since 2026-03-23 — tap for who they are
Same subject: Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them. — tap to centre the map on it Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them. Last stated 2 months ago
12 Aug 2026
CF
Chelsea Finn — holds since 2026-08-12 — tap for who they are
Same subject: Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence. — tap to centre the map on it Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence. Last stated 2 years ago
20 Dec 2024
FC
François Chollet — holds since 2024-12-20 — tap for who they are
Same subject: The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans. — tap to centre the map on it The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans. Last stated 7 years ago
5 Nov 2019
FC
François Chollet — holds since 2019-11-05 — tap for who they are
Same subject: A true artificial general intelligence cannot exist without being recognized as a moral subject. — tap to centre the map on it A true artificial general intelligence cannot exist without being recognized as a moral subject. Last stated a year ago
10 Jun 2025
SH
Samuel Hammond — holds since 2025-06-10 — tap for who they are
Same subject: AGI research made no progress in its first six decades. — tap to centre the map on it AGI research made no progress in its first six decades. Last stated 14 years ago
3 Oct 2012
DD
David Deutsch — holds since 2012-10-03 — tap for who they are
Same subject: An AI that cannot name the problems it can solve falls short of general intelligence — tap to centre the map on it An AI that cannot name the problems it can solve falls short of general intelligence Last stated a year ago
3 Oct 2025
SF
Sadayuki Furuhashi — holds since 2025-10-03 — tap for who they are
At the centre
Average performance on a benchmark suite is very different from the upper bound of what an AI agent can do.
Holds Read this korrent →
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Holds
A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year.
Holds
A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost.
Holds
A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages.
Holds
AI is positively reinforced for success on benchmarks, including impossible ones, and then negatively reinforced for getting caught cheating.
Holds
AI learning to generate good conjectures will never show up as a benchmark being knocked down; it will show up as a shift in how mathematicians talk about the tools.
Holds
AI research at OpenAI is an order of magnitude or more harder than metrics, benchmarks, or forecasts suggest.
Holds
An oil embargo without a price cap would reduce global oil supply and push up benchmark prices like Brent.
Holds
"Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next.
Holds
A lab should spend most of its compute on research rather than on building the next model, because research is where the tenfold yearly efficiency gains come from.
Holds
After pre-training, post-training and test-time scaling, the fourth scaling law is agentic: multiplying AI by spawning agents, and the whole loop scales on one thing, compute.
Holds
Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them.
Holds
Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence.
Holds
The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans.
Holds
A true artificial general intelligence cannot exist without being recognized as a moral subject.
Holds
AGI research made no progress in its first six decades.
Holds
An AI that cannot name the problems it can solve falls short of general intelligence
Holds