Tap a claim on the ring to put it at the centre.
← Evals are not the durable asset people think they are: one survives…
17 connected korrents · 15 moments from 5 Nov 2019 to 8 Oct 2026.
Everything filed under benchmarks
benchmarks
Everything filed under ARC-AGI
ARC-AGI
Everything filed under LLMs
LLMs
Everything filed under scaling laws
scaling laws
Everything filed under design
design
Everything filed under coding agents
coding agents
Everything filed under code generation
code generation
Everything filed under AI alignment
AI alignment Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: Evals are not the durable asset people think they are: one survives about one to three model generations before it saturates and has to be thrown away. Evals are not the durable asset people think they are: one survives about one to three model generations before it saturates and has to be thrown away. Last stated 3 months ago
27 Jul 2026
BC
Boris Cherny — holds since 2026-07-27 — tap for who they are
Same subject: A project that only shows the models are not ready yet is a win if it leaves behind an eval to retry with later models. — tap to centre the map on it A project that only shows the models are not ready yet is a win if it leaves behind an eval to retry with later models. Last stated a month ago
10 Sept 2026
MK
Mike Krieger — holds since 2026-09-10 — tap for who they are
Same subject: Models look far better on evals than they are in the world because researchers, inadvertently, take inspiration from the evals when they build RL environments. — tap to centre the map on it Models look far better on evals than they are in the world because researchers, inadvertently, take inspiration from the evals when they build RL environments. Last stated 11 months ago
25 Nov 2025
IS
Ilya Sutskever — holds since 2025-11-25 — tap for who they are
Same subject: Before building on a gap in the general models, work out whether that gap survives six months or three years. — tap to centre the map on it Before building on a gap in the general models, work out whether that gap survives six months or three years. Last stated 3 months ago
30 Jul 2026
JD
Jeff Dean — holds since 2026-07-30 — tap for who they are
Same subject: A visual eval lets designers who are not in the code build complex tooling and autonomous runs they can still shape and control the way they are used to. — tap to centre the map on it A visual eval lets designers who are not in the code build complex tooling and autonomous runs they can still shape and control the way they are used to. Last stated 2 months ago
11 Aug 2026
KZ
Kyle Zantos — holds since 2026-08-11 — tap for who they are
Same subject: A year from now every coding model on the market will still take a bad instruction at face value and go fix the listed issues, where a human senior engineer would insist on a rewrite. — tap to centre the map on it A year from now every coding model on the market will still take a bad instruction at face value and go fix the listed issues, where a human senior engineer would insist on a rewrite. Last stated 5 months ago
24 May 2026
DS
Dan Shipper — holds since 2026-05-24 — tap for who they are
Same subject: Super-exponential injection of external resources is not sustainable in the long-term. — tap to centre the map on it Super-exponential injection of external resources is not sustainable in the long-term. Last stated 2 weeks ago
8 Oct 2026
FC
François Chollet — holds since 2026-10-08 — tap for who they are
Same subject: You can now generate ten thousand variants of a function faster than you could write it once, and that economics pushes software towards replacement rather than repair. — tap to centre the map on it You can now generate ten thousand variants of a function faster than you could write it once, and that economics pushes software towards replacement rather than repair. Last stated 2 months ago
12 Aug 2026
CM
Charity Majors — holds since 2026-08-12 — tap for who they are
Same subject: If declines in chain-of-thought monitorability come only from capability gains, CoT monitoring is unlikely to last another year without active improvement. — tap to centre the map on it If declines in chain-of-thought monitorability come only from capability gains, CoT monitoring is unlikely to last another year without active improvement. Last stated a month ago
8 Sept 2026
ZM
Zvi Mowshowitz — holds since 2026-09-08 — tap for who they are
Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. — tap to centre the map on it A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. Last stated 2 months ago
3 Sept 2026
DR
Dax Raad — holds since 2026-09-03 — tap for who they are
Same subject: A carmaker's claim to be the safest is mostly an artefact of comparing a new car against a fleet average twelve years old. — tap to centre the map on it A carmaker's claim to be the safest is mostly an artefact of comparing a new car against a fleet average twelve years old. Last stated 3 years ago
19 Dec 2023
PK
Philip Koopman — holds since 2023-12-19 — tap for who they are
Same subject: A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget. — tap to centre the map on it A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores. — tap to centre the map on it Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores. Last stated 2 years ago
13 May 2024
SR
Sebastian Ruder — holds since 2024-05-13 — tap for who they are
Same subject: Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used. — tap to centre the map on it Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans. — tap to centre the map on it The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans. Last stated 7 years ago
5 Nov 2019
FC
François Chollet — holds since 2019-11-05 — tap for who they are
Same subject: Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them. — tap to centre the map on it Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them. Last stated 2 months ago
12 Aug 2026
CF
Chelsea Finn — holds since 2026-08-12 — tap for who they are
Same subject: Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence. — tap to centre the map on it Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence. Last stated 2 years ago
20 Dec 2024
FC
François Chollet — holds since 2024-12-20 — tap for who they are
At the centre
Evals are not the durable asset people think they are: one survives about one to three model generations before it saturates and has to be thrown away.
Holds Read this korrent →
A project that only shows the models are not ready yet is a win if it leaves behind an eval to retry with later models.
Holds
Models look far better on evals than they are in the world because researchers, inadvertently, take inspiration from the evals when they build RL environments.
Holds
Before building on a gap in the general models, work out whether that gap survives six months or three years.
Holds
A visual eval lets designers who are not in the code build complex tooling and autonomous runs they can still shape and control the way they are used to.
Holds
A year from now every coding model on the market will still take a bad instruction at face value and go fix the listed issues, where a human senior engineer would insist on a rewrite.
Holds
Super-exponential injection of external resources is not sustainable in the long-term.
Holds
You can now generate ten thousand variants of a function faster than you could write it once, and that economics pushes software towards replacement rather than repair.
Holds
If declines in chain-of-thought monitorability come only from capability gains, CoT monitoring is unlikely to last another year without active improvement.
Holds
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Holds
A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year.
Holds
A carmaker's claim to be the safest is mostly an artefact of comparing a new car against a fleet average twelve years old.
Holds
A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget.
Holds
Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.
Holds
Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used.
Holds
The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans.
Holds
Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them.
Holds
Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence.
Holds