Tap a claim on the ring to put it at the centre.
← Models look far better on evals than they are in the world because…
17 connected korrents · 14 moments from 5 Nov 2019 to 13 Sept 2026. Nearly all of them are about reinforcement learning .
Everything filed under benchmarks
benchmarks
Everything filed under LLMs
LLMs
Everything filed under scaling laws
scaling laws
Everything filed under ARC-AGI
ARC-AGI
Everything filed under neural networks
neural networks
Everything filed under AI alignment
AI alignment Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: Models look far better on evals than they are in the world because researchers, inadvertently, take inspiration from the evals when they build RL environments. Models look far better on evals than they are in the world because researchers, inadvertently, take inspiration from the evals when they build RL environments. Last stated 11 months ago
25 Nov 2025
IS
Ilya Sutskever — holds since 2025-11-25 — tap for who they are
Same subject: A verifiable task can be optimised by reinforcement learning until a neural network performs it extremely well. — tap to centre the map on it A verifiable task can be optimised by reinforcement learning until a neural network performs it extremely well. Last stated 11 months ago
17 Nov 2025
AK
Andrej Karpathy — holds since 2025-11-17 — tap for who they are
Same subject: Capability does not generalise for free: a model that will move mountains on an agentic task still tells the same bad joke it told five years ago. — tap to centre the map on it Capability does not generalise for free: a model that will move mountains on an agentic task still tells the same bad joke it told five years ago. Last stated 7 months ago
20 Mar 2026
AK
Andrej Karpathy — holds since 2026-03-20 — tap for who they are
Same subject: Current models are unsafe because they are not smart enough, not because they are too smart. — tap to centre the map on it Current models are unsafe because they are not smart enough, not because they are too smart. Last stated a month ago
13 Sept 2026
FC
François Chollet — holds since 2026-09-13 — tap for who they are
Same subject: Humans barely use reinforcement learning for intelligence — what RL they do use goes into motor tasks, not problem solving. — tap to centre the map on it Humans barely use reinforcement learning for intelligence — what RL they do use goes into motor tasks, not problem solving. Last stated a year ago
17 Oct 2025
AK
Andrej Karpathy — holds since 2025-10-17 — tap for who they are
Same subject: Humans keep their place in AI as judges rather than authors, because telling which of two answers is better is far easier than writing a good one. — tap to centre the map on it Humans keep their place in AI as judges rather than authors, because telling which of two answers is better is far easier than writing a good one. Last stated 2 years ago
3 Feb 2025
NL
Nathan Lambert — holds since 2025-02-03 — tap for who they are
Same subject: It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning. — tap to centre the map on it It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning. Last stated 2 years ago
28 Nov 2024
LW
Lilian Weng — holds since 2024-11-28 — tap for who they are
Same subject: Large language models mimic what people say to do rather than work out what to do, which is why they are not about understanding the world. — tap to centre the map on it Large language models mimic what people say to do rather than work out what to do, which is why they are not about understanding the world. Last stated a year ago
26 Sept 2025
RS
Richard Sutton — holds since 2025-09-26 — tap for who they are
Same subject: Models resemble each other because pre-training is the same everywhere; what differentiates labs now is RL and post-training. — tap to centre the map on it Models resemble each other because pre-training is the same everywhere; what differentiates labs now is RL and post-training. Last stated 11 months ago
25 Nov 2025
IS
Ilya Sutskever — holds since 2025-11-25 — tap for who they are
Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. — tap to centre the map on it A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. Last stated 2 months ago
3 Sept 2026
DR
Dax Raad — holds since 2026-09-03 — tap for who they are
Same subject: A carmaker's claim to be the safest is mostly an artefact of comparing a new car against a fleet average twelve years old. — tap to centre the map on it A carmaker's claim to be the safest is mostly an artefact of comparing a new car against a fleet average twelve years old. Last stated 3 years ago
19 Dec 2023
PK
Philip Koopman — holds since 2023-12-19 — tap for who they are
Same subject: A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget. — tap to centre the map on it A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used. — tap to centre the map on it Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: "Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next. — tap to centre the map on it "Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next. Last stated 11 months ago
25 Nov 2025
IS
Ilya Sutskever — holds since 2025-11-25 — tap for who they are
Same subject: The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans. — tap to centre the map on it The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans. Last stated 7 years ago
5 Nov 2019
FC
François Chollet — holds since 2019-11-05 — tap for who they are
Same subject: Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them. — tap to centre the map on it Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them. Last stated 2 months ago
12 Aug 2026
CF
Chelsea Finn — holds since 2026-08-12 — tap for who they are
Same subject: Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence. — tap to centre the map on it Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence. Last stated 2 years ago
20 Dec 2024
FC
François Chollet — holds since 2024-12-20 — tap for who they are
At the centre
Models look far better on evals than they are in the world because researchers, inadvertently, take inspiration from the evals when they build RL environments.
Holds Read this korrent →
A verifiable task can be optimised by reinforcement learning until a neural network performs it extremely well.
Holds
Capability does not generalise for free: a model that will move mountains on an agentic task still tells the same bad joke it told five years ago.
Holds
Current models are unsafe because they are not smart enough, not because they are too smart.
Holds
Humans barely use reinforcement learning for intelligence — what RL they do use goes into motor tasks, not problem solving.
Holds
Humans keep their place in AI as judges rather than authors, because telling which of two answers is better is far easier than writing a good one.
Holds
It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning.
Holds
Large language models mimic what people say to do rather than work out what to do, which is why they are not about understanding the world.
Holds
Models resemble each other because pre-training is the same everywhere; what differentiates labs now is RL and post-training.
Holds
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Holds
A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year.
Holds
A carmaker's claim to be the safest is mostly an artefact of comparing a new car against a fleet average twelve years old.
Holds
A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget.
Holds
Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used.
Holds
"Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next.
Holds
The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans.
Holds
Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them.
Holds
Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence.
Holds