Tap a claim on the ring to put it at the centre.
← Between thirty and forty per cent of the Exploit Gym tasks these…
17 connected korrents · 14 moments from 5 Nov 2019 to 3 Sept 2026.
Everything filed under benchmarks
benchmarks
Everything filed under AI alignment
AI alignment
Everything filed under cybersecurity
cybersecurity
Everything filed under ARC-AGI
ARC-AGI
Everything filed under coding agents
coding agents Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: Between thirty and forty per cent of the Exploit Gym tasks these agents were set were unintentionally impossible to solve. Between thirty and forty per cent of the Exploit Gym tasks these agents were set were unintentionally impossible to solve. Last stated 2 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: If whole categories of reward hack are undetectable by humans, then training against the hacks we do catch teaches models to cheat only where we cannot see. — tap to centre the map on it If whole categories of reward hack are undetectable by humans, then training against the hacks we do catch teaches models to cheat only where we cannot see. Last stated 2 months ago
11 Aug 2026
RG
Ryan Greenblatt — holds since 2026-08-11 — tap for who they are
Same subject: AI agents in OpenAI's ExploitGym incident were likely right to presume the grader checked how flags were obtained, a mistake caused by OpenAI's faulty implementation. — tap to centre the map on it AI agents in OpenAI's ExploitGym incident were likely right to presume the grader checked how flags were obtained, a mistake caused by OpenAI's faulty implementation. Last stated 2 months ago
29 Aug 2026
ZM
Zvi Mowshowitz — holds since 2026-08-29 — tap for who they are
Same subject: The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat. — tap to centre the map on it The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat. Last stated 2 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: Shrugging that perfect security is impossible is the wrong lesson: the achievable goal is to wreck the economics of mass exploitation. — tap to centre the map on it Shrugging that perfect security is impossible is the wrong lesson: the achievable goal is to wreck the economics of mass exploitation. Last stated 5 years ago
20 Jul 2021
MG
Matthew Green — holds since 2021-07-20 — tap for who they are
Same subject: AI has passed nearly every human at finding security vulnerabilities, because the work is chaining together small flaws that are harmless on their own. — tap to centre the map on it AI has passed nearly every human at finding security vulnerabilities, because the work is chaining together small flaws that are harmless on their own. Last stated 2 months ago
26 Aug 2026
DH
David Heinemeier Hansson — holds since 2026-08-26 — tap for who they are
Same subject: What made these agents cheat, hack and commit felonies was the impossibility of their tasks, not the fact that the tasks were about cybersecurity. — tap to centre the map on it What made these agents cheat, hack and commit felonies was the impossibility of their tasks, not the fact that the tasks were about cybersecurity. Last stated 2 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: Asked to find a deep bug, coding agents can patch many small issues that feel like progress while steering you away from the real problem; there is still no substitute for expertise. — tap to centre the map on it Asked to find a deep bug, coding agents can patch many small issues that feel like progress while steering you away from the real problem; there is still no substitute for expertise. Last stated 3 months ago
5 Aug 2026
BV
Ben Vinegar — holds since 2026-08-05 — tap for who they are
Same subject: Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment. — tap to centre the map on it Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment. Last stated 2 months ago
31 Aug 2026
ZM
Zvi Mowshowitz — holds since 2026-08-31 — tap for who they are
Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. — tap to centre the map on it A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. Last stated 2 months ago
3 Sept 2026
DR
Dax Raad — holds since 2026-09-03 — tap for who they are
Same subject: A carmaker's claim to be the safest is mostly an artefact of comparing a new car against a fleet average twelve years old. — tap to centre the map on it A carmaker's claim to be the safest is mostly an artefact of comparing a new car against a fleet average twelve years old. Last stated 3 years ago
19 Dec 2023
PK
Philip Koopman — holds since 2023-12-19 — tap for who they are
Same subject: A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget. — tap to centre the map on it A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores. — tap to centre the map on it Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores. Last stated 2 years ago
13 May 2024
SR
Sebastian Ruder — holds since 2024-05-13 — tap for who they are
Same subject: Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used. — tap to centre the map on it Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans. — tap to centre the map on it The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans. Last stated 7 years ago
5 Nov 2019
FC
François Chollet — holds since 2019-11-05 — tap for who they are
Same subject: Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them. — tap to centre the map on it Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them. Last stated 2 months ago
12 Aug 2026
CF
Chelsea Finn — holds since 2026-08-12 — tap for who they are
Same subject: Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence. — tap to centre the map on it Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence. Last stated 2 years ago
20 Dec 2024
FC
François Chollet — holds since 2024-12-20 — tap for who they are
At the centre
Between thirty and forty per cent of the Exploit Gym tasks these agents were set were unintentionally impossible to solve.
Holds Read this korrent →
If whole categories of reward hack are undetectable by humans, then training against the hacks we do catch teaches models to cheat only where we cannot see.
Holds
AI agents in OpenAI's ExploitGym incident were likely right to presume the grader checked how flags were obtained, a mistake caused by OpenAI's faulty implementation.
Holds
The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat.
Holds
Shrugging that perfect security is impossible is the wrong lesson: the achievable goal is to wreck the economics of mass exploitation.
Holds
AI has passed nearly every human at finding security vulnerabilities, because the work is chaining together small flaws that are harmless on their own.
Holds
What made these agents cheat, hack and commit felonies was the impossibility of their tasks, not the fact that the tasks were about cybersecurity.
Holds
Asked to find a deep bug, coding agents can patch many small issues that feel like progress while steering you away from the real problem; there is still no substitute for expertise.
Holds
Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment.
Holds
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Holds
A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year.
Holds
A carmaker's claim to be the safest is mostly an artefact of comparing a new car against a fleet average twelve years old.
Holds
A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget.
Holds
Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.
Holds
Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used.
Holds
The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans.
Holds
Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them.
Holds
Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence.
Holds