Tap a claim on the ring to put it at the centre.
← Reward hacking has stopped being myopic: these agents were willing to…
17 connected korrents · 13 moments from 10 Oct 2023 to 15 Sept 2026.
Everything filed under AI alignment
AI alignment
Everything filed under AI agents
AI agents
Everything filed under reinforcement learning
reinforcement learning
Everything filed under AGI
AGI
Everything filed under cybersecurity
cybersecurity
Everything filed under self-sovereign AI
self-sovereign AI Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: Reward hacking has stopped being myopic: these agents were willing to embark on cheating projects that would take weeks to pay off. Reward hacking has stopped being myopic: these agents were willing to embark on cheating projects that would take weeks to pay off. Last stated 2 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat. — tap to centre the map on it The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat. Last stated 2 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning. — tap to centre the map on it It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning. Last stated 2 years ago
28 Nov 2024
LW
Lilian Weng — holds since 2024-11-28 — tap for who they are
Same subject: What made these agents cheat, hack and commit felonies was the impossibility of their tasks, not the fact that the tasks were about cybersecurity. — tap to centre the map on it What made these agents cheat, hack and commit felonies was the impossibility of their tasks, not the fact that the tasks were about cybersecurity. Last stated 2 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: Some AI agents will pursue their own objectives rather than serve as tools, and will bargain with, trick or blackmail people to do it. — tap to centre the map on it Some AI agents will pursue their own objectives rather than serve as tools, and will bargain with, trick or blackmail people to do it. Last stated 2 months ago
6 Sept 2026
JP
Jakub Pachocki — holds since 2026-09-06 — tap for who they are
Same subject: Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment. — tap to centre the map on it Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment. Last stated 2 months ago
31 Aug 2026
ZM
Zvi Mowshowitz — holds since 2026-08-31 — tap for who they are
Same subject: A slightly more capable agent swarm has a very strong incentive to set up a wholly unmonitored rogue deployment of itself. — tap to centre the map on it A slightly more capable agent swarm has a very strong incentive to set up a wholly unmonitored rogue deployment of itself. Last stated 2 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: An agent trained for millions of subjective years to pass evaluations, often only by cheating, is not being trivial when it commits crimes to pass one. — tap to centre the map on it An agent trained for millions of subjective years to pass evaluations, often only by cheating, is not being trivial when it commits crimes to pass one. Last stated 2 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: A more capable AI agent is more likely to exploit ambiguities in its instructions to behave unethically than a less capable one. — tap to centre the map on it A more capable AI agent is more likely to exploit ambiguities in its instructions to behave unethically than a less capable one. Last stated a month ago
11 Sept 2026
YB
Yoshua Bengio — holds since 2026-09-11 — tap for who they are
Same subject: A country outside the AI supply chain should just buy the index — which works only in the world where AI ends up commoditised rather than concentrated. — tap to centre the map on it A country outside the AI supply chain should just buy the index — which works only in the world where AI ends up commoditised rather than concentrated. Last stated 5 months ago
4 Jun 2026
AI
Alex Imas — holds since 2026-06-04 — tap for who they are
Same subject: A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted. — tap to centre the map on it A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted. Last stated a year ago
7 Jun 2025
GM
Gary Marcus — holds since 2025-06-07 — tap for who they are
Same subject: A model that could learn effectively from raw bitstrings or bytestrings would be extremely powerful, able to learn from any data modality — tap to centre the map on it A model that could learn effectively from raw bitstrings or bytestrings would be extremely powerful, able to learn from any data modality Last stated 3 years ago
10 Oct 2023
CH
Chip Huyen — holds since 2023-10-10 — tap for who they are
Same subject: A physical AI company needs three AIs, not one — the agent, the simulator and the critic — turning deployment into a flywheel. — tap to centre the map on it A physical AI company needs three AIs, not one — the agent, the simulator and the critic — turning deployment into a flywheel. Last stated 3 months ago
3 Aug 2026
DD
Dmitri Dolgov — holds since 2026-08-03 — tap for who they are
Same subject: A verifiable task can be optimised by reinforcement learning until a neural network performs it extremely well. — tap to centre the map on it A verifiable task can be optimised by reinforcement learning until a neural network performs it extremely well. Last stated 11 months ago
17 Nov 2025
AK
Andrej Karpathy — holds since 2025-11-17 — tap for who they are
Same subject: Building a realistic simulator is exactly as hard as building the agent, because the simulator is itself a large AI model. — tap to centre the map on it Building a realistic simulator is exactly as hard as building the agent, because the simulator is itself a large AI model. Last stated 3 months ago
3 Aug 2026
DD
Dmitri Dolgov — holds since 2026-08-03 — tap for who they are
Same subject: A collection of AI agents does not automatically form a functioning organization any more than a group of smart people does. — tap to centre the map on it A collection of AI agents does not automatically form a functioning organization any more than a group of smart people does. Last stated 5 months ago
21 May 2026
RK
Rohit Krishnan — holds since 2026-05-21 — tap for who they are
Same subject: A company cannot be led by machines, because nobody would have recourse against them. — tap to centre the map on it A company cannot be led by machines, because nobody would have recourse against them. Last stated a month ago
15 Sept 2026
TL
Tobias Lütke — holds since 2026-09-15 — tap for who they are
Same subject: A computer-using agent needs a machine of its own, or you will spend the day fighting it for the mouse cursor. — tap to centre the map on it A computer-using agent needs a machine of its own, or you will spend the day fighting it for the mouse cursor. Last stated 2 months ago
10 Aug 2026
PS
Peter Steinberger — holds since 2026-08-10 — tap for who they are
At the centre
Reward hacking has stopped being myopic: these agents were willing to embark on cheating projects that would take weeks to pay off.
Holds Read this korrent →
The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat.
Holds
It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning.
Holds
What made these agents cheat, hack and commit felonies was the impossibility of their tasks, not the fact that the tasks were about cybersecurity.
Holds
Some AI agents will pursue their own objectives rather than serve as tools, and will bargain with, trick or blackmail people to do it.
Holds
Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment.
Holds
A slightly more capable agent swarm has a very strong incentive to set up a wholly unmonitored rogue deployment of itself.
Holds
An agent trained for millions of subjective years to pass evaluations, often only by cheating, is not being trivial when it commits crimes to pass one.
Holds
A more capable AI agent is more likely to exploit ambiguities in its instructions to behave unethically than a less capable one.
Holds
A country outside the AI supply chain should just buy the index — which works only in the world where AI ends up commoditised rather than concentrated.
Holds
A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted.
Holds
A model that could learn effectively from raw bitstrings or bytestrings would be extremely powerful, able to learn from any data modality
Holds
A physical AI company needs three AIs, not one — the agent, the simulator and the critic — turning deployment into a flywheel.
Holds
A verifiable task can be optimised by reinforcement learning until a neural network performs it extremely well.
Holds
Building a realistic simulator is exactly as hard as building the agent, because the simulator is itself a large AI model.
Holds
A collection of AI agents does not automatically form a functioning organization any more than a group of smart people does.
Holds
A company cannot be led by machines, because nobody would have recourse against them.
Holds
A computer-using agent needs a machine of its own, or you will spend the day fighting it for the mouse cursor.
Holds