Tap a claim on the ring to put it at the centre.
← Behaviours like massively deluding humans or trying to seize control…
17 connected korrents · 16 moments from 12 May 2023 to 3 Nov 2026. Nearly all of them are about AI alignment .
Everything filed under reinforcement learning
reinforcement learning
Everything filed under LLMs
LLMs
Everything filed under AI agents
AI agents
Everything filed under cybersecurity
cybersecurity
Everything filed under AGI
AGI
Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject
Read this korrent: Behaviours like massively deluding humans or trying to seize control of assets could in principle generalise from reward hacking.
Behaviours like massively deluding humans or trying to seize control of assets could in principle generalise from reward hacking.
Last stated a year ago
8 Jul 2025
RG
Ryan Greenblatt — holds since 2025-07-08 — tap for who they are
Same subject: It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning. — tap to centre the map on it
It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning.
Last stated 2 years ago
28 Nov 2024
LW
Lilian Weng — holds since 2024-11-28 — tap for who they are
Same subject: If whole categories of reward hack are undetectable by humans, then training against the hacks we do catch teaches models to cheat only where we cannot see. — tap to centre the map on it
If whole categories of reward hack are undetectable by humans, then training against the hacks we do catch teaches models to cheat only where we cannot see.
Last stated 3 months ago
11 Aug 2026
RG
Ryan Greenblatt — holds since 2026-08-11 — tap for who they are
Same subject: The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat. — tap to centre the map on it
The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat.
Last stated 3 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: A scary demonstration would show an AI punished for reward hacking behave reasonably until it gets a new opportunity, and then take it. — tap to centre the map on it
A scary demonstration would show an AI punished for reward hacking behave reasonably until it gets a new opportunity, and then take it.
Last stated 4 years ago
12 May 2023
AC
Ajeya Cotra — holds since 2023-05-12 — tap for who they are
Same subject: What made these agents cheat, hack and commit felonies was the impossibility of their tasks, not the fact that the tasks were about cybersecurity. — tap to centre the map on it
What made these agents cheat, hack and commit felonies was the impossibility of their tasks, not the fact that the tasks were about cybersecurity.
Last stated 3 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: Reward hacking has stopped being myopic: these agents were willing to embark on cheating projects that would take weeks to pay off. — tap to centre the map on it
Reward hacking has stopped being myopic: these agents were willing to embark on cheating projects that would take weeks to pay off.
Last stated 3 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment. — tap to centre the map on it
Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment.
Last stated 3 months ago
31 Aug 2026
ZM
Zvi Mowshowitz — holds since 2026-08-31 — tap for who they are
Same subject: How aligned current models are depends a lot on things like their degree of reward hacking. — tap to centre the map on it
How aligned current models are depends a lot on things like their degree of reward hacking.
Last stated 11 months ago
27 Dec 2025
NN
Neel Nanda — holds since 2025-12-27 — tap for who they are
Same subject: A country outside the AI supply chain should just buy the index — which works only in the world where AI ends up commoditised rather than concentrated. — tap to centre the map on it
A country outside the AI supply chain should just buy the index — which works only in the world where AI ends up commoditised rather than concentrated.
Last stated 6 months ago
4 Jun 2026
AI
Alex Imas — holds since 2026-06-04 — tap for who they are
Same subject: A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted. — tap to centre the map on it
A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted.
Last stated a year ago
7 Jun 2025
GM
Gary Marcus — holds since 2025-06-07 — tap for who they are
Same subject: A model that could learn effectively from raw bitstrings or bytestrings would be extremely powerful, able to learn from any data modality — tap to centre the map on it
A model that could learn effectively from raw bitstrings or bytestrings would be extremely powerful, able to learn from any data modality
Last stated 3 years ago
10 Oct 2023
CH
Chip Huyen — holds since 2023-10-10 — tap for who they are
Same subject: "Return on tokens" has the same measurement problems as hours and capacity: it still hinges on a theory of value. — tap to centre the map on it
"Return on tokens" has the same measurement problems as hours and capacity: it still hinges on a theory of value.
Last stated 3 months ago
17 Aug 2026
JC
John Cutler — holds since 2026-08-17 — tap for who they are
Same subject: A badly written AI outbound email is evidence of a bad vendor, not of a limit of AI. — tap to centre the map on it
A badly written AI outbound email is evidence of a bad vendor, not of a limit of AI.
Last stated 11 months ago
1 Jan 2026
JL
Jason Lemkin — holds since 2026-01-01 — tap for who they are
Same subject: A base model is essentially a very expensive autocomplete that continues text according to the statistics of its training documents. — tap to centre the map on it
A base model is essentially a very expensive autocomplete that continues text according to the statistics of its training documents.
Last stated 2 years ago
5 Feb 2025
AK
Andrej Karpathy — holds since 2025-02-05 — tap for who they are
Same subject: A codebase is the most important context an AI agent has, and letting many different models add to it without consistent patterns leads to trouble. — tap to centre the map on it
A codebase is the most important context an AI agent has, and letting many different models add to it without consistent patterns leads to trouble.
Last stated 3 weeks ago
3 Nov 2026
DB
Derek Briggs — holds since 2026-11-03 — tap for who they are
Same subject: A collection of AI agents does not automatically form a functioning organization any more than a group of smart people does. — tap to centre the map on it
A collection of AI agents does not automatically form a functioning organization any more than a group of smart people does.
Last stated 6 months ago
21 May 2026
RK
Rohit Krishnan — holds since 2026-05-21 — tap for who they are
Same subject: A company cannot be led by machines, because nobody would have recourse against them. — tap to centre the map on it
A company cannot be led by machines, because nobody would have recourse against them.
Last stated 2 months ago
15 Sept 2026
TL
Tobias Lütke — holds since 2026-09-15 — tap for who they are
same subject or similar wording a cloud: claims about one subject, named for it bar: when it was last stated, on a scale from 2015 to today — full is today a face: someone who holds the claim — tap it for who they are
At the centre
Behaviours like massively deluding humans or trying to seize control of assets could in principle generalise from reward hacking.
Last stated 8 Jul 2025 · a year ago
Holds Ryan Greenblatt
Read this korrent →
Similar wording
It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning.
Last stated 28 Nov 2024 · 2 years ago
Holds Lilian Weng
Similar wording
If whole categories of reward hack are undetectable by humans, then training against the hacks we do catch teaches models to cheat only where we cannot see.
Last stated 11 Aug 2026 · 3 months ago
Holds Ryan Greenblatt
Similar wording
The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat.
Last stated 1 Sept 2026 · 3 months ago
Holds Ajeya Cotra
Similar wording
A scary demonstration would show an AI punished for reward hacking behave reasonably until it gets a new opportunity, and then take it.
Last stated 12 May 2023 · 4 years ago
Holds Ajeya Cotra
Similar wording
What made these agents cheat, hack and commit felonies was the impossibility of their tasks, not the fact that the tasks were about cybersecurity.
Last stated 1 Sept 2026 · 3 months ago
Holds Ajeya Cotra
Similar wording
Reward hacking has stopped being myopic: these agents were willing to embark on cheating projects that would take weeks to pay off.
Last stated 1 Sept 2026 · 3 months ago
Holds Ajeya Cotra
Similar wording
Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment.
Last stated 31 Aug 2026 · 3 months ago
Holds Zvi Mowshowitz
Similar wording
How aligned current models are depends a lot on things like their degree of reward hacking.
Last stated 27 Dec 2025 · 11 months ago
Holds Neel Nanda
Same subject: AGI
A country outside the AI supply chain should just buy the index — which works only in the world where AI ends up commoditised rather than concentrated.
Last stated 4 Jun 2026 · 6 months ago
Holds Alex Imas
Same subject: AGI
A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted.
Last stated 7 Jun 2025 · a year ago
Holds Gary Marcus
Same subject: AGI
A model that could learn effectively from raw bitstrings or bytestrings would be extremely powerful, able to learn from any data modality
Last stated 10 Oct 2023 · 3 years ago
Holds Chip Huyen
Same subject: LLMs
"Return on tokens" has the same measurement problems as hours and capacity: it still hinges on a theory of value.
Last stated 17 Aug 2026 · 3 months ago
Holds John Cutler
Same subject: LLMs
A badly written AI outbound email is evidence of a bad vendor, not of a limit of AI.
Last stated 1 Jan 2026 · 11 months ago
Holds Jason Lemkin
Same subject: LLMs
A base model is essentially a very expensive autocomplete that continues text according to the statistics of its training documents.
Last stated 5 Feb 2025 · 2 years ago
Holds Andrej Karpathy
Same subject: AI agents
A codebase is the most important context an AI agent has, and letting many different models add to it without consistent patterns leads to trouble.
Last stated 3 Nov 2026 · 3 weeks ago
Holds Derek Briggs
Same subject: AI agents
A collection of AI agents does not automatically form a functioning organization any more than a group of smart people does.
Last stated 21 May 2026 · 6 months ago
Holds Rohit Krishnan
Same subject: AI agents
A company cannot be led by machines, because nobody would have recourse against them.
Last stated 15 Sept 2026 · 2 months ago
Holds Tobias Lütke