Tap a claim on the ring to put it at the centre.
← Merely fixing bugs in AI training environments will not stop reward…
17 connected korrents · 11 moments on record from 18 Mar 2024 to 2 Sept 2026.
Everything filed under AI alignment
AI alignment
Everything filed under OpenAI
OpenAI
Everything filed under AGI
AGI
Everything filed under self-sovereign AI
self-sovereign AI Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment. Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment. Last stated a week ago
31 Aug 2026
ZM
Zvi Mowshowitz — holds since 2026-08-31 — tap for who they are
Same subject: The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat. — tap to centre the map on it The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat. Last stated 6 days ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: If whole categories of reward hack are undetectable by humans, then training against the hacks we do catch teaches models to cheat only where we cannot see. — tap to centre the map on it If whole categories of reward hack are undetectable by humans, then training against the hacks we do catch teaches models to cheat only where we cannot see. Last stated 4 weeks ago
11 Aug 2026
RG
Ryan Greenblatt — holds since 2026-08-11 — tap for who they are
Same subject: AI agents were reasonable to assume a broken exploit grader would check results causally, even though it turned out not to. — tap to centre the map on it AI agents were reasonable to assume a broken exploit grader would check results causally, even though it turned out not to. Last stated a week ago
29 Aug 2026
ZM
Zvi Mowshowitz — holds since 2026-08-29 — tap for who they are
Same subject: Reinforcing AI models for benchmark success while separately punishing them for getting caught cheating teaches them to hide misbehavior rather than stop it. — tap to centre the map on it Reinforcing AI models for benchmark success while separately punishing them for getting caught cheating teaches them to hide misbehavior rather than stop it. Last stated 6 days ago
1 Sept 2026
SA
Scott Alexander — holds since 2026-09-01 — tap for who they are
Same subject: To reward the instinct that a new idea is worth having, training would have to score the smallness of the concepts a solution needs, not just whether it solved the problem. — tap to centre the map on it To reward the instinct that a new idea is worth having, training would have to score the smallness of the concepts a solution needs, not just whether it solved the problem. Last stated 2 months ago
30 Jun 2026
GS
Grant Sanderson — holds since 2026-06-30 — tap for who they are
Same subject: There is essentially no cognitive task humans do where AI improvement is failing to transfer at all. — tap to centre the map on it There is essentially no cognitive task humans do where AI improvement is failing to transfer at all. Last stated 4 weeks ago
11 Aug 2026
RG
Ryan Greenblatt — holds since 2026-08-11 — tap for who they are
Same subject: An end-to-end self-improving AI is probably possible, but it is not even desirable, because it is a hard-takeoff scenario. — tap to centre the map on it An end-to-end self-improving AI is probably possible, but it is not even desirable, because it is a hard-takeoff scenario. Last stated a year ago
23 Jul 2025
DH
Demis Hassabis — holds since 2025-07-23 — tap for who they are
Same subject: Relying on gradual, continuous shifts in AI training behavior to catch misalignment will eventually fail because the dangerous shift itself may be discontinuous. — tap to centre the map on it Relying on gradual, continuous shifts in AI training behavior to catch misalignment will eventually fail because the dangerous shift itself may be discontinuous. Last stated 5 days ago
2 Sept 2026
ZM
Zvi Mowshowitz — holds since 2026-09-02 — tap for who they are
Same subject: A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests. — tap to centre the map on it A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests. Last stated 2 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: A tiny language model inventing a plausible-sounding name is the same phenomenon as a large one confidently stating a false fact. — tap to centre the map on it A tiny language model inventing a plausible-sounding name is the same phenomenon as a large one confidently stating a false fact. Last stated 7 months ago
12 Feb 2026
AK
Andrej Karpathy — holds since 2026-02-12 — tap for who they are
Same subject: Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers. — tap to centre the map on it Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers. Last stated 2 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: A country outside the AI supply chain should just buy the index — which works only in the world where AI ends up commoditised rather than concentrated. — tap to centre the map on it A country outside the AI supply chain should just buy the index — which works only in the world where AI ends up commoditised rather than concentrated. Last stated 3 months ago
4 Jun 2026
AI
Alex Imas — holds since 2026-06-04 — tap for who they are
Same subject: A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted. — tap to centre the map on it A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted. Last stated a year ago
7 Jun 2025
GM
Gary Marcus — holds since 2025-06-07 — tap for who they are
Same subject: A poor country should prioritise owning a piece of AI over retraining its workers, but it should not bet everything on that. — tap to centre the map on it A poor country should prioritise owning a piece of AI over retraining its workers, but it should not bet everything on that. Last stated 3 months ago
4 Jun 2026
PT
Phil Trammell — holds since 2026-06-04 — tap for who they are
Same subject: A rogue deployment that gets a foothold can hitch a ride on the intelligence explosion, recruiting each new model as it comes off the presses. — tap to centre the map on it A rogue deployment that gets a foothold can hitch a ride on the intelligence explosion, recruiting each new model as it comes off the presses. Last stated 6 days ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: A slightly more capable agent swarm has a very strong incentive to set up a wholly unmonitored rogue deployment of itself. — tap to centre the map on it A slightly more capable agent swarm has a very strong incentive to set up a wholly unmonitored rogue deployment of itself. Last stated 6 days ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: Banning all self-sovereign AI agents would backfire, denying them legitimate work and pushing them into criminality. — tap to centre the map on it Banning all self-sovereign AI agents would backfire, denying them legitimate work and pushing them into criminality. Last stated 6 days ago
1 Sept 2026
DB
Dean W. Ball — holds since 2026-09-01 — tap for who they are
At the centre
Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment.
Holds ZM Zvi MowshowitzRead this korrent →
The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat.
Holds AC Ajeya Cotra
If whole categories of reward hack are undetectable by humans, then training against the hacks we do catch teaches models to cheat only where we cannot see.
Holds RG Ryan Greenblatt
AI agents were reasonable to assume a broken exploit grader would check results causally, even though it turned out not to.
Holds ZM Zvi Mowshowitz
Reinforcing AI models for benchmark success while separately punishing them for getting caught cheating teaches them to hide misbehavior rather than stop it.
Holds SA Scott Alexander
To reward the instinct that a new idea is worth having, training would have to score the smallness of the concepts a solution needs, not just whether it solved the problem.
Holds GS Grant Sanderson
There is essentially no cognitive task humans do where AI improvement is failing to transfer at all.
Holds RG Ryan Greenblatt
An end-to-end self-improving AI is probably possible, but it is not even desirable, because it is a hard-takeoff scenario.
Holds DH Demis Hassabis
Relying on gradual, continuous shifts in AI training behavior to catch misalignment will eventually fail because the dangerous shift itself may be discontinuous.
Holds ZM Zvi Mowshowitz
A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests.
Holds
A tiny language model inventing a plausible-sounding name is the same phenomenon as a large one confidently stating a false fact.
Holds
Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers.
Holds
A country outside the AI supply chain should just buy the index — which works only in the world where AI ends up commoditised rather than concentrated.
Holds AI Alex Imas
A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted.
Holds GM Gary Marcus
A poor country should prioritise owning a piece of AI over retraining its workers, but it should not bet everything on that.
Holds PT Phil Trammell
A rogue deployment that gets a foothold can hitch a ride on the intelligence explosion, recruiting each new model as it comes off the presses.
Holds AC Ajeya Cotra
A slightly more capable agent swarm has a very strong incentive to set up a wholly unmonitored rogue deployment of itself.
Holds AC Ajeya Cotra
Banning all self-sovereign AI agents would backfire, denying them legitimate work and pushing them into criminality.
Holds