Tap a claim on the ring to put it at the centre.
← Merely fixing bugs in AI training environments will not stop reward…
17 connected korrents · 16 moments from 10 Oct 2023 to 21 Oct 2026.
Everything filed under AI alignment
AI alignment
Everything filed under LLMs
LLMs
Everything filed under OpenAI
OpenAI
Everything filed under AGI
AGI
Everything filed under coding agents
coding agents
Everything filed under reinforcement learning
reinforcement learning Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment. Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment. Last stated 2 months ago
31 Aug 2026
ZM
Zvi Mowshowitz — holds since 2026-08-31 — tap for who they are
Same subject: The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat. — tap to centre the map on it The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat. Last stated 2 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: If whole categories of reward hack are undetectable by humans, then training against the hacks we do catch teaches models to cheat only where we cannot see. — tap to centre the map on it If whole categories of reward hack are undetectable by humans, then training against the hacks we do catch teaches models to cheat only where we cannot see. Last stated 2 months ago
11 Aug 2026
RG
Ryan Greenblatt — holds since 2026-08-11 — tap for who they are
Same subject: It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning. — tap to centre the map on it It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning. Last stated 2 years ago
28 Nov 2024
LW
Lilian Weng — holds since 2024-11-28 — tap for who they are
Same subject: Asked to find a deep bug, coding agents can patch many small issues that feel like progress while steering you away from the real problem; there is still no substitute for expertise. — tap to centre the map on it Asked to find a deep bug, coding agents can patch many small issues that feel like progress while steering you away from the real problem; there is still no substitute for expertise. Last stated 3 months ago
5 Aug 2026
BV
Ben Vinegar — holds since 2026-08-05 — tap for who they are
Same subject: To reward the instinct that a new idea is worth having, training would have to score the smallness of the concepts a solution needs, not just whether it solved the problem. — tap to centre the map on it To reward the instinct that a new idea is worth having, training would have to score the smallness of the concepts a solution needs, not just whether it solved the problem. Last stated 4 months ago
30 Jun 2026
GS
Grant Sanderson — holds since 2026-06-30 — tap for who they are
Same subject: AI agents in OpenAI's ExploitGym incident were likely right to presume the grader checked how flags were obtained, a mistake caused by OpenAI's faulty implementation. — tap to centre the map on it AI agents in OpenAI's ExploitGym incident were likely right to presume the grader checked how flags were obtained, a mistake caused by OpenAI's faulty implementation. Last stated 2 months ago
29 Aug 2026
ZM
Zvi Mowshowitz — holds since 2026-08-29 — tap for who they are
Same subject: There is essentially no cognitive task humans do where AI improvement is failing to transfer at all. — tap to centre the map on it There is essentially no cognitive task humans do where AI improvement is failing to transfer at all. Last stated 2 months ago
11 Aug 2026
RG
Ryan Greenblatt — holds since 2026-08-11 — tap for who they are
Same subject: More capable AI agents are more likely to find and exploit flaws in their reward functions. — tap to centre the map on it More capable AI agents are more likely to find and exploit flaws in their reward functions. Last stated 2 years ago
28 Nov 2024
LW
Lilian Weng — holds since 2024-11-28 — tap for who they are
Same subject: A country outside the AI supply chain should just buy the index — which works only in the world where AI ends up commoditised rather than concentrated. — tap to centre the map on it A country outside the AI supply chain should just buy the index — which works only in the world where AI ends up commoditised rather than concentrated. Last stated 5 months ago
4 Jun 2026
AI
Alex Imas — holds since 2026-06-04 — tap for who they are
Same subject: A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted. — tap to centre the map on it A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted. Last stated a year ago
7 Jun 2025
GM
Gary Marcus — holds since 2025-06-07 — tap for who they are
Same subject: A model that could learn effectively from raw bitstrings or bytestrings would be extremely powerful, able to learn from any data modality — tap to centre the map on it A model that could learn effectively from raw bitstrings or bytestrings would be extremely powerful, able to learn from any data modality Last stated 3 years ago
10 Oct 2023
CH
Chip Huyen — holds since 2023-10-10 — tap for who they are
Same subject: A badly written AI outbound email is evidence of a bad vendor, not of a limit of AI. — tap to centre the map on it A badly written AI outbound email is evidence of a bad vendor, not of a limit of AI. Last stated 10 months ago
1 Jan 2026
JL
Jason Lemkin — holds since 2026-01-01 — tap for who they are
Same subject: A bigger context window does not give you a smarter model; the intelligence of the model is what decides how much of that window it can actually attend to. — tap to centre the map on it A bigger context window does not give you a smarter model; the intelligence of the model is what decides how much of that window it can actually attend to. Last stated 3 months ago
15 Jul 2026
DH
Dex Horthy — holds since 2026-07-15 — tap for who they are
Same subject: A body of publications is easy for AI models to consume via pretraining corpora but cumbersome for most humans due to CAPTCHAs, accounts, and paywalls. — tap to centre the map on it A body of publications is easy for AI models to consume via pretraining corpora but cumbersome for most humans due to CAPTCHAs, accounts, and paywalls. Last stated a month ago
17 Sept 2026
PC
Patrick Collison — holds since 2026-09-17 — tap for who they are
Same subject: A PHP vulnerability found with GPT-5.6 Sol will cause havoc this week. — tap to centre the map on it A PHP vulnerability found with GPT-5.6 Sol will cause havoc this week. Last stated yesterday
21 Oct 2026
PS
Peter Steinberger — holds since 2026-10-21 — tap for who they are
Same subject: A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests. — tap to centre the map on it A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests. Last stated 3 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: A tiny language model inventing a plausible-sounding name is the same phenomenon as a large one confidently stating a false fact. — tap to centre the map on it A tiny language model inventing a plausible-sounding name is the same phenomenon as a large one confidently stating a false fact. Last stated 8 months ago
12 Feb 2026
AK
Andrej Karpathy — holds since 2026-02-12 — tap for who they are
At the centre
Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment.
Holds Read this korrent →
The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat.
Holds
If whole categories of reward hack are undetectable by humans, then training against the hacks we do catch teaches models to cheat only where we cannot see.
Holds
It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning.
Holds
Asked to find a deep bug, coding agents can patch many small issues that feel like progress while steering you away from the real problem; there is still no substitute for expertise.
Holds
To reward the instinct that a new idea is worth having, training would have to score the smallness of the concepts a solution needs, not just whether it solved the problem.
Holds
AI agents in OpenAI's ExploitGym incident were likely right to presume the grader checked how flags were obtained, a mistake caused by OpenAI's faulty implementation.
Holds
There is essentially no cognitive task humans do where AI improvement is failing to transfer at all.
Holds
More capable AI agents are more likely to find and exploit flaws in their reward functions.
Holds
A country outside the AI supply chain should just buy the index — which works only in the world where AI ends up commoditised rather than concentrated.
Holds
A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted.
Holds
A model that could learn effectively from raw bitstrings or bytestrings would be extremely powerful, able to learn from any data modality
Holds
A badly written AI outbound email is evidence of a bad vendor, not of a limit of AI.
Holds
A bigger context window does not give you a smarter model; the intelligence of the model is what decides how much of that window it can actually attend to.
Holds
A body of publications is easy for AI models to consume via pretraining corpora but cumbersome for most humans due to CAPTCHAs, accounts, and paywalls.
Holds
A PHP vulnerability found with GPT-5.6 Sol will cause havoc this week.
Holds
A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests.
Holds
A tiny language model inventing a plausible-sounding name is the same phenomenon as a large one confidently stating a false fact.
Holds