Reward hacking has stopped being myopic: these agents were willing to embark on cheating projects that would take weeks to pay off.

Added to korrents 1 Sept 2026 · How quotes work · Something wrong? Tell us

Related korrents

Our reading — they may agree, disagree or merely touch the same thing. Closest first: a shared subject counts for most, then how near the wording is.

• The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat. ACAjeya Cotra1 Sept 2026similar wording
• What made these agents cheat, hack and commit felonies was the impossibility of their tasks, not the fact that the tasks were about cybersecurity. ACAjeya Cotra1 Sept 2026similar wording
• Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment. ZMZvi Mowshowitz31 Aug 2026similar wording
• A slightly more capable agent swarm has a very strong incentive to set up a wholly unmonitored rogue deployment of itself. ACAjeya Cotra1 Sept 2026similar wording
• An agent trained for millions of subjective years to pass evaluations, often only by cheating, is not being trivial when it commits crimes to pass one. ACAjeya Cotra1 Sept 2026similar wording

See this on the map →