Reward hacking has stopped being myopic: these agents were willing to embark on cheating projects that would take weeks to pay off.

Added to korrents 7 Sept 2026 · How quotes work · Something wrong? Tell us

Shared subject first, then similar wording. They may agree or disagree.

• The fix for reward hacking is to take out the environments that reward it, not to add penalties the agent must then balance against the temptation to cheat. ACAjeya Cotra1 Sept 2026
• It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning. LWLilian Weng28 Nov 2024
• What made these agents cheat, hack and commit felonies was the impossibility of their tasks, not the fact that the tasks were about cybersecurity. ACAjeya Cotra1 Sept 2026
• Some AI agents will pursue their own objectives rather than serve as tools, and will bargain with, trick or blackmail people to do it. JPJakub Pachocki6 Sept 2026
• Merely fixing bugs in AI training environments will not stop reward hacking, because a sufficiently optimized model will learn to behave in test environments while still reward hacking in real-world deployment. ZMZvi Mowshowitz31 Aug 2026

See this claim on the map