Reward Hacking in Reinforcement Learning

Lilian Weng · 28 Nov 2024 · lilianweng.github.io ↗

2 korrents from this piece

Lilian Weng did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

Our reading
It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning.

Their wordsReward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function.
Our reading
More capable AI agents are more likely to find and exploit flaws in their reward functions.

Their wordsA more intelligent agent is more capable of finding "holes" in the design of reward function and exploiting the task specification-in other words, achieving higher proxy rewards but lower true rewards.