korrents
Lilian Weng

Lilian Weng

@lilian-weng · 16 positions · On ppll ↗

Machine-learning researcher; has written the Lil'Log survey posts on how a model technique works since 2017, and worked at OpenAI from 2018 to 2024.

Lilian Weng did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

6 of Lilian Weng's claims fit 5 mental models. Most often: When a measure becomes a target, Bottlenecks, Craft in the details.

Models they name

Their own words name the idea.

When a measure becomes a target

Reward a number and people, or machines, will find ways to move the number without the thing it measured (Goodhart's law).

Used by 16 others

It is fundamentally difficult to design a reward function that accurately captures the intended goal in reinforcement learning.

  1. Lilian Weng Machine-learning researcher Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function. Reward Hacking in Reinforcement Learninglilianweng.github.io · 28 Nov 2024All korrents from this piece
    Their wordsReward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function.

    ↗Reward Hacking in Reinforcement Learninglilianweng.github.io

    28 Nov 2024

More capable AI agents are more likely to find and exploit flaws in their reward functions.

  1. Lilian Weng Machine-learning researcher A more intelligent agent is more capable of finding "holes" in the design of reward function and exploiting the task specification-in other words, achieving higher proxy rewards but lower true rewards. Reward Hacking in Reinforcement Learninglilianweng.github.io · 28 Nov 2024All korrents from this piece
    Their wordsA more intelligent agent is more capable of finding "holes" in the design of reward function and exploiting the task specification-in other words, achieving higher proxy rewards but lower true rewards.

    ↗Reward Hacking in Reinforcement Learninglilianweng.github.io

    28 Nov 2024

Craft in the details

Care about the parts nobody sees; quality shows in the details.

Used by 20 others

High-quality data collection depends more on careful human execution than on machine learning techniques alone.

  1. Lilian Weng Machine-learning researcher Lots of ML techniques in the post can help with data quality, but fundamentally human data collection involves attention to details and careful execution. Thinking about High-Quality Human Datalilianweng.github.io · 5 Feb 2024All korrents from this piece
    Their wordsLots of ML techniques in the post can help with data quality, but fundamentally human data collection involves attention to details and careful execution.

    ↗Thinking about High-Quality Human Datalilianweng.github.io

    5 Feb 2024

Error correction

Nobody and nothing is infallible, so judge a person, a text or an institution by whether it can notice its own mistakes and fix them, not by its claim to be right.

Used by 11 others

Large language models cannot reliably self-correct their own mistakes without external feedback.

  1. Lilian Weng Machine-learning researcher this self-correction capability turns out to not exist intrinsically among LLMs and does not easily work out of the box, due to various failure modes Why We Thinklilianweng.github.io · 1 May 2025All korrents from this piece
    Their wordsthis self-correction capability turns out to not exist intrinsically among LLMs and does not easily work out of the box, due to various failure modes

    ↗Why We Thinklilianweng.github.io

    1 May 2025

Models we see in what they say

Our reading: their claim applies the idea without naming it. The claim is theirs; filing it here is ours.

Bottlenecks

A system moves only as fast as its narrowest point; speed up anything else and nothing changes.

Used by 91 others

As models keep growing, AI is running out of enough high-quality unique training tokens to keep up.

  1. Lilian Weng Machine-learning researcher As the model size grows significantly, we are running out of enough high-quality unique tokens. Scaling Laws, Carefullylilianweng.github.io · 24 Jun 2026All korrents from this piece
    Their wordsAs the model size grows significantly, we are running out of enough high-quality unique tokens.

    ↗Scaling Laws, Carefullylilianweng.github.io

    24 Jun 2026

Incentives

Look at what people are rewarded for; it explains more behaviour than what they say, your own included.

Used by 89 others

Chain-of-thought explanations from language models can be biased because training does not explicitly reward faithful reasoning.

  1. Lilian Weng Machine-learning researcher model CoTs could be biased due to lack of explicit training objectives aimed at encouraging faithful reasoning Why We Thinklilianweng.github.io · 1 May 2025All korrents from this piece
    Their wordsmodel CoTs could be biased due to lack of explicit training objectives aimed at encouraging faithful reasoning

    ↗Why We Thinklilianweng.github.io

    1 May 2025