korrents

korrents · papers

Training language models to follow instructions with human feedback (with 19 co-authors)

Jan Leike · 4 Mar 2022 · arxiv.org ↗

4 korrents from this paper

In plain words

Teaching language models via human good-answer examples and output rankings makes a small one preferred to a 100x larger plain model, with less lying and toxicity. This shows just making them bigger fails to make them do what users want and gives a workable fix that keeps most other skills intact. It reports experimental results on fine-tuning with human feedback for broad instructions, extending earlier techniques used mainly for summarization.

Our summary of the paper, not the authors' words — written to be readable without the field's vocabulary, from the stored copy of the paper and nothing else. Drafted with xai:grok-4.5 and checked by a person. The authors' own sentences are the quotes below.

Near this, by wording

Papers whose claims are worded most like this one's, found by the same hourly pass that draws the map. It is a measure of LANGUAGE, not of agreement or of citation: two papers can be near each other here and flatly contradict one another.

Jan Leike did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

  1. Our reading

    Increasing language model size does not inherently improve following of user intent.

    Their wordsMaking language models bigger does not inherently make them better at following a user's intent.

  2. Our reading

    Large language models are not aligned with their users.

    Their wordsFor example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users.

  3. Our reading

    Fine-tuning with human feedback is a promising direction for aligning language models with human intent.

    Their wordsEven though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.

  4. Our reading

    The next-token language modeling objective is misaligned with following user instructions helpfully and safely.

    Their wordsThis is because the language modeling objective used for many recent large LMs-predicting the next token on a webpage from the internet-is different from the objective "follow the user's instructions helpfully and safely" (Radford et al.,, 2019; Brown et al.,, 2020; Fedus et al.,, 2021; Rae et al.,, 2021; Thoppilan et al.,, 2022). Thus, we say that the language modeling objective is misaligned.