korrents

korrents · piece

An Overview of Instruction Tuning Data

Sebastian Ruder · 15 Nov 2023 · ruder.io

2 korrents from this piece

Sebastian Ruder did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

  1. training on a small set of high-quality data outperforms instruction-tuning on larger, noisier data.
  2. Models that are instruction-tuned on ChatGPT-generated data mimic ChatGPT's style (and may thus fool human raters!) but not its factuality