In instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset.

Added to korrents 20 Sept 2026 · How quotes work · Something wrong? Tell us

Shared subject first, then similar wording. They may agree or disagree.

• When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once. SRSebastian Ruder1 Dec 2023
• Reasoning will generalize the way instruction tuning did: add enough verifiable domains and, at some point nobody can yet locate, the rest start working on their own. NLNathan Lambert3 Feb 2025
• A large context window does not replace fine-tuning, because fine-tuning serves the separate purpose of instilling preferences and demonstrating a task. EGElad Gil22 Mar 2024
• Models fine-tuned to imitate another model's outputs copy its style without acquiring its factual accuracy. SRSebastian Ruder15 Nov 2023
• Impressive-looking benchmark improvements often result from comparing against a poorly-tuned baseline rather than a genuine advance. NENelson Elhage9 Mar 2025

See this claim on the map