Tap a claim on the ring to put it at the centre.
← In instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset.
17 connected korrents · 15 moments from 29 Sept 2018 to 5 Nov 2026. Nearly all of them are about LLMs .
Everything filed under simplicity
simplicity
Everything filed under OpenAI
OpenAI
Everything filed under scaling laws
scaling laws
Everything filed under reinforcement learning
reinforcement learning
Everything filed under AI alignment
AI alignment
Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject
Read this korrent: In instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset.
In instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset.
Last stated 3 years ago
15 Nov 2023
SR
Sebastian Ruder — holds since 2023-11-15 — tap for who they are
Same subject: The argument that RAG and long context make fine-tuning unnecessary holds when information is missing, but not when the model struggles with the task itself. — tap to centre the map on it
The argument that RAG and long context make fine-tuning unnecessary holds when information is missing, but not when the model struggles with the task itself.
Last stated 2 weeks ago
5 Nov 2026
AR
Aishwarya Naresh Reganti — holds since 2026-11-05 — tap for who they are
Same subject: When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once. — tap to centre the map on it
When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once.
Last stated 3 years ago
1 Dec 2023
SR
Sebastian Ruder — holds since 2023-12-01 — tap for who they are
Same subject: Reasoning will generalize the way instruction tuning did: add enough verifiable domains and, at some point nobody can yet locate, the rest start working on their own. — tap to centre the map on it
Reasoning will generalize the way instruction tuning did: add enough verifiable domains and, at some point nobody can yet locate, the rest start working on their own.
Last stated 2 years ago
3 Feb 2025
NL
Nathan Lambert — holds since 2025-02-03 — tap for who they are
Same subject: Given good data, room to operate and sound numerical conditioning, models will learn on their own. — tap to centre the map on it
Given good data, room to operate and sound numerical conditioning, models will learn on their own.
Last stated 3 years ago
8 Aug 2023
DA
Dario Amodei — holds since 2023-08-08 — tap for who they are
Same subject: Instability in small-scale RL training does not necessarily mean the algorithm is wrong; the scale may be too small to see its benefits. — tap to centre the map on it
Instability in small-scale RL training does not necessarily mean the algorithm is wrong; the scale may be too small to see its benefits.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: A large context window does not replace fine-tuning, because fine-tuning serves the separate purpose of instilling preferences and demonstrating a task. — tap to centre the map on it
A large context window does not replace fine-tuning, because fine-tuning serves the separate purpose of instilling preferences and demonstrating a task.
Last stated 3 years ago
22 Mar 2024
EG
Elad Gil — holds since 2024-03-22 — tap for who they are
Same subject: Fine-tuning-based alignment does not remove a model's worrying knowledge and abilities; it only teaches the model not to output them. — tap to centre the map on it
Fine-tuning-based alignment does not remove a model's worrying knowledge and abilities; it only teaches the model not to output them.
Last stated 3 years ago
8 Aug 2023
DA
Dario Amodei — holds since 2023-08-08 — tap for who they are
Same subject: Models fine-tuned to imitate another model's outputs copy its style without acquiring its factual accuracy. — tap to centre the map on it
Models fine-tuned to imitate another model's outputs copy its style without acquiring its factual accuracy.
Last stated 3 years ago
15 Nov 2023
SR
Sebastian Ruder — holds since 2023-11-15 — tap for who they are
Same subject: A browser works better as a single window than as multiple separate windows. — tap to centre the map on it
A browser works better as a single window than as multiple separate windows.
Last stated 2 months ago
15 Sept 2026
CC
Chris Coyier — holds since 2026-09-15 — tap for who they are
Same subject: A business is more likely to fail from taking on too much complexity than from lacking resources. — tap to centre the map on it
A business is more likely to fail from taking on too much complexity than from lacking resources.
Last stated 8 years ago
29 Sept 2018
TG
Tren Griffin — holds since 2018-09-29 — tap for who they are
Same subject: A business succeeds through a small number of highly effective distribution channels, not many mediocre ones. — tap to centre the map on it
A business succeeds through a small number of highly effective distribution channels, not many mediocre ones.
Last stated 3 years ago
17 Feb 2024
AW
Adam Wiggins — holds since 2024-02-17 — tap for who they are
Same subject: A PHP vulnerability found with GPT-5.6 Sol will cause havoc this week. — tap to centre the map on it
A PHP vulnerability found with GPT-5.6 Sol will cause havoc this week.
Last stated a month ago
21 Oct 2026
PS
Peter Steinberger — holds since 2026-10-21 — tap for who they are
Same subject: A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests. — tap to centre the map on it
A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests.
Last stated 3 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers. — tap to centre the map on it
Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers.
Last stated 3 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: "Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next. — tap to centre the map on it
"Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next.
Last stated 12 months ago
25 Nov 2025
IS
Ilya Sutskever — holds since 2025-11-25 — tap for who they are
Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Last stated 5 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: A CPU and memory crunch is coming. — tap to centre the map on it
A CPU and memory crunch is coming.
Last stated 4 weeks ago
26 Oct 2026
DM
Dillon Mulroy — holds since 2026-10-26 — tap for who they are
same subject or similar wording a cloud: claims about one subject, named for it bar: when it was last stated, on a scale from 2015 to today — full is today a face: someone who holds the claim — tap it for who they are
At the centre
In instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset.
Last stated 15 Nov 2023 · 3 years ago
Holds Sebastian Ruder
Read this korrent →
Similar wording
The argument that RAG and long context make fine-tuning unnecessary holds when information is missing, but not when the model struggles with the task itself.
Last stated 5 Nov 2026 · 2 weeks ago
Holds Aishwarya Naresh Reganti
Similar wording
When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once.
Last stated 1 Dec 2023 · 3 years ago
Holds Sebastian Ruder
Similar wording
Reasoning will generalize the way instruction tuning did: add enough verifiable domains and, at some point nobody can yet locate, the rest start working on their own.
Last stated 3 Feb 2025 · 2 years ago
Holds Nathan Lambert
Similar wording
Given good data, room to operate and sound numerical conditioning, models will learn on their own.
Last stated 8 Aug 2023 · 3 years ago
Holds Dario Amodei
Similar wording
Instability in small-scale RL training does not necessarily mean the algorithm is wrong; the scale may be too small to see its benefits.
Last stated 10 Oct 2026 · a month ago
Holds Sebastian Raschka
Similar wording
A large context window does not replace fine-tuning, because fine-tuning serves the separate purpose of instilling preferences and demonstrating a task.
Last stated 22 Mar 2024 · 3 years ago
Holds Elad Gil
Similar wording
Fine-tuning-based alignment does not remove a model's worrying knowledge and abilities; it only teaches the model not to output them.
Last stated 8 Aug 2023 · 3 years ago
Holds Dario Amodei
Similar wording
Models fine-tuned to imitate another model's outputs copy its style without acquiring its factual accuracy.
Last stated 15 Nov 2023 · 3 years ago
Holds Sebastian Ruder
Same subject: simplicity
A browser works better as a single window than as multiple separate windows.
Last stated 15 Sept 2026 · 2 months ago
Holds Chris Coyier
Same subject: simplicity
A business is more likely to fail from taking on too much complexity than from lacking resources.
Last stated 29 Sept 2018 · 8 years ago
Holds Tren Griffin
Same subject: simplicity
A business succeeds through a small number of highly effective distribution channels, not many mediocre ones.
Last stated 17 Feb 2024 · 3 years ago
Holds Adam Wiggins
Same subject: OpenAI
A PHP vulnerability found with GPT-5.6 Sol will cause havoc this week.
Last stated 21 Oct 2026 · a month ago
Holds Peter Steinberger
Same subject: OpenAI
A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests.
Last stated 18 Mar 2024 · 3 years ago
Holds Sam Altman
Same subject: OpenAI
Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers.
Last stated 18 Mar 2024 · 3 years ago
Holds Sam Altman
Same subject: scaling laws
"Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next.
Last stated 25 Nov 2025 · 12 months ago
Holds Ilya Sutskever
Same subject: scaling laws
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Last stated 26 Jun 2026 · 5 months ago
Holds Noam Brown
Same subject: scaling laws
A CPU and memory crunch is coming.
Last stated 26 Oct 2026 · 4 weeks ago
Holds Dillon Mulroy