Tap a claim on the ring to put it at the centre.
← In instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset.
15 connected korrents · 14 moments from 11 Jun 2021 to 27 Sept 2026.
Everything filed under LLMs
LLMs
Everything filed under AI writing
AI writing
Everything filed under OpenAI
OpenAI
Everything filed under scaling laws
scaling laws
Everything filed under benchmarks
benchmarks
Everything filed under AGI
AGI Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: In instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset. In instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset. Last stated 3 years ago
15 Nov 2023
SR
Sebastian Ruder — holds since 2023-11-15 — tap for who they are
Same subject: When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once. — tap to centre the map on it When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once. Last stated 3 years ago
1 Dec 2023
SR
Sebastian Ruder — holds since 2023-12-01 — tap for who they are
Same subject: Reasoning will generalize the way instruction tuning did: add enough verifiable domains and, at some point nobody can yet locate, the rest start working on their own. — tap to centre the map on it Reasoning will generalize the way instruction tuning did: add enough verifiable domains and, at some point nobody can yet locate, the rest start working on their own. Last stated 2 years ago
3 Feb 2025
NL
Nathan Lambert — holds since 2025-02-03 — tap for who they are
Same subject: Models fine-tuned to imitate another model's outputs copy its style without acquiring its factual accuracy. — tap to centre the map on it Models fine-tuned to imitate another model's outputs copy its style without acquiring its factual accuracy. Last stated 3 years ago
15 Nov 2023
SR
Sebastian Ruder — holds since 2023-11-15 — tap for who they are
Same subject: Impressive-looking benchmark improvements often result from comparing against a poorly-tuned baseline rather than a genuine advance. — tap to centre the map on it Impressive-looking benchmark improvements often result from comparing against a poorly-tuned baseline rather than a genuine advance. Last stated 2 years ago
9 Mar 2025
NE
Nelson Elhage — holds since 2025-03-09 — tap for who they are
Same subject: What sits in a model's context is far clearer to it than its training data, which is trillions of tokens stirred into a soup of parameters. — tap to centre the map on it What sits in a model's context is far clearer to it than its training data, which is trillions of tokens stirred into a soup of parameters. Last stated 2 months ago
30 Jul 2026
JD
Jeff Dean — holds since 2026-07-30 — tap for who they are
Same subject: Intelligence is the execution of small, simplicity-weighted programs found by search over smooth loss landscapes in overparameterized networks, approaching Bayes-optimal reasoning as scale increases. — tap to centre the map on it Intelligence is the execution of small, simplicity-weighted programs found by search over smooth loss landscapes in overparameterized networks, approaching Bayes-optimal reasoning as scale increases. Last stated 5 years ago
11 Jun 2021
GW
Gwern — holds since 2021-06-11 — tap for who they are
Same subject: "Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next. — tap to centre the map on it "Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next. Last stated 10 months ago
25 Nov 2025
IS
Ilya Sutskever — holds since 2025-11-25 — tap for who they are
Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. Last stated 3 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: A lab should spend most of its compute on research rather than on building the next model, because research is where the tenfold yearly efficiency gains come from. — tap to centre the map on it A lab should spend most of its compute on research rather than on building the next model, because research is where the tenfold yearly efficiency gains come from. Last stated 7 months ago
13 Mar 2026
DP
Dylan Patel — holds since 2026-03-13 — tap for who they are
Same subject: A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests. — tap to centre the map on it A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests. Last stated 3 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: A technique that lets an AI model's reasoning shift outside its visible Chain of Thought is dangerous, both because it works and because a leading lab is willing to deploy it. — tap to centre the map on it A technique that lets an AI model's reasoning shift outside its visible Chain of Thought is dangerous, both because it works and because a leading lab is willing to deploy it. Last stated 4 weeks ago
3 Sept 2026
ZM
Zvi Mowshowitz — holds since 2026-09-03 — tap for who they are
Same subject: Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers. — tap to centre the map on it Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers. Last stated 3 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: A CEO who sends out an AI-written strategy memo is modelling that it is fine to outsource thinking and strategy. — tap to centre the map on it A CEO who sends out an AI-written strategy memo is modelling that it is fine to outsource thinking and strategy. Last stated 5 days ago
27 Sept 2026
MG
Molly Graham — holds since 2026-09-27 — tap for who they are
Same subject: A stream of AI-written papers with any error rate at all becomes insufferable, because finding the error costs more than the paper is worth even at ninety-nine percent. — tap to centre the map on it A stream of AI-written papers with any error rate at all becomes insufferable, because finding the error costs more than the paper is worth even at ninety-nine percent. Last stated 3 months ago
30 Jun 2026
GS
Grant Sanderson — holds since 2026-06-30 — tap for who they are
Same subject: AI can now produce in minutes work that used to take weeks to build. — tap to centre the map on it AI can now produce in minutes work that used to take weeks to build. Last stated 4 months ago
26 May 2026
CB
Carlos Alexandro Becker — holds since 2026-05-26 — tap for who they are
At the centre
In instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset.
Holds Read this korrent →
When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once.
Holds
Reasoning will generalize the way instruction tuning did: add enough verifiable domains and, at some point nobody can yet locate, the rest start working on their own.
Holds
Models fine-tuned to imitate another model's outputs copy its style without acquiring its factual accuracy.
Holds
Impressive-looking benchmark improvements often result from comparing against a poorly-tuned baseline rather than a genuine advance.
Holds
What sits in a model's context is far clearer to it than its training data, which is trillions of tokens stirred into a soup of parameters.
Holds
Intelligence is the execution of small, simplicity-weighted programs found by search over smooth loss landscapes in overparameterized networks, approaching Bayes-optimal reasoning as scale increases.
Holds
"Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next.
Holds
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Holds
A lab should spend most of its compute on research rather than on building the next model, because research is where the tenfold yearly efficiency gains come from.
Holds
A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests.
Holds
A technique that lets an AI model's reasoning shift outside its visible Chain of Thought is dangerous, both because it works and because a leading lab is willing to deploy it.
Holds
Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers.
Holds
A CEO who sends out an AI-written strategy memo is modelling that it is fine to outsource thinking and strategy.
Holds
A stream of AI-written papers with any error rate at all becomes insufferable, because finding the error costs more than the paper is worth even at ninety-nine percent.
Holds
AI can now produce in minutes work that used to take weeks to build.
Holds