Tap a claim on the ring to put it at the centre.
← The Bitter Lesson is misread as being about scale; it is about…
17 connected korrents · 13 moments from 10 Dec 2015 to 9 Sept 2026.
Everything filed under neural networks
neural networks
Everything filed under scaling laws
scaling laws
Everything filed under design
design
Everything filed under LLMs
LLMs
Everything filed under mathematics
mathematics
Everything filed under AI alignment
AI alignment
Everything filed under transformers
transformers Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: The Bitter Lesson is misread as being about scale; it is about refusing to add human priors, because a clever specific fix always loses to a simple scalable one. The Bitter Lesson is misread as being about scale; it is about refusing to add human priors, because a clever specific fix always loses to a simple scalable one. Last stated 2 years ago
3 Feb 2025
NL
Nathan Lambert — holds since 2025-02-03 — tap for who they are
Same subject: The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out. — tap to centre the map on it The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out. Last stated a year ago
17 Oct 2025
AK
Andrej Karpathy — holds since 2025-10-17 — tap for who they are
Same subject: Generative AI has failed to produce the learning gains it was promised to deliver and instead shows evidence of harming learning. — tap to centre the map on it Generative AI has failed to produce the learning gains it was promised to deliver and instead shows evidence of harming learning. Last stated 4 months ago
24 Jun 2026
DM
Dan Meyer — holds since 2026-06-24 — tap for who they are
Same subject: Bigger is smarter, always: bringing your special domain knowledge to the model is the wrong side of the bitter lesson. — tap to centre the map on it Bigger is smarter, always: bringing your special domain knowledge to the model is the wrong side of the bitter lesson. Last stated 7 months ago
11 Mar 2026
SY
Steve Yegge — holds since 2026-03-11 — tap for who they are
Same subject: The bitter lesson is only an empirical observation about a particular seventy years of history, and need not hold for the next seventy. — tap to centre the map on it The bitter lesson is only an empirical observation about a particular seventy years of history, and need not hold for the next seventy. Last stated a year ago
26 Sept 2025
RS
Richard Sutton — holds since 2025-09-26 — tap for who they are
Same subject: Large language models start in exactly the wrong place, because they try to get by without a goal and therefore without any sense of better or worse. — tap to centre the map on it Large language models start in exactly the wrong place, because they try to get by without a goal and therefore without any sense of better or worse. Last stated a year ago
26 Sept 2025
RS
Richard Sutton — holds since 2025-09-26 — tap for who they are
Same subject: That people are good at maths and coding, which evolution never selected for, shows human learning is not a complicated evolutionary prior but something more fundamental. — tap to centre the map on it That people are good at maths and coding, which evolution never selected for, shows human learning is not a complicated evolutionary prior but something more fundamental. Last stated 11 months ago
25 Nov 2025
IS
Ilya Sutskever — holds since 2025-11-25 — tap for who they are
Same subject: Very capable AI will be harder to align than current systems, because the loop of spotting a bad behaviour and patching the training that caused it breaks down. — tap to centre the map on it Very capable AI will be harder to align than current systems, because the loop of spotting a bad behaviour and patching the training that caused it breaks down. Last stated 2 months ago
11 Aug 2026
RG
Ryan Greenblatt — holds since 2026-08-11 — tap for who they are
Same subject: Pre-training is not the process by which humans learn; it sits somewhere between human learning and human evolution. — tap to centre the map on it Pre-training is not the process by which humans learn; it sits somewhere between human learning and human evolution. Last stated 8 months ago
13 Feb 2026
DA
Dario Amodei — holds since 2026-02-13 — tap for who they are
Same subject: A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable. — tap to centre the map on it A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable. Last stated 11 years ago
10 Dec 2015
JS
Jian Sun — holds since 2015-12-10 — tap for who they are
KH
Kaiming He — holds since 2015-12-10 — tap for who they are
Same subject: A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it. — tap to centre the map on it A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it. Last stated 9 years ago
9 Mar 2018
MC
Michael Carbin — holds since 2018-03-09 — tap for who they are
JF
Jonathan Frankle — holds since 2018-03-09 — tap for who they are
Same subject: A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. — tap to centre the map on it A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
Same subject: A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work. — tap to centre the map on it A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work. Last stated 9 years ago
9 Mar 2018
MC
Michael Carbin — holds since 2018-03-09 — tap for who they are
JF
Jonathan Frankle — holds since 2018-03-09 — tap for who they are
Same subject: Intelligence is the execution of small, simplicity-weighted programs found by search over smooth loss landscapes in overparameterized networks, approaching Bayes-optimal reasoning as scale increases. — tap to centre the map on it Intelligence is the execution of small, simplicity-weighted programs found by search over smooth loss landscapes in overparameterized networks, approaching Bayes-optimal reasoning as scale increases. Last stated 5 years ago
11 Jun 2021
GW
Gwern — holds since 2021-06-11 — tap for who they are
Same subject: Pruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free. — tap to centre the map on it Pruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free. Last stated 9 years ago
9 Mar 2018
MC
Michael Carbin — holds since 2018-03-09 — tap for who they are
JF
Jonathan Frankle — holds since 2018-03-09 — tap for who they are
Same subject: Self-attention can yield more interpretable models. — tap to centre the map on it Self-attention can yield more interpretable models. Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
Same subject: Sequence transduction can use a simple architecture based solely on attention, without recurrence or convolutions. — tap to centre the map on it Sequence transduction can use a simple architecture based solely on attention, without recurrence or convolutions. Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
Same subject: Looped/recurrent-depth transformer architectures can improve model quality at a fixed compute budget when the model is large enough. — tap to centre the map on it Looped/recurrent-depth transformer architectures can improve model quality at a fixed compute budget when the model is large enough. Last stated a month ago
9 Sept 2026
SR
Sebastian Raschka — holds since 2026-09-09 — tap for who they are
At the centre
The Bitter Lesson is misread as being about scale; it is about refusing to add human priors, because a clever specific fix always loses to a simple scalable one.
Holds Read this korrent →
The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.
Holds
Generative AI has failed to produce the learning gains it was promised to deliver and instead shows evidence of harming learning.
Holds
Bigger is smarter, always: bringing your special domain knowledge to the model is the wrong side of the bitter lesson.
Holds
The bitter lesson is only an empirical observation about a particular seventy years of history, and need not hold for the next seventy.
Holds
Large language models start in exactly the wrong place, because they try to get by without a goal and therefore without any sense of better or worse.
Holds
That people are good at maths and coding, which evolution never selected for, shows human learning is not a complicated evolutionary prior but something more fundamental.
Holds
Very capable AI will be harder to align than current systems, because the loop of spotting a bad behaviour and patching the training that caused it breaks down.
Holds
Pre-training is not the process by which humans learn; it sits somewhere between human learning and human evolution.
Holds
A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable.
Holds
A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it.
Holds
A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.
Holds
A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work.
Holds
Intelligence is the execution of small, simplicity-weighted programs found by search over smooth loss landscapes in overparameterized networks, approaching Bayes-optimal reasoning as scale increases.
Holds
Pruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free.
Holds
Self-attention can yield more interpretable models.
Holds
Sequence transduction can use a simple architecture based solely on attention, without recurrence or convolutions.
Holds
Looped/recurrent-depth transformer architectures can improve model quality at a fixed compute budget when the model is large enough.
Holds