Tap a claim on the ring to put it at the centre.
← As models keep growing, AI is running out of enough high-quality unique training tokens to keep up.
17 connected korrents · 15 moments from 10 Dec 2015 to 27 Sept 2026.
Everything filed under LLMs
LLMs
Everything filed under neural networks
neural networks
Everything filed under OpenAI
OpenAI
Everything filed under scaling laws
scaling laws
Everything filed under social media
social media
Everything filed under transformers
transformers Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: As models keep growing, AI is running out of enough high-quality unique training tokens to keep up. As models keep growing, AI is running out of enough high-quality unique training tokens to keep up. Last stated 3 months ago
24 Jun 2026
LW
Lilian Weng — holds since 2026-06-24 — tap for who they are
Same subject: As AI models keep scaling up, the value of any single training source keeps shrinking. — tap to centre the map on it As AI models keep scaling up, the value of any single training source keeps shrinking. Last stated 4 weeks ago
7 Sept 2026
KK
Kevin Kelly — holds since 2026-09-07 — tap for who they are
Same subject: If nobody creates content any more because everyone just asks an AI, there is nothing left for AI to train on. — tap to centre the map on it If nobody creates content any more because everyone just asks an AI, there is nothing left for AI to train on. Last stated 2 months ago
8 Aug 2026
PL
Pieter Levels — holds since 2026-08-08 — tap for who they are
Same subject: More AI models that deliver top-tier performance at lower cost are needed. — tap to centre the map on it More AI models that deliver top-tier performance at lower cost are needed. Last stated 3 weeks ago
10 Sept 2026
KD
Kent C. Dodds — holds since 2026-09-10 — tap for who they are
Same subject: AI will not run out of data, because there is already enough real-world data to build the simulators that generate more of it. — tap to centre the map on it AI will not run out of data, because there is already enough real-world data to build the simulators that generate more of it. Last stated a year ago
23 Jul 2025
DH
Demis Hassabis — holds since 2025-07-23 — tap for who they are
Same subject: Available online text data is becoming a limiting factor for AI training, and repeating it yields diminishing returns. — tap to centre the map on it Available online text data is becoming a limiting factor for AI training, and repeating it yields diminishing returns. Last stated 3 years ago
19 Dec 2023
SR
Sebastian Ruder — holds since 2023-12-19 — tap for who they are
Same subject: Token prices have stayed flat because labs deliberately kept models smaller than expected in order to get more experimental cycles. — tap to centre the map on it Token prices have stayed flat because labs deliberately kept models smaller than expected in order to get more experimental cycles. Last stated 2 months ago
11 Aug 2026
RG
Ryan Greenblatt — holds since 2026-08-11 — tap for who they are
Same subject: The rapid advancement of AI models means that the best model in the world will be quickly surpassed by competitors. — tap to centre the map on it The rapid advancement of AI models means that the best model in the world will be quickly surpassed by competitors. Last stated 5 days ago
27 Sept 2026
SW
Simon Willison — holds since 2026-09-27 — tap for who they are
Same subject: Platforms will lose the ability to detect AI-generated content as the models improve, and should say how confident they are rather than pretend. — tap to centre the map on it Platforms will lose the ability to detect AI-generated content as the models improve, and should say how confident they are rather than pretend. Last stated 3 months ago
9 Jul 2026
AM
Adam Mosseri — holds since 2026-07-09 — tap for who they are
Same subject: A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable. — tap to centre the map on it A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable. Last stated 11 years ago
10 Dec 2015
JS
Jian Sun — holds since 2015-12-10 — tap for who they are
KH
Kaiming He — holds since 2015-12-10 — tap for who they are
Same subject: A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it. — tap to centre the map on it A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it. Last stated 9 years ago
9 Mar 2018
MC
Michael Carbin — holds since 2018-03-09 — tap for who they are
JF
Jonathan Frankle — holds since 2018-03-09 — tap for who they are
Same subject: A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. — tap to centre the map on it A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
Same subject: Self-attention can yield more interpretable models. — tap to centre the map on it Self-attention can yield more interpretable models. Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
Same subject: Sequence transduction can use a simple architecture based solely on attention, without recurrence or convolutions. — tap to centre the map on it Sequence transduction can use a simple architecture based solely on attention, without recurrence or convolutions. Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
Same subject: Looped/recurrent-depth transformer architectures can improve model quality at a fixed compute budget when the model is large enough. — tap to centre the map on it Looped/recurrent-depth transformer architectures can improve model quality at a fixed compute budget when the model is large enough. Last stated 3 weeks ago
9 Sept 2026
SR
Sebastian Raschka — holds since 2026-09-09 — tap for who they are
Same subject: A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests. — tap to centre the map on it A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests. Last stated 3 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: A technique that lets an AI model's reasoning shift outside its visible Chain of Thought is dangerous, both because it works and because a leading lab is willing to deploy it. — tap to centre the map on it A technique that lets an AI model's reasoning shift outside its visible Chain of Thought is dangerous, both because it works and because a leading lab is willing to deploy it. Last stated 4 weeks ago
3 Sept 2026
ZM
Zvi Mowshowitz — holds since 2026-09-03 — tap for who they are
Same subject: Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers. — tap to centre the map on it Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers. Last stated 3 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
At the centre
As models keep growing, AI is running out of enough high-quality unique training tokens to keep up.
Holds Read this korrent →
As AI models keep scaling up, the value of any single training source keeps shrinking.
Holds
If nobody creates content any more because everyone just asks an AI, there is nothing left for AI to train on.
Holds
More AI models that deliver top-tier performance at lower cost are needed.
Holds
AI will not run out of data, because there is already enough real-world data to build the simulators that generate more of it.
Holds
Available online text data is becoming a limiting factor for AI training, and repeating it yields diminishing returns.
Holds
Token prices have stayed flat because labs deliberately kept models smaller than expected in order to get more experimental cycles.
Holds
The rapid advancement of AI models means that the best model in the world will be quickly surpassed by competitors.
Holds
Platforms will lose the ability to detect AI-generated content as the models improve, and should say how confident they are rather than pretend.
Holds
A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable.
Holds
A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it.
Holds
A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.
Holds
Self-attention can yield more interpretable models.
Holds
Sequence transduction can use a simple architecture based solely on attention, without recurrence or convolutions.
Holds
Looped/recurrent-depth transformer architectures can improve model quality at a fixed compute budget when the model is large enough.
Holds
A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests.
Holds
A technique that lets an AI model's reasoning shift outside its visible Chain of Thought is dangerous, both because it works and because a leading lab is willing to deploy it.
Holds
Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers.
Holds