Tap a claim on the ring to put it at the centre.
← Scaling up next-token prediction based on web text data is hitting a wall.
17 connected korrents · 15 moments from 18 Dec 2020 to 21 Nov 2026. Nearly all of them are about LLMs .
Everything filed under scaling laws
scaling laws
Everything filed under open source
open source
Everything filed under OpenAI
OpenAI
Everything filed under reinforcement learning
reinforcement learning
Everything filed under transformers
transformers
Everything filed under AI alignment
AI alignment
Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject
Read this korrent: Scaling up next-token prediction based on web text data is hitting a wall.
Scaling up next-token prediction based on web text data is hitting a wall.
Last stated 2 days ago
20 Nov 2026
FC
François Chollet — holds since 2026-11-20 — tap for who they are
Same subject: Available online text data is becoming a limiting factor for AI training, and repeating it yields diminishing returns. — tap to centre the map on it
Available online text data is becoming a limiting factor for AI training, and repeating it yields diminishing returns.
Last stated 3 years ago
19 Dec 2023
SR
Sebastian Ruder — holds since 2023-12-19 — tap for who they are
Same subject: Until AI learns from something richer than text, its limited data will leave it struggling with the richness of language. — tap to centre the map on it
Until AI learns from something richer than text, its limited data will leave it struggling with the richness of language.
Last stated 6 years ago
18 Dec 2020
MS
Marcus du Sautoy — holds since 2020-12-18 — tap for who they are
Same subject: The current AI stack is not among the top 50% most efficient approaches to learning. — tap to centre the map on it
The current AI stack is not among the top 50% most efficient approaches to learning.
Last stated yesterday
21 Nov 2026
FC
François Chollet — holds since 2026-11-21 — tap for who they are
Same subject: Large language models could still plateau, and that possibility should be held open even though no evidence of it has appeared. — tap to centre the map on it
Large language models could still plateau, and that possibility should be held open even though no evidence of it has appeared.
Last stated 3 months ago
26 Aug 2026
DH
David Heinemeier Hansson — holds since 2026-08-26 — tap for who they are
Same subject: As models keep growing, AI is running out of enough high-quality unique training tokens to keep up. — tap to centre the map on it
As models keep growing, AI is running out of enough high-quality unique training tokens to keep up.
Last stated 5 months ago
24 Jun 2026
LW
Lilian Weng — holds since 2026-06-24 — tap for who they are
Same subject: The next-token language modeling objective is misaligned with following user instructions helpfully and safely. — tap to centre the map on it
The next-token language modeling objective is misaligned with following user instructions helpfully and safely.
Last stated 5 years ago
4 Mar 2022
JL
Jan Leike — holds since 2022-03-04 — tap for who they are
Same subject: At the level of loss functions, language-model pre-training is remarkably simple, though running it at scale is a complete engineering feat. — tap to centre the map on it
At the level of loss functions, language-model pre-training is remarkably simple, though running it at scale is a complete engineering feat.
Last stated 4 months ago
16 Jul 2026
GS
Grant Sanderson — holds since 2026-07-16 — tap for who they are
Same subject: By this time next year there will be significantly more tokens flowing, making token efficiency very important. — tap to centre the map on it
By this time next year there will be significantly more tokens flowing, making token efficiency very important.
Last stated a month ago
11 Oct 2026
LR
Lee Robinson — holds since 2026-10-11 — tap for who they are
Same subject: A PHP vulnerability found with GPT-5.6 Sol will cause havoc this week. — tap to centre the map on it
A PHP vulnerability found with GPT-5.6 Sol will cause havoc this week.
Last stated a month ago
21 Oct 2026
PS
Peter Steinberger — holds since 2026-10-21 — tap for who they are
Same subject: A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests. — tap to centre the map on it
A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests.
Last stated 3 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers. — tap to centre the map on it
Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers.
Last stated 3 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: A business making hundreds of millions from an open source project owes it money or contribution, and can be refused its trademarks if it gives neither. — tap to centre the map on it
A business making hundreds of millions from an open source project owes it money or contribution, and can be refused its trademarks if it gives neither.
Last stated 2 years ago
26 Sept 2024
MM
Matt Mullenweg — holds since 2024-09-26 — tap for who they are
Same subject: A codec does not really exist until everyone can decode it. — tap to centre the map on it
A codec does not really exist until everyone can decode it.
Last stated 6 months ago
31 May 2026
JK
Jean-Baptiste Kempf — holds since 2026-05-31 — tap for who they are
Same subject: A company that will not accept a project’s terms is free to go and use a more permissive project instead. — tap to centre the map on it
A company that will not accept a project’s terms is free to go and use a more permissive project instead.
Last stated 2 years ago
26 Sept 2024
MM
Matt Mullenweg — holds since 2024-09-26 — tap for who they are
Same subject: A feedback loop that reinforces a behavior pulls it toward whatever improves the loop's own score. — tap to centre the map on it
A feedback loop that reinforces a behavior pulls it toward whatever improves the loop's own score.
Last stated 3 months ago
26 Aug 2026
HK
Henrik Karlsson — holds since 2026-08-26 — tap for who they are
Same subject: A format reward should only be added when the answer is already correct. — tap to centre the map on it
A format reward should only be added when the answer is already correct.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: A model should not be called a reasoning model unless it has been trained with reinforcement learning with verifiable rewards. — tap to centre the map on it
A model should not be called a reasoning model unless it has been trained with reinforcement learning with verifiable rewards.
Last stated 2 months ago
3 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-03 — tap for who they are
same subject or similar wording a cloud: claims about one subject, named for it bar: when it was last stated, on a scale from 2015 to today — full is today a face: someone who holds the claim — tap it for who they are
At the centre
Scaling up next-token prediction based on web text data is hitting a wall.
Last stated 20 Nov 2026 · 2 days ago
Holds François Chollet
Read this korrent →
Similar wording
Available online text data is becoming a limiting factor for AI training, and repeating it yields diminishing returns.
Last stated 19 Dec 2023 · 3 years ago
Holds Sebastian Ruder
Similar wording
Until AI learns from something richer than text, its limited data will leave it struggling with the richness of language.
Last stated 18 Dec 2020 · 6 years ago
Holds Marcus du Sautoy
Similar wording
The current AI stack is not among the top 50% most efficient approaches to learning.
Last stated 21 Nov 2026 · yesterday
Holds François Chollet
Similar wording
Large language models could still plateau, and that possibility should be held open even though no evidence of it has appeared.
Last stated 26 Aug 2026 · 3 months ago
Holds David Heinemeier Hansson
Similar wording
As models keep growing, AI is running out of enough high-quality unique training tokens to keep up.
Last stated 24 Jun 2026 · 5 months ago
Holds Lilian Weng
Similar wording
The next-token language modeling objective is misaligned with following user instructions helpfully and safely.
Last stated 4 Mar 2022 · 5 years ago
Holds Jan Leike
Similar wording
At the level of loss functions, language-model pre-training is remarkably simple, though running it at scale is a complete engineering feat.
Last stated 16 Jul 2026 · 4 months ago
Holds Grant Sanderson
Similar wording
By this time next year there will be significantly more tokens flowing, making token efficiency very important.
Last stated 11 Oct 2026 · a month ago
Holds Lee Robinson
Same subject: OpenAI
A PHP vulnerability found with GPT-5.6 Sol will cause havoc this week.
Last stated 21 Oct 2026 · a month ago
Holds Peter Steinberger
Same subject: OpenAI
A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests.
Last stated 18 Mar 2024 · 3 years ago
Holds Sam Altman
Same subject: OpenAI
Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers.
Last stated 18 Mar 2024 · 3 years ago
Holds Sam Altman
Same subject: open source
A business making hundreds of millions from an open source project owes it money or contribution, and can be refused its trademarks if it gives neither.
Last stated 26 Sept 2024 · 2 years ago
Holds Matt Mullenweg
Same subject: open source
A codec does not really exist until everyone can decode it.
Last stated 31 May 2026 · 6 months ago
Holds Jean-Baptiste Kempf
Same subject: open source
A company that will not accept a project’s terms is free to go and use a more permissive project instead.
Last stated 26 Sept 2024 · 2 years ago
Holds Matt Mullenweg
Same subject: reinforcement learning
A feedback loop that reinforces a behavior pulls it toward whatever improves the loop's own score.
Last stated 26 Aug 2026 · 3 months ago
Holds Henrik Karlsson
Same subject: reinforcement learning
A format reward should only be added when the answer is already correct.
Last stated 10 Oct 2026 · a month ago
Holds Sebastian Raschka
Same subject: reinforcement learning
A model should not be called a reasoning model unless it has been trained with reinforcement learning with verifiable rewards.
Last stated 3 Oct 2026 · 2 months ago
Holds Sebastian Raschka