Tap a claim on the ring to put it at the centre.
← The knowledge a model soaks up in pre-training is holding it back;…
17 connected korrents · 20 moments from 1 Jan 1998 to 21 Sept 2026. Nearly all of them are about neural networks .
Everything filed under LLMs
LLMs
Everything filed under reinforcement learning
reinforcement learning
Everything filed under open source
open source
Everything filed under data centers
data centers
Everything filed under AI and jobs
AI and jobs
Everything filed under computer vision
computer vision Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out. The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out. Last stated a year ago
17 Oct 2025
AK
Andrej Karpathy — holds since 2025-10-17 — tap for who they are
Same subject: A neural network's latent space is closer to an uncopyrightable syntax than to copyrightable expression. — tap to centre the map on it A neural network's latent space is closer to an uncopyrightable syntax than to copyrightable expression. Last stated 2 months ago
7 Sept 2026
KK
Kevin Kelly — holds since 2026-09-07 — tap for who they are
Same subject: A verifiable task can be optimised by reinforcement learning until a neural network performs it extremely well. — tap to centre the map on it A verifiable task can be optimised by reinforcement learning until a neural network performs it extremely well. Last stated 11 months ago
17 Nov 2025
AK
Andrej Karpathy — holds since 2025-11-17 — tap for who they are
Same subject: Access to large-scale generalist AI systems that could be weaponized should be limited, meaning their code and neural network parameters should not be released open-source. — tap to centre the map on it Access to large-scale generalist AI systems that could be weaponized should be limited, meaning their code and neural network parameters should not be released open-source. Last stated 3 years ago
24 Jun 2023
YB
Yoshua Bengio — holds since 2023-06-24 — tap for who they are
Same subject: AI is unlikely to be a hyperprofits engine for research labs whose attached companies make money serving neural-network calculations from datacenters. — tap to centre the map on it AI is unlikely to be a hyperprofits engine for research labs whose attached companies make money serving neural-network calculations from datacenters. Last stated a month ago
21 Sept 2026
BD
Brad DeLong — holds since 2026-09-21 — tap for who they are
Same subject: An agent can understand a maximally simplified explanation and still be unable to come up with it — that gap is what is left of the expert's job. — tap to centre the map on it An agent can understand a maximally simplified explanation and still be unable to come up with it — that gap is what is left of the expert's job. Last stated 7 months ago
20 Mar 2026
AK
Andrej Karpathy — holds since 2026-03-20 — tap for who they are
Same subject: Anything nature shaped can be learned efficiently by a classical neural network, because evolutionary processes leave structure behind. — tap to centre the map on it Anything nature shaped can be learned efficiently by a classical neural network, because evolutionary processes leave structure behind. Last stated a year ago
23 Jul 2025
DH
Demis Hassabis — holds since 2025-07-23 — tap for who they are
Same subject: Complete reverse-engineering of neural networks is a less promising research direction than studying model biology and applying interpretability usefully — tap to centre the map on it Complete reverse-engineering of neural networks is a less promising research direction than studying model biology and applying interpretability usefully Last stated a year ago
19 Aug 2025
NN
Neel Nanda — holds since 2025-08-19 — tap for who they are
Same subject: Convolutional neural networks are specifically designed to handle variability in two-dimensional shapes. — tap to centre the map on it Convolutional neural networks are specifically designed to handle variability in two-dimensional shapes. Last stated 29 years ago
1 Jan 1998
YB
Yoshua Bengio — holds since 1998-01-01 — tap for who they are
Same subject: AI progress is faster than people expect, and ordinary scaling can be enough to solve problems that looked very hard. — tap to centre the map on it AI progress is faster than people expect, and ordinary scaling can be enough to solve problems that looked very hard. Last stated 2 years ago
2 Feb 2025
AC
Ajeya Cotra — holds since 2023-08-29 — tap for who they are
SA
Scott Alexander — holds since 2022-09-12 — tap for who they are
MB
Miles Brundage — holds since 2025-02-02 — tap for who they are
Same subject: Current techniques restrict pre-trained representation power because standard language models are unidirectional. — tap to centre the map on it Current techniques restrict pre-trained representation power because standard language models are unidirectional. Last stated 8 years ago
11 Oct 2018
KT
Kristina Toutanova — holds since 2018-10-11 — tap for who they are
MC
Ming-Wei Chang — holds since 2018-10-11 — tap for who they are
KL
Kenton Lee — holds since 2018-10-11 — tap for who they are
JD
Jacob Devlin — holds since 2018-10-11 — tap for who they are
Same subject: Emergent abilities come from generalisation turning compositional, not merely from a training set containing a lot of stuff. — tap to centre the map on it Emergent abilities come from generalisation turning compositional, not merely from a training set containing a lot of stuff. Last stated a year ago
12 Sept 2025
SL
Sergey Levine — holds since 2025-09-12 — tap for who they are
Same subject: If reasoning can be separated from facts, the frontier labs lose their advantage, because nobody needs a million-GPU cluster any more. — tap to centre the map on it If reasoning can be separated from facts, the frontier labs lose their advantage, because nobody needs a million-GPU cluster any more. Last stated 2 years ago
19 Jun 2024
AS
Aravind Srinivas — holds since 2024-06-19 — tap for who they are
Same subject: Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores. — tap to centre the map on it Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores. Last stated 2 years ago
13 May 2024
SR
Sebastian Ruder — holds since 2024-05-13 — tap for who they are
Same subject: Large language models could still plateau, and that possibility should be held open even though no evidence of it has appeared. — tap to centre the map on it Large language models could still plateau, and that possibility should be held open even though no evidence of it has appeared. Last stated 2 months ago
26 Aug 2026
DH
David Heinemeier Hansson — holds since 2026-08-26 — tap for who they are
Same subject: As reasoning models and agents retain more tokens for longer, KV-cache size and attention cost become the main constraint on scaling context. — tap to centre the map on it As reasoning models and agents retain more tokens for longer, KV-cache size and attention cost become the main constraint on scaling context. Last stated 5 months ago
16 May 2026
SR
Sebastian Raschka — holds since 2026-05-16 — tap for who they are
Same subject: Context engineering has stayed relevant for a year because it is grounded in how transformer attention works, and it will matter to anyone building on AI until post-transformer models arrive. — tap to centre the map on it Context engineering has stayed relevant for a year because it is grounded in how transformer attention works, and it will matter to anyone building on AI until post-transformer models arrive. Last stated 3 months ago
15 Jul 2026
DH
Dex Horthy — holds since 2026-07-15 — tap for who they are
Same subject: Directly optimizing prompt embeddings (prefix-tuning) yields better performance than even a single optimized text prompt. — tap to centre the map on it Directly optimizing prompt embeddings (prefix-tuning) yields better performance than even a single optimized text prompt. Last stated 6 years ago
4 Feb 2021
GW
Gwern — holds since 2021-02-04 — tap for who they are
At the centre
The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.
Holds Read this korrent →
A neural network's latent space is closer to an uncopyrightable syntax than to copyrightable expression.
Holds
A verifiable task can be optimised by reinforcement learning until a neural network performs it extremely well.
Holds
Access to large-scale generalist AI systems that could be weaponized should be limited, meaning their code and neural network parameters should not be released open-source.
Holds
AI is unlikely to be a hyperprofits engine for research labs whose attached companies make money serving neural-network calculations from datacenters.
Holds
An agent can understand a maximally simplified explanation and still be unable to come up with it — that gap is what is left of the expert's job.
Holds
Anything nature shaped can be learned efficiently by a classical neural network, because evolutionary processes leave structure behind.
Holds
Complete reverse-engineering of neural networks is a less promising research direction than studying model biology and applying interpretability usefully
Holds
Convolutional neural networks are specifically designed to handle variability in two-dimensional shapes.
Holds
AI progress is faster than people expect, and ordinary scaling can be enough to solve problems that looked very hard.
Holds
Current techniques restrict pre-trained representation power because standard language models are unidirectional.
Holds
Emergent abilities come from generalisation turning compositional, not merely from a training set containing a lot of stuff.
Holds
If reasoning can be separated from facts, the frontier labs lose their advantage, because nobody needs a million-GPU cluster any more.
Holds
Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.
Holds
Large language models could still plateau, and that possibility should be held open even though no evidence of it has appeared.
Holds
As reasoning models and agents retain more tokens for longer, KV-cache size and attention cost become the main constraint on scaling context.
Holds
Context engineering has stayed relevant for a year because it is grounded in how transformer attention works, and it will matter to anyone building on AI until post-transformer models arrive.
Holds
Directly optimizing prompt embeddings (prefix-tuning) yields better performance than even a single optimized text prompt.
Holds