Tap a claim on the ring to put it at the centre.
← Supervised pretraining on ImageNet is now obsolete for transfer…
17 connected korrents · 17 moments from 1 Jun 2009 to 18 Sept 2026.
Everything filed under neural networks
neural networks
Everything filed under scaling laws
scaling laws
Everything filed under pruning
pruning
Everything filed under LLMs
LLMs
Everything filed under GANs
GANs
Everything filed under computer vision
computer vision
Everything filed under transformers
transformers Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Read this korrent: Supervised pretraining on ImageNet is now obsolete for transfer learning, and ImageNet itself is just a contaminated validation set. Supervised pretraining on ImageNet is now obsolete for transfer learning, and ImageNet itself is just a contaminated validation set. Last stated 6 years ago
6 Apr 2021
GW
Gwern — holds since 2021-04-06 — tap for who they are
Same subject: We have almost no automated techniques for making a system transfer what it learns, and none of the few we have are used in modern deep learning. — tap to centre the map on it We have almost no automated techniques for making a system transfer what it learns, and none of the few we have are used in modern deep learning. Last stated a year ago
26 Sept 2025
RS
Richard Sutton — holds since 2025-09-26 — tap for who they are
Same subject: The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out. — tap to centre the map on it The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out. Last stated a year ago
17 Oct 2025
AK
Andrej Karpathy — holds since 2025-10-17 — tap for who they are
Same subject: An adversarial GAN loss should only supplement a larger unsupervised or semi-supervised generative model rather than stand alone as a primary approach. — tap to centre the map on it An adversarial GAN loss should only supplement a larger unsupervised or semi-supervised generative model rather than stand alone as a primary approach. Last stated 6 years ago
4 Feb 2021
GW
Gwern — holds since 2021-02-04 — tap for who they are
Same subject: ImageNet is much larger in scale and diversity and more accurate than existing image datasets. — tap to centre the map on it ImageNet is much larger in scale and diversity and more accurate than existing image datasets. Last stated 17 years ago
1 Jun 2009
FL
Fei-Fei Li — holds since 2009-06-01 — tap for who they are
Same subject: The internet is already past peak truth, and picking out training data worth learning from will only get harder. — tap to centre the map on it The internet is already past peak truth, and picking out training data worth learning from will only get harder. Last stated 5 months ago
13 May 2026
AH
Anders Hejlsberg — holds since 2026-05-13 — tap for who they are
Same subject: Unsupervised learning trained on unfiltered, internet-scale image data now genuinely works. — tap to centre the map on it Unsupervised learning trained on unfiltered, internet-scale image data now genuinely works. Last stated 6 years ago
6 Apr 2021
GW
Gwern — holds since 2021-04-06 — tap for who they are
Same subject: As models get better at following instructions, techniques like finetuning and constrained sampling for structured outputs will become less necessary — tap to centre the map on it As models get better at following instructions, techniques like finetuning and constrained sampling for structured outputs will become less necessary Last stated 3 years ago
16 Jan 2024
CH
Chip Huyen — holds since 2024-01-16 — tap for who they are
Same subject: Pre-trained representations reduce the need for many heavily-engineered task-specific architectures. — tap to centre the map on it Pre-trained representations reduce the need for many heavily-engineered task-specific architectures. Last stated 8 years ago
11 Oct 2018
KT
Kristina Toutanova — holds since 2018-10-11 — tap for who they are
MC
Ming-Wei Chang — holds since 2018-10-11 — tap for who they are
KL
Kenton Lee — holds since 2018-10-11 — tap for who they are
JD
Jacob Devlin — holds since 2018-10-11 — tap for who they are
Same subject: Batteries will replace transmission as the cheapest way to keep the lights on, and the grid will shrink rather than grow. — tap to centre the map on it Batteries will replace transmission as the cheapest way to keep the lights on, and the grid will shrink rather than grow. Last stated 10 months ago
8 Dec 2025
CH
Casey Handmer — holds since 2023-10-11 — tap for who they are
CH
Casey Handmer — holds since 2025-12-08 — tap for who they are
Same subject: Most agent harnesses need some pruning during context compaction anyway for cost reasons, which Jev might help with. — tap to centre the map on it Most agent harnesses need some pruning during context compaction anyway for cost reasons, which Jev might help with. Last stated a month ago
18 Sept 2026
AR
Armin Ronacher — holds since 2026-09-18 — tap for who they are
Same subject: Neuroplasticity is as much the removal of connections as the growth of new ones, so pruning is a form of learning rather than a form of decay. — tap to centre the map on it Neuroplasticity is as much the removal of connections as the growth of new ones, so pruning is a form of learning rather than a form of decay. Last stated 3 months ago
20 Jul 2026
TW
Tommy Wood — holds since 2026-07-20 — tap for who they are
Same subject: Sequence transduction can use a simple architecture based solely on attention, without recurrence or convolutions. — tap to centre the map on it Sequence transduction can use a simple architecture based solely on attention, without recurrence or convolutions. Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
Same subject: A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. — tap to centre the map on it A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
Same subject: Looped/recurrent-depth transformer architectures can improve model quality at a fixed compute budget when the model is large enough. — tap to centre the map on it Looped/recurrent-depth transformer architectures can improve model quality at a fixed compute budget when the model is large enough. Last stated a month ago
9 Sept 2026
SR
Sebastian Raschka — holds since 2026-09-09 — tap for who they are
Same subject: "Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next. — tap to centre the map on it "Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next. Last stated 11 months ago
25 Nov 2025
IS
Ilya Sutskever — holds since 2025-11-25 — tap for who they are
Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. Last stated 4 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: A lab should spend most of its compute on research rather than on building the next model, because research is where the tenfold yearly efficiency gains come from. — tap to centre the map on it A lab should spend most of its compute on research rather than on building the next model, because research is where the tenfold yearly efficiency gains come from. Last stated 7 months ago
13 Mar 2026
DP
Dylan Patel — holds since 2026-03-13 — tap for who they are
At the centre
Supervised pretraining on ImageNet is now obsolete for transfer learning, and ImageNet itself is just a contaminated validation set.
Holds Read this korrent →
We have almost no automated techniques for making a system transfer what it learns, and none of the few we have are used in modern deep learning.
Holds
The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.
Holds
An adversarial GAN loss should only supplement a larger unsupervised or semi-supervised generative model rather than stand alone as a primary approach.
Holds
ImageNet is much larger in scale and diversity and more accurate than existing image datasets.
Holds
The internet is already past peak truth, and picking out training data worth learning from will only get harder.
Holds
Unsupervised learning trained on unfiltered, internet-scale image data now genuinely works.
Holds
As models get better at following instructions, techniques like finetuning and constrained sampling for structured outputs will become less necessary
Holds
Pre-trained representations reduce the need for many heavily-engineered task-specific architectures.
Holds
Batteries will replace transmission as the cheapest way to keep the lights on, and the grid will shrink rather than grow.
Holds
Most agent harnesses need some pruning during context compaction anyway for cost reasons, which Jev might help with.
Holds
Neuroplasticity is as much the removal of connections as the growth of new ones, so pruning is a form of learning rather than a form of decay.
Holds
Sequence transduction can use a simple architecture based solely on attention, without recurrence or convolutions.
Holds
A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.
Holds
Looped/recurrent-depth transformer architectures can improve model quality at a fixed compute budget when the model is large enough.
Holds
"Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next.
Holds
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Holds
A lab should spend most of its compute on research rather than on building the next model, because research is where the tenfold yearly efficiency gains come from.
Holds