Tap a claim on the ring to put it at the centre.
← Training in fp16 required gradient scaling, which is annoying and an extra source of state and complexity.
17 connected korrents · 14 moments from 10 Dec 2015 to 10 Oct 2026.
Everything filed under neural networks
neural networks
Everything filed under LLMs
LLMs
Everything filed under simplicity
simplicity
Everything filed under software performance
software performance
Everything filed under scaling laws
scaling laws
Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject
Read this korrent: Training in fp16 required gradient scaling, which is annoying and an extra source of state and complexity.
Training in fp16 required gradient scaling, which is annoying and an extra source of state and complexity.
Last stated 2 years ago
9 Jun 2024
AK
Andrej Karpathy — holds since 2024-06-09 — tap for who they are
Same subject: PyTorch DDP's recommended no_sync pattern for gradient accumulation is super ugly, since it makes you copy-paste code into a context manager. — tap to centre the map on it
PyTorch DDP's recommended no_sync pattern for gradient accumulation is super ugly, since it makes you copy-paste code into a context manager.
Last stated 2 years ago
9 Jun 2024
AK
Andrej Karpathy — holds since 2024-06-09 — tap for who they are
Same subject: Extra softmax operations should be avoided in training code because softmax is expensive at scale. — tap to centre the map on it
Extra softmax operations should be avoided in training code because softmax is expensive at scale.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: Criticism that a simpler, scaled-up architecture underperforms a heavily hand-optimized baseline actually demonstrates why that architecture-scaling research matters. — tap to centre the map on it
Criticism that a simpler, scaled-up architecture underperforms a heavily hand-optimized baseline actually demonstrates why that architecture-scaling research matters.
Last stated 5 years ago
11 Jun 2021
GW
Gwern — holds since 2021-06-11 — tap for who they are
Same subject: The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out. — tap to centre the map on it
The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.
Last stated a year ago
17 Oct 2025
AK
Andrej Karpathy — holds since 2025-10-17 — tap for who they are
Same subject: Human feedback cannot be scaled up enough, so compute-heavy methods that rely on humans will need some scaled supervision method. — tap to centre the map on it
Human feedback cannot be scaled up enough, so compute-heavy methods that rely on humans will need some scaled supervision method.
Last stated 2 years ago
11 Nov 2024
DA
Dario Amodei — holds since 2024-11-11 — tap for who they are
Same subject: Language models should be thought of as somewhat out of date, because pre-training is too expensive to repeat often. — tap to centre the map on it
Language models should be thought of as somewhat out of date, because pre-training is too expensive to repeat often.
Last stated 2 years ago
27 Feb 2025
AK
Andrej Karpathy — holds since 2025-02-27 — tap for who they are
Same subject: Reinforcement learning cannot be scaled for robots the way it was for language models, because every attempt spends real robot hours instead of data-centre compute. — tap to centre the map on it
Reinforcement learning cannot be scaled for robots the way it was for language models, because every attempt spends real robot hours instead of data-centre compute.
Last stated 3 months ago
12 Aug 2026
CF
Chelsea Finn — holds since 2026-08-12 — tap for who they are
Same subject: At the level of loss functions, language-model pre-training is remarkably simple, though running it at scale is a complete engineering feat. — tap to centre the map on it
At the level of loss functions, language-model pre-training is remarkably simple, though running it at scale is a complete engineering feat.
Last stated 4 months ago
16 Jul 2026
GS
Grant Sanderson — holds since 2026-07-16 — tap for who they are
Same subject: A curve, which is what a deep learning model is, is a very bad substrate for discrete computation. — tap to centre the map on it
A curve, which is what a deep learning model is, is a very bad substrate for discrete computation.
Last stated 2 years ago
6 Nov 2024
FC
François Chollet — holds since 2024-11-06 — tap for who they are
Same subject: A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable. — tap to centre the map on it
A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable.
Last stated 11 years ago
10 Dec 2015
JS
Jian Sun — holds since 2015-12-10 — tap for who they are
KH
Kaiming He — holds since 2015-12-10 — tap for who they are
Same subject: A project-level agent memory file works like a neural net, steering the agent much as a fine-tuned LoRA steers a model. — tap to centre the map on it
A project-level agent memory file works like a neural net, steering the agent much as a fine-tuned LoRA steers a model.
Last stated 3 months ago
23 Aug 2026
KC
Kun Chen — holds since 2026-08-23 — tap for who they are
Same subject: A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it. — tap to centre the map on it
A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it.
Last stated 9 years ago
9 Mar 2018
MC
Michael Carbin — holds since 2018-03-09 — tap for who they are
JF
Jonathan Frankle — holds since 2018-03-09 — tap for who they are
Same subject: A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work. — tap to centre the map on it
A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work.
Last stated 9 years ago
9 Mar 2018
MC
Michael Carbin — holds since 2018-03-09 — tap for who they are
JF
Jonathan Frankle — holds since 2018-03-09 — tap for who they are
Same subject: Pruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free. — tap to centre the map on it
Pruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free.
Last stated 9 years ago
9 Mar 2018
MC
Michael Carbin — holds since 2018-03-09 — tap for who they are
JF
Jonathan Frankle — holds since 2018-03-09 — tap for who they are
Same subject: A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. — tap to centre the map on it
A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.
Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
Same subject: Linear probes are very strong classifiers, and attention-head probes are even stronger. — tap to centre the map on it
Linear probes are very strong classifiers, and attention-head probes are even stronger.
Last stated 11 months ago
20 Dec 2025
NN
Neel Nanda — holds since 2025-12-20 — tap for who they are
Same subject: Self-attention can yield more interpretable models. — tap to centre the map on it
Self-attention can yield more interpretable models.
Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
same subject or similar wording a cloud: claims about one subject, named for it bar: when it was last stated, on a scale from 2015 to today — full is today a face: someone who holds the claim — tap it for who they are
At the centre
Training in fp16 required gradient scaling, which is annoying and an extra source of state and complexity.
Last stated 9 Jun 2024 · 2 years ago
Holds Andrej Karpathy
Read this korrent →
Similar wording
PyTorch DDP's recommended no_sync pattern for gradient accumulation is super ugly, since it makes you copy-paste code into a context manager.
Last stated 9 Jun 2024 · 2 years ago
Holds Andrej Karpathy
Similar wording
Extra softmax operations should be avoided in training code because softmax is expensive at scale.
Last stated 10 Oct 2026 · a month ago
Holds Sebastian Raschka
Similar wording
Criticism that a simpler, scaled-up architecture underperforms a heavily hand-optimized baseline actually demonstrates why that architecture-scaling research matters.
Last stated 11 Jun 2021 · 5 years ago
Holds Gwern
Similar wording
The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.
Last stated 17 Oct 2025 · a year ago
Holds Andrej Karpathy
Similar wording
Human feedback cannot be scaled up enough, so compute-heavy methods that rely on humans will need some scaled supervision method.
Last stated 11 Nov 2024 · 2 years ago
Holds Dario Amodei
Similar wording
Language models should be thought of as somewhat out of date, because pre-training is too expensive to repeat often.
Last stated 27 Feb 2025 · 2 years ago
Holds Andrej Karpathy
Similar wording
Reinforcement learning cannot be scaled for robots the way it was for language models, because every attempt spends real robot hours instead of data-centre compute.
Last stated 12 Aug 2026 · 3 months ago
Holds Chelsea Finn
Similar wording
At the level of loss functions, language-model pre-training is remarkably simple, though running it at scale is a complete engineering feat.
Last stated 16 Jul 2026 · 4 months ago
Holds Grant Sanderson
Same subject: neural networks
A curve, which is what a deep learning model is, is a very bad substrate for discrete computation.
Last stated 6 Nov 2024 · 2 years ago
Holds François Chollet
Same subject: neural networks
A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable.
Last stated 10 Dec 2015 · 11 years ago
Holds Jian Sun Kaiming He
Same subject: neural networks
A project-level agent memory file works like a neural net, steering the agent much as a fine-tuned LoRA steers a model.
Last stated 23 Aug 2026 · 3 months ago
Holds Kun Chen
Same subject: pruning
A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it.
Last stated 9 Mar 2018 · 9 years ago
Holds Michael Carbin Jonathan Frankle
Same subject: pruning
A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work.
Last stated 9 Mar 2018 · 9 years ago
Holds Michael Carbin Jonathan Frankle
Same subject: pruning
Pruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free.
Last stated 9 Mar 2018 · 9 years ago
Holds Michael Carbin Jonathan Frankle
Same subject: transformers
A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.
Last stated 12 Jun 2017 · 9 years ago
Holds Illia Polosukhin Jakob Uszkoreit Niki Parmar Ashish Vaswani Noam Shazeer Aidan N. Gomez Llion Jones Łukasz Kaiser
Same subject: transformers
Linear probes are very strong classifiers, and attention-head probes are even stronger.
Last stated 20 Dec 2025 · 11 months ago
Holds Neel Nanda
Same subject: transformers
Self-attention can yield more interpretable models.
Last stated 12 Jun 2017 · 9 years ago
Holds Illia Polosukhin Jakob Uszkoreit Niki Parmar Ashish Vaswani Noam Shazeer Aidan N. Gomez Llion Jones Łukasz Kaiser