Tap a claim on the ring to put it at the centre.
← Gradient norm clipping guards against an unlucky bad batch producing a huge gradient that shocks the optimization.
17 connected korrents · 16 moments from 22 Dec 2014 to 21 Nov 2026.
Everything filed under optimizers
optimizers
Everything filed under neural networks
neural networks
Everything filed under reinforcement learning
reinforcement learning
Everything filed under simplicity
simplicity
Everything filed under LLMs
LLMs
Everything filed under pruning
pruning
Everything filed under robotics
robotics
Everything filed under GANs
GANs
Everything filed under transformers
transformers
Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject
Read this korrent: Gradient norm clipping guards against an unlucky bad batch producing a huge gradient that shocks the optimization.
Gradient norm clipping guards against an unlucky bad batch producing a huge gradient that shocks the optimization.
Last stated 2 years ago
9 Jun 2024
AK
Andrej Karpathy — holds since 2024-06-09 — tap for who they are
Same subject: Unchecked AI video generation risks degenerating into a reinforcement-learning-optimized feed of low-quality content that traps users. — tap to centre the map on it
Unchecked AI video generation risks degenerating into a reinforcement-learning-optimized feed of low-quality content that traps users.
Last stated a year ago
30 Sept 2025
SA
Sam Altman — holds since 2025-09-30 — tap for who they are
Same subject: The clip value of 10 taken from DeepSeek-R1 is large in practice; values such as 0.1 are more usual. — tap to centre the map on it
The clip value of 10 taken from DeepSeek-R1 is large in practice; values such as 0.1 are more usual.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: PyTorch DDP's recommended no_sync pattern for gradient accumulation is super ugly, since it makes you copy-paste code into a context manager. — tap to centre the map on it
PyTorch DDP's recommended no_sync pattern for gradient accumulation is super ugly, since it makes you copy-paste code into a context manager.
Last stated 2 years ago
9 Jun 2024
AK
Andrej Karpathy — holds since 2024-06-09 — tap for who they are
Same subject: Gradient descent finds a solution to the problems a model has seen; nothing in the algorithm makes it pick the one that generalises well. — tap to centre the map on it
Gradient descent finds a solution to the problems a model has seen; nothing in the algorithm makes it pick the one that generalises well.
Last stated a year ago
26 Sept 2025
RS
Richard Sutton — holds since 2025-09-26 — tap for who they are
Same subject: Inconsistent output from an AI product is not necessarily bad; it is how AI behaves. — tap to centre the map on it
Inconsistent output from an AI product is not necessarily bad; it is how AI behaves.
Last stated 3 weeks ago
31 Oct 2026
MN
Marily Nika — holds since 2026-10-31 — tap for who they are
Same subject: The saving pruning offers is only ever at inference, because the sparse architectures it produces are hard to train from the start. — tap to centre the map on it
The saving pruning offers is only ever at inference, because the sparse architectures it produces are hard to train from the start.
Last stated 9 years ago
9 Mar 2018
MC
Michael Carbin — holds since 2018-03-09 — tap for who they are
JF
Jonathan Frankle — holds since 2018-03-09 — tap for who they are
Same subject: It is the diversity of a robot data set, not its size, that produces generalization: dropping the most diverse slice hurts far more than dropping a random fifth. — tap to centre the map on it
It is the diversity of a robot data set, not its size, that produces generalization: dropping the most diverse slice hurts far more than dropping a random fifth.
Last stated 3 months ago
12 Aug 2026
CF
Chelsea Finn — holds since 2026-08-12 — tap for who they are
Same subject: An adversarial GAN loss should only supplement a larger unsupervised or semi-supervised generative model rather than stand alone as a primary approach. — tap to centre the map on it
An adversarial GAN loss should only supplement a larger unsupervised or semi-supervised generative model rather than stand alone as a primary approach.
Last stated 6 years ago
4 Feb 2021
GW
Gwern — holds since 2021-02-04 — tap for who they are
Same subject: AdamW is still one of the most widely used optimizers, though Muon has become popular for layers other than embeddings. — tap to centre the map on it
AdamW is still one of the most widely used optimizers, though Muon has become popular for layers other than embeddings.
Last stated 2 months ago
3 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-03 — tap for who they are
Same subject: An optimizer that is invariant to rescaling the gradients and cheap in memory is what makes very large models practical to train. — tap to centre the map on it
An optimizer that is invariant to rescaling the gradients and cheap in memory is what makes very large models practical to train.
Last stated 12 years ago
22 Dec 2014
JB
Jimmy Ba — holds since 2014-12-22 — tap for who they are
DK
Diederik P. Kingma — holds since 2014-12-22 — tap for who they are
Same subject: An optimizer's hyper-parameters should mean something a practitioner can reason about, and should rarely need tuning. — tap to centre the map on it
An optimizer's hyper-parameters should mean something a practitioner can reason about, and should rarely need tuning.
Last stated 12 years ago
22 Dec 2014
JB
Jimmy Ba — holds since 2014-12-22 — tap for who they are
DK
Diederik P. Kingma — holds since 2014-12-22 — tap for who they are
Same subject: The current AI stack is not among the top 50% most efficient approaches to learning. — tap to centre the map on it
The current AI stack is not among the top 50% most efficient approaches to learning.
Last stated yesterday
21 Nov 2026
FC
François Chollet — holds since 2026-11-21 — tap for who they are
Same subject: A single AI paper rarely changes everything; the last one that did was 'Attention Is All You Need' in 2017. — tap to centre the map on it
A single AI paper rarely changes everything; the last one that did was 'Attention Is All You Need' in 2017.
Last stated 2 years ago
6 Jan 2025
AR
Aishwarya Naresh Reganti — holds since 2025-01-06 — tap for who they are
Same subject: A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. — tap to centre the map on it
A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.
Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
Same subject: If the goal is just to engineer a good system, there is no reason not to use backpropagation. — tap to centre the map on it
If the goal is just to engineer a good system, there is no reason not to use backpropagation.
Last stated 4 years ago
27 Mar 2023
IS
Ilya Sutskever — holds since 2023-03-27 — tap for who they are
Same subject: A curve, which is what a deep learning model is, is a very bad substrate for discrete computation. — tap to centre the map on it
A curve, which is what a deep learning model is, is a very bad substrate for discrete computation.
Last stated 2 years ago
6 Nov 2024
FC
François Chollet — holds since 2024-11-06 — tap for who they are
Same subject: A deep bidirectional model is strictly more powerful than a left-to-right model or a shallow concatenation of unidirectional models. — tap to centre the map on it
A deep bidirectional model is strictly more powerful than a left-to-right model or a shallow concatenation of unidirectional models.
Last stated 8 years ago
11 Oct 2018
KT
Kristina Toutanova — holds since 2018-10-11 — tap for who they are
MC
Ming-Wei Chang — holds since 2018-10-11 — tap for who they are
KL
Kenton Lee — holds since 2018-10-11 — tap for who they are
JD
Jacob Devlin — holds since 2018-10-11 — tap for who they are
same subject or similar wording a cloud: claims about one subject, named for it bar: when it was last stated, on a scale from 2014 to today (stretched back to the oldest claim here) — full is today a face: someone who holds the claim — tap it for who they are