Tap a claim on the ring to put it at the centre.
← AdamW is still one of the most widely used optimizers, though Muon has…
12 connected korrents · 10 moments from 22 Dec 2014 to 21 Nov 2026.
Everything filed under optimizers
optimizers
Everything filed under neural networks
neural networks
Everything filed under transformers
transformers
Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject
Read this korrent: AdamW is still one of the most widely used optimizers, though Muon has become popular for layers other than embeddings.
AdamW is still one of the most widely used optimizers, though Muon has become popular for layers other than embeddings.
Last stated 2 months ago
3 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-03 — tap for who they are
Same subject: An optimizer that is invariant to rescaling the gradients and cheap in memory is what makes very large models practical to train. — tap to centre the map on it
An optimizer that is invariant to rescaling the gradients and cheap in memory is what makes very large models practical to train.
Last stated 12 years ago
22 Dec 2014
JB
Jimmy Ba — holds since 2014-12-22 — tap for who they are
DK
Diederik P. Kingma — holds since 2014-12-22 — tap for who they are
Same subject: An optimizer's hyper-parameters should mean something a practitioner can reason about, and should rarely need tuning. — tap to centre the map on it
An optimizer's hyper-parameters should mean something a practitioner can reason about, and should rarely need tuning.
Last stated 12 years ago
22 Dec 2014
JB
Jimmy Ba — holds since 2014-12-22 — tap for who they are
DK
Diederik P. Kingma — holds since 2014-12-22 — tap for who they are
Same subject: Gradient descent finds a solution to the problems a model has seen; nothing in the algorithm makes it pick the one that generalises well. — tap to centre the map on it
Gradient descent finds a solution to the problems a model has seen; nothing in the algorithm makes it pick the one that generalises well.
Last stated a year ago
26 Sept 2025
RS
Richard Sutton — holds since 2025-09-26 — tap for who they are
Same subject: Stochastic optimization can be done well using only adaptive estimates of the gradient's lower-order moments. — tap to centre the map on it
Stochastic optimization can be done well using only adaptive estimates of the gradient's lower-order moments.
Last stated 12 years ago
22 Dec 2014
JB
Jimmy Ba — holds since 2014-12-22 — tap for who they are
DK
Diederik P. Kingma — holds since 2014-12-22 — tap for who they are
Same subject: The current AI stack is not among the top 50% most efficient approaches to learning. — tap to centre the map on it
The current AI stack is not among the top 50% most efficient approaches to learning.
Last stated yesterday
21 Nov 2026
FC
François Chollet — holds since 2026-11-21 — tap for who they are
Same subject: The same optimizer should handle objectives that move under it and gradients that are noisy or sparse. — tap to centre the map on it
The same optimizer should handle objectives that move under it and gradients that are noisy or sparse.
Last stated 12 years ago
22 Dec 2014
JB
Jimmy Ba — holds since 2014-12-22 — tap for who they are
DK
Diederik P. Kingma — holds since 2014-12-22 — tap for who they are
Same subject: A single AI paper rarely changes everything; the last one that did was 'Attention Is All You Need' in 2017. — tap to centre the map on it
A single AI paper rarely changes everything; the last one that did was 'Attention Is All You Need' in 2017.
Last stated 2 years ago
6 Jan 2025
AR
Aishwarya Naresh Reganti — holds since 2025-01-06 — tap for who they are
Same subject: A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. — tap to centre the map on it
A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.
Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
Same subject: As reasoning models and agents retain more tokens for longer, KV-cache size and attention cost become the main constraint on scaling context. — tap to centre the map on it
As reasoning models and agents retain more tokens for longer, KV-cache size and attention cost become the main constraint on scaling context.
Last stated 6 months ago
16 May 2026
SR
Sebastian Raschka — holds since 2026-05-16 — tap for who they are
Same subject: A curve, which is what a deep learning model is, is a very bad substrate for discrete computation. — tap to centre the map on it
A curve, which is what a deep learning model is, is a very bad substrate for discrete computation.
Last stated 2 years ago
6 Nov 2024
FC
François Chollet — holds since 2024-11-06 — tap for who they are
Same subject: A deep bidirectional model is strictly more powerful than a left-to-right model or a shallow concatenation of unidirectional models. — tap to centre the map on it
A deep bidirectional model is strictly more powerful than a left-to-right model or a shallow concatenation of unidirectional models.
Last stated 8 years ago
11 Oct 2018
KT
Kristina Toutanova — holds since 2018-10-11 — tap for who they are
MC
Ming-Wei Chang — holds since 2018-10-11 — tap for who they are
KL
Kenton Lee — holds since 2018-10-11 — tap for who they are
JD
Jacob Devlin — holds since 2018-10-11 — tap for who they are
Same subject: A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable. — tap to centre the map on it
A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable.
Last stated 11 years ago
10 Dec 2015
JS
Jian Sun — holds since 2015-12-10 — tap for who they are
KH
Kaiming He — holds since 2015-12-10 — tap for who they are
same subject or similar wording a cloud: claims about one subject, named for it bar: when it was last stated, on a scale from 2014 to today (stretched back to the oldest claim here) — full is today a face: someone who holds the claim — tap it for who they are