korrents

On the map

Tap a claim on the ring to put it at the centre.

← AdamW is still one of the most widely used optimizers, though Muon has…

12 connected korrents · 10 moments from 22 Dec 2014 to 21 Nov 2026.

Everything filed under optimizers optimizers Everything filed under neural networks neural networks Everything filed under transformers transformers Same subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subject Read this korrent: AdamW is still one of the most widely used optimizers, though Muon has become popular for layers other than embeddings. AdamW is still one of the most widely usedoptimizers, though Muon has become popularfor layers other than embeddings. Last stated 2 months ago 3 Oct 2026 SR Sebastian Raschka — holds since 2026-10-03 — tap for who they are Same subject: An optimizer that is invariant to rescaling the gradients and cheap in memory is what makes very large models practical to train. — tap to centre the map on it An optimizer that is invariant torescaling the gradients and cheap inmemory is what makes very largemodels practical to train. Last stated 12 years ago 22 Dec 2014 JB Jimmy Ba — holds since 2014-12-22 — tap for who they are DK Diederik P. Kingma — holds since 2014-12-22 — tap for who they are Same subject: An optimizer's hyper-parameters should mean something a practitioner can reason about, and should rarely need tuning. — tap to centre the map on it An optimizer's hyper-parametersshould mean something a practitionercan reason about, and should rarelyneed tuning. Last stated 12 years ago 22 Dec 2014 JB Jimmy Ba — holds since 2014-12-22 — tap for who they are DK Diederik P. Kingma — holds since 2014-12-22 — tap for who they are Same subject: Gradient descent finds a solution to the problems a model has seen; nothing in the algorithm makes it pick the one that generalises well. — tap to centre the map on it Gradient descent finds a solution tothe problems a model has seen;nothing in the algorithm makes itpick the one that generalises well. Last stated a year ago 26 Sept 2025 RS Richard Sutton — holds since 2025-09-26 — tap for who they are Same subject: Stochastic optimization can be done well using only adaptive estimates of the gradient's lower-order moments. — tap to centre the map on it Stochastic optimization can be donewell using only adaptive estimatesof the gradient's lower-ordermoments. Last stated 12 years ago 22 Dec 2014 JB Jimmy Ba — holds since 2014-12-22 — tap for who they are DK Diederik P. Kingma — holds since 2014-12-22 — tap for who they are Same subject: The current AI stack is not among the top 50% most efficient approaches to learning. — tap to centre the map on it The current AI stack is not amongthe top 50% most efficientapproaches to learning. Last stated yesterday 21 Nov 2026 FC François Chollet — holds since 2026-11-21 — tap for who they are Same subject: The same optimizer should handle objectives that move under it and gradients that are noisy or sparse. — tap to centre the map on it The same optimizer should handleobjectives that move under it andgradients that are noisy or sparse. Last stated 12 years ago 22 Dec 2014 JB Jimmy Ba — holds since 2014-12-22 — tap for who they are DK Diederik P. Kingma — holds since 2014-12-22 — tap for who they are Same subject: A single AI paper rarely changes everything; the last one that did was 'Attention Is All You Need' in 2017. — tap to centre the map on it A single AI paper rarely changeseverything; the last one that didwas 'Attention Is All You Need' in2017. Last stated 2 years ago 6 Jan 2025 AR Aishwarya Naresh Reganti — holds since 2025-01-06 — tap for who they are Same subject: A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. — tap to centre the map on it A transduction model can relyentirely on self-attention for inputand output representations withoutsequence-aligned RNNs orconvolution. Last stated 9 years ago 12 Jun 2017 IP Illia Polosukhin — holds since 2017-06-12 — tap for who they are JU Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are NP Niki Parmar — holds since 2017-06-12 — tap for who they are AV Ashish Vaswani — holds since 2017-06-12 — tap for who they are NS Noam Shazeer — holds since 2017-06-12 — tap for who they are AG Aidan N. Gomez — holds since 2017-06-12 — tap for who they are +2 2 more Same subject: As reasoning models and agents retain more tokens for longer, KV-cache size and attention cost become the main constraint on scaling context. — tap to centre the map on it As reasoning models and agentsretain more tokens for longer,KV-cache size and attention costbecome the main constraint onscaling context. Last stated 6 months ago 16 May 2026 SR Sebastian Raschka — holds since 2026-05-16 — tap for who they are Same subject: A curve, which is what a deep learning model is, is a very bad substrate for discrete computation. — tap to centre the map on it A curve, which is what a deeplearning model is, is a very badsubstrate for discrete computation. Last stated 2 years ago 6 Nov 2024 FC François Chollet — holds since 2024-11-06 — tap for who they are Same subject: A deep bidirectional model is strictly more powerful than a left-to-right model or a shallow concatenation of unidirectional models. — tap to centre the map on it A deep bidirectional model isstrictly more powerful than aleft-to-right model or a shallowconcatenation of unidirectionalmodels. Last stated 8 years ago 11 Oct 2018 KT Kristina Toutanova — holds since 2018-10-11 — tap for who they are MC Ming-Wei Chang — holds since 2018-10-11 — tap for who they are KL Kenton Lee — holds since 2018-10-11 — tap for who they are JD Jacob Devlin — holds since 2018-10-11 — tap for who they are Same subject: A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable. — tap to centre the map on it A layer should learn a residual withreference to its own input ratherthan an unreferenced function, whichis what makes great depth trainable. Last stated 11 years ago 10 Dec 2015 JS Jian Sun — holds since 2015-12-10 — tap for who they are KH Kaiming He — holds since 2015-12-10 — tap for who they are
same subject or similar wordinga cloud: claims about one subject, named for itbar: when it was last stated, on a scale from 2014 to today (stretched back to the oldest claim here) — full is todaya face: someone who holds the claim — tap it for who they are

At the centre AdamW is still one of the most widely used optimizers, though Muon has become popular for layers other than embeddings. Last stated 3 Oct 2026 · 2 months ago Holds Sebastian Raschka Read this korrent →