korrents

On the map

Tap a claim on the ring to put it at the centre.

← Gradient norm clipping guards against an unlucky bad batch producing a huge gradient that shocks the optimization.

17 connected korrents · 16 moments from 22 Dec 2014 to 21 Nov 2026.

Everything filed under optimizers optimizers Everything filed under neural networks neural networks Everything filed under reinforcement learning reinforcement learning Everything filed under simplicity simplicity Everything filed under LLMs LLMs Everything filed under pruning pruning Everything filed under robotics robotics Everything filed under GANs GANs Everything filed under transformers transformers Same subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subject Read this korrent: Gradient norm clipping guards against an unlucky bad batch producing a huge gradient that shocks the optimization. Gradient norm clipping guards against anunlucky bad batch producing a hugegradient that shocks the optimization. Last stated 2 years ago 9 Jun 2024 AK Andrej Karpathy — holds since 2024-06-09 — tap for who they are Same subject: Unchecked AI video generation risks degenerating into a reinforcement-learning-optimized feed of low-quality content that traps users. — tap to centre the map on it Unchecked AI video generation risksdegenerating into areinforcement-learning-optimizedfeed of low-quality content thattraps users. Last stated a year ago 30 Sept 2025 SA Sam Altman — holds since 2025-09-30 — tap for who they are Same subject: The clip value of 10 taken from DeepSeek-R1 is large in practice; values such as 0.1 are more usual. — tap to centre the map on it The clip value of 10 taken fromDeepSeek-R1 is large in practice;values such as 0.1 are more usual. Last stated a month ago 10 Oct 2026 SR Sebastian Raschka — holds since 2026-10-10 — tap for who they are Same subject: PyTorch DDP's recommended no_sync pattern for gradient accumulation is super ugly, since it makes you copy-paste code into a context manager. — tap to centre the map on it PyTorch DDP's recommended no_syncpattern for gradient accumulation issuper ugly, since it makes youcopy-paste code into a contextmanager. Last stated 2 years ago 9 Jun 2024 AK Andrej Karpathy — holds since 2024-06-09 — tap for who they are Same subject: Gradient descent finds a solution to the problems a model has seen; nothing in the algorithm makes it pick the one that generalises well. — tap to centre the map on it Gradient descent finds a solution tothe problems a model has seen;nothing in the algorithm makes itpick the one that generalises well. Last stated a year ago 26 Sept 2025 RS Richard Sutton — holds since 2025-09-26 — tap for who they are Same subject: Inconsistent output from an AI product is not necessarily bad; it is how AI behaves. — tap to centre the map on it Inconsistent output from an AIproduct is not necessarily bad; itis how AI behaves. Last stated 3 weeks ago 31 Oct 2026 MN Marily Nika — holds since 2026-10-31 — tap for who they are Same subject: The saving pruning offers is only ever at inference, because the sparse architectures it produces are hard to train from the start. — tap to centre the map on it The saving pruning offers is onlyever at inference, because thesparse architectures it produces arehard to train from the start. Last stated 9 years ago 9 Mar 2018 MC Michael Carbin — holds since 2018-03-09 — tap for who they are JF Jonathan Frankle — holds since 2018-03-09 — tap for who they are Same subject: It is the diversity of a robot data set, not its size, that produces generalization: dropping the most diverse slice hurts far more than dropping a random fifth. — tap to centre the map on it It is the diversity of a robot dataset, not its size, that producesgeneralization: dropping the mostdiverse slice hurts far more thandropping a random fifth. Last stated 3 months ago 12 Aug 2026 CF Chelsea Finn — holds since 2026-08-12 — tap for who they are Same subject: An adversarial GAN loss should only supplement a larger unsupervised or semi-supervised generative model rather than stand alone as a primary approach. — tap to centre the map on it An adversarial GAN loss should onlysupplement a larger unsupervised orsemi-supervised generative modelrather than stand alone as a primaryapproach. Last stated 6 years ago 4 Feb 2021 GW Gwern — holds since 2021-02-04 — tap for who they are Same subject: AdamW is still one of the most widely used optimizers, though Muon has become popular for layers other than embeddings. — tap to centre the map on it AdamW is still one of the mostwidely used optimizers, though Muonhas become popular for layers otherthan embeddings. Last stated 2 months ago 3 Oct 2026 SR Sebastian Raschka — holds since 2026-10-03 — tap for who they are Same subject: An optimizer that is invariant to rescaling the gradients and cheap in memory is what makes very large models practical to train. — tap to centre the map on it An optimizer that is invariant torescaling the gradients and cheap inmemory is what makes very largemodels practical to train. Last stated 12 years ago 22 Dec 2014 JB Jimmy Ba — holds since 2014-12-22 — tap for who they are DK Diederik P. Kingma — holds since 2014-12-22 — tap for who they are Same subject: An optimizer's hyper-parameters should mean something a practitioner can reason about, and should rarely need tuning. — tap to centre the map on it An optimizer's hyper-parametersshould mean something a practitionercan reason about, and should rarelyneed tuning. Last stated 12 years ago 22 Dec 2014 JB Jimmy Ba — holds since 2014-12-22 — tap for who they are DK Diederik P. Kingma — holds since 2014-12-22 — tap for who they are Same subject: The current AI stack is not among the top 50% most efficient approaches to learning. — tap to centre the map on it The current AI stack is not amongthe top 50% most efficientapproaches to learning. Last stated yesterday 21 Nov 2026 FC François Chollet — holds since 2026-11-21 — tap for who they are Same subject: A single AI paper rarely changes everything; the last one that did was 'Attention Is All You Need' in 2017. — tap to centre the map on it A single AI paper rarely changeseverything; the last one that didwas 'Attention Is All You Need' in2017. Last stated 2 years ago 6 Jan 2025 AR Aishwarya Naresh Reganti — holds since 2025-01-06 — tap for who they are Same subject: A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. — tap to centre the map on it A transduction model can relyentirely on self-attention for inputand output representations withoutsequence-aligned RNNs orconvolution. Last stated 9 years ago 12 Jun 2017 IP Illia Polosukhin — holds since 2017-06-12 — tap for who they are JU Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are NP Niki Parmar — holds since 2017-06-12 — tap for who they are AV Ashish Vaswani — holds since 2017-06-12 — tap for who they are NS Noam Shazeer — holds since 2017-06-12 — tap for who they are AG Aidan N. Gomez — holds since 2017-06-12 — tap for who they are +2 2 more Same subject: If the goal is just to engineer a good system, there is no reason not to use backpropagation. — tap to centre the map on it If the goal is just to engineer agood system, there is no reason notto use backpropagation. Last stated 4 years ago 27 Mar 2023 IS Ilya Sutskever — holds since 2023-03-27 — tap for who they are Same subject: A curve, which is what a deep learning model is, is a very bad substrate for discrete computation. — tap to centre the map on it A curve, which is what a deeplearning model is, is a very badsubstrate for discrete computation. Last stated 2 years ago 6 Nov 2024 FC François Chollet — holds since 2024-11-06 — tap for who they are Same subject: A deep bidirectional model is strictly more powerful than a left-to-right model or a shallow concatenation of unidirectional models. — tap to centre the map on it A deep bidirectional model isstrictly more powerful than aleft-to-right model or a shallowconcatenation of unidirectionalmodels. Last stated 8 years ago 11 Oct 2018 KT Kristina Toutanova — holds since 2018-10-11 — tap for who they are MC Ming-Wei Chang — holds since 2018-10-11 — tap for who they are KL Kenton Lee — holds since 2018-10-11 — tap for who they are JD Jacob Devlin — holds since 2018-10-11 — tap for who they are
same subject or similar wordinga cloud: claims about one subject, named for itbar: when it was last stated, on a scale from 2014 to today (stretched back to the oldest claim here) — full is todaya face: someone who holds the claim — tap it for who they are

At the centre Gradient norm clipping guards against an unlucky bad batch producing a huge gradient that shocks the optimization. Last stated 9 Jun 2024 · 2 years ago Holds Andrej Karpathy Read this korrent →