korrents

On the map

Tap a claim on the ring to put it at the centre.

← Training in fp16 required gradient scaling, which is annoying and an extra source of state and complexity.

17 connected korrents · 14 moments from 10 Dec 2015 to 10 Oct 2026.

Everything filed under neural networks neural networks Everything filed under LLMs LLMs Everything filed under simplicity simplicity Everything filed under software performance software performance Everything filed under scaling laws scaling laws Same subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subject Read this korrent: Training in fp16 required gradient scaling, which is annoying and an extra source of state and complexity. Training in fp16 required gradientscaling, which is annoying and an extrasource of state and complexity. Last stated 2 years ago 9 Jun 2024 AK Andrej Karpathy — holds since 2024-06-09 — tap for who they are Same subject: PyTorch DDP's recommended no_sync pattern for gradient accumulation is super ugly, since it makes you copy-paste code into a context manager. — tap to centre the map on it PyTorch DDP's recommended no_syncpattern for gradient accumulation issuper ugly, since it makes youcopy-paste code into a contextmanager. Last stated 2 years ago 9 Jun 2024 AK Andrej Karpathy — holds since 2024-06-09 — tap for who they are Same subject: Extra softmax operations should be avoided in training code because softmax is expensive at scale. — tap to centre the map on it Extra softmax operations should beavoided in training code becausesoftmax is expensive at scale. Last stated a month ago 10 Oct 2026 SR Sebastian Raschka — holds since 2026-10-10 — tap for who they are Same subject: Criticism that a simpler, scaled-up architecture underperforms a heavily hand-optimized baseline actually demonstrates why that architecture-scaling research matters. — tap to centre the map on it Criticism that a simpler, scaled-uparchitecture underperforms a heavilyhand-optimized baseline actuallydemonstrates why thatarchitecture-scaling researchmatters. Last stated 5 years ago 11 Jun 2021 GW Gwern — holds since 2021-06-11 — tap for who they are Same subject: The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out. — tap to centre the map on it The knowledge a model soaks up inpre-training is holding it back;what we actually want is theintelligence with the knowledgestripped out. Last stated a year ago 17 Oct 2025 AK Andrej Karpathy — holds since 2025-10-17 — tap for who they are Same subject: Human feedback cannot be scaled up enough, so compute-heavy methods that rely on humans will need some scaled supervision method. — tap to centre the map on it Human feedback cannot be scaled upenough, so compute-heavy methodsthat rely on humans will need somescaled supervision method. Last stated 2 years ago 11 Nov 2024 DA Dario Amodei — holds since 2024-11-11 — tap for who they are Same subject: Language models should be thought of as somewhat out of date, because pre-training is too expensive to repeat often. — tap to centre the map on it Language models should be thought ofas somewhat out of date, becausepre-training is too expensive torepeat often. Last stated 2 years ago 27 Feb 2025 AK Andrej Karpathy — holds since 2025-02-27 — tap for who they are Same subject: Reinforcement learning cannot be scaled for robots the way it was for language models, because every attempt spends real robot hours instead of data-centre compute. — tap to centre the map on it Reinforcement learning cannot bescaled for robots the way it was forlanguage models, because everyattempt spends real robot hoursinstead of data-centre compute. Last stated 3 months ago 12 Aug 2026 CF Chelsea Finn — holds since 2026-08-12 — tap for who they are Same subject: At the level of loss functions, language-model pre-training is remarkably simple, though running it at scale is a complete engineering feat. — tap to centre the map on it At the level of loss functions,language-model pre-training isremarkably simple, though running itat scale is a complete engineeringfeat. Last stated 4 months ago 16 Jul 2026 GS Grant Sanderson — holds since 2026-07-16 — tap for who they are Same subject: A curve, which is what a deep learning model is, is a very bad substrate for discrete computation. — tap to centre the map on it A curve, which is what a deeplearning model is, is a very badsubstrate for discrete computation. Last stated 2 years ago 6 Nov 2024 FC François Chollet — holds since 2024-11-06 — tap for who they are Same subject: A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable. — tap to centre the map on it A layer should learn a residual withreference to its own input ratherthan an unreferenced function, whichis what makes great depth trainable. Last stated 11 years ago 10 Dec 2015 JS Jian Sun — holds since 2015-12-10 — tap for who they are KH Kaiming He — holds since 2015-12-10 — tap for who they are Same subject: A project-level agent memory file works like a neural net, steering the agent much as a fine-tuned LoRA steers a model. — tap to centre the map on it A project-level agent memory fileworks like a neural net, steeringthe agent much as a fine-tuned LoRAsteers a model. Last stated 3 months ago 23 Aug 2026 KC Kun Chen — holds since 2026-08-23 — tap for who they are Same subject: A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it. — tap to centre the map on it A small enough winning ticket learnsfaster than the network it was cutout of, and ends up more accuratethan it. Last stated 9 years ago 9 Mar 2018 MC Michael Carbin — holds since 2018-03-09 — tap for who they are JF Jonathan Frankle — holds since 2018-03-09 — tap for who they are Same subject: A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work. — tap to centre the map on it A winning ticket wins on its initialweights: the connections it keepsstarted at values that happen tomake training work. Last stated 9 years ago 9 Mar 2018 MC Michael Carbin — holds since 2018-03-09 — tap for who they are JF Jonathan Frankle — holds since 2018-03-09 — tap for who they are Same subject: Pruning cuts the parameters of a trained network by over 90% without costing accuracy, so inference gets smaller and faster for free. — tap to centre the map on it Pruning cuts the parameters of atrained network by over 90% withoutcosting accuracy, so inference getssmaller and faster for free. Last stated 9 years ago 9 Mar 2018 MC Michael Carbin — holds since 2018-03-09 — tap for who they are JF Jonathan Frankle — holds since 2018-03-09 — tap for who they are Same subject: A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. — tap to centre the map on it A transduction model can relyentirely on self-attention for inputand output representations withoutsequence-aligned RNNs orconvolution. Last stated 9 years ago 12 Jun 2017 IP Illia Polosukhin — holds since 2017-06-12 — tap for who they are JU Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are NP Niki Parmar — holds since 2017-06-12 — tap for who they are AV Ashish Vaswani — holds since 2017-06-12 — tap for who they are NS Noam Shazeer — holds since 2017-06-12 — tap for who they are AG Aidan N. Gomez — holds since 2017-06-12 — tap for who they are +2 2 more Same subject: Linear probes are very strong classifiers, and attention-head probes are even stronger. — tap to centre the map on it Linear probes are very strongclassifiers, and attention-headprobes are even stronger. Last stated 11 months ago 20 Dec 2025 NN Neel Nanda — holds since 2025-12-20 — tap for who they are Same subject: Self-attention can yield more interpretable models. — tap to centre the map on it Self-attention can yield moreinterpretable models. Last stated 9 years ago 12 Jun 2017 IP Illia Polosukhin — holds since 2017-06-12 — tap for who they are JU Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are NP Niki Parmar — holds since 2017-06-12 — tap for who they are AV Ashish Vaswani — holds since 2017-06-12 — tap for who they are NS Noam Shazeer — holds since 2017-06-12 — tap for who they are AG Aidan N. Gomez — holds since 2017-06-12 — tap for who they are +2 2 more
same subject or similar wordinga cloud: claims about one subject, named for itbar: when it was last stated, on a scale from 2015 to today — full is todaya face: someone who holds the claim — tap it for who they are

At the centre Training in fp16 required gradient scaling, which is annoying and an extra source of state and complexity. Last stated 9 Jun 2024 · 2 years ago Holds Andrej Karpathy Read this korrent →