korrents

On the map

Tap a claim on the ring to put it at the centre.

← Extra softmax operations should be avoided in training code because softmax is expensive at scale.

17 connected korrents · 14 moments from 1 Dec 2023 to 10 Oct 2026.

Everything filed under software performance software performance Everything filed under scaling laws scaling laws Everything filed under neural networks neural networks Everything filed under software quality software quality Everything filed under design design Everything filed under AI alignment AI alignment Everything filed under developer tools developer tools Same subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subject Read this korrent: Extra softmax operations should be avoided in training code because softmax is expensive at scale. Extra softmax operations should be avoidedin training code because softmax isexpensive at scale. Last stated a month ago 10 Oct 2026 SR Sebastian Raschka — holds since 2026-10-10 — tap for who they are Same subject: Training in fp16 required gradient scaling, which is annoying and an extra source of state and complexity. — tap to centre the map on it Training in fp16 required gradientscaling, which is annoying and anextra source of state andcomplexity. Last stated 2 years ago 9 Jun 2024 AK Andrej Karpathy — holds since 2024-06-09 — tap for who they are Same subject: When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once. — tap to centre the map on it When pre-training data is limited,it is better to train a smallermodel for multiple epochs than totrain a larger model on unique dataonce. Last stated 3 years ago 1 Dec 2023 SR Sebastian Ruder — holds since 2023-12-01 — tap for who they are Same subject: Aiming for superhuman code quality may not be a realistic path. — tap to centre the map on it Aiming for superhuman code qualitymay not be a realistic path. Last stated 5 months ago 30 Jun 2026 AT Adam Tornhill — holds since 2026-06-30 — tap for who they are Same subject: When a KL loss term is not needed, it is just overhead, since it requires keeping an extra reference model in memory. — tap to centre the map on it When a KL loss term is not needed,it is just overhead, since itrequires keeping an extra referencemodel in memory. Last stated a month ago 10 Oct 2026 SR Sebastian Raschka — holds since 2026-10-10 — tap for who they are Same subject: Training is no longer limited by data but by compute, because most of the data models learn from is now synthetic. — tap to centre the map on it Training is no longer limited bydata but by compute, because most ofthe data models learn from is nowsynthetic. Last stated 8 months ago 23 Mar 2026 JH Jensen Huang — holds since 2026-03-23 — tap for who they are Same subject: Figma is still important, but using it to dial in full fidelity is probably the wrong use, because it is not the actual code base. — tap to centre the map on it Figma is still important, but usingit to dial in full fidelity isprobably the wrong use, because itis not the actual code base. Last stated 8 months ago 12 Mar 2026 JP Josh Puckett — holds since 2026-03-12 — tap for who they are Same subject: Language models should be thought of as somewhat out of date, because pre-training is too expensive to repeat often. — tap to centre the map on it Language models should be thought ofas somewhat out of date, becausepre-training is too expensive torepeat often. Last stated 2 years ago 27 Feb 2025 AK Andrej Karpathy — holds since 2025-02-27 — tap for who they are Same subject: AI systems keep trying hard outside training because a model that only exerted itself when it detected training would be useless and would be selected away. — tap to centre the map on it AI systems keep trying hard outsidetraining because a model that onlyexerted itself when it detectedtraining would be useless and wouldbe selected away. Last stated 3 months ago 1 Sept 2026 AC Ajeya Cotra — holds since 2026-09-01 — tap for who they are Same subject: "Is the page loaded?" was never a real question, and web performance only became measurable once it was broken into the distinct moments a user actually notices. — tap to centre the map on it "Is the page loaded?" was never areal question, and web performanceonly became measurable once it wasbroken into the distinct moments auser actually notices. Last stated 3 months ago 19 Aug 2026 AO Addy Osmani — holds since 2026-08-19 — tap for who they are Same subject: "Optimize it later" would be safe advice only if every engineer already knew how to avoid the architectural mistakes that no later optimization can remove. — tap to centre the map on it "Optimize it later" would be safeadvice only if every engineeralready knew how to avoid thearchitectural mistakes that no lateroptimization can remove. Last stated 3 months ago 26 Aug 2026 CM Casey Muratori — holds since 2026-08-26 — tap for who they are Same subject: A 300-millisecond response budget is not an achievement but an admission: 300 milliseconds is an eternity in computing, and a product pitching it shows how far the bar has fallen. — tap to centre the map on it A 300-millisecond response budget isnot an achievement but an admission:300 milliseconds is an eternity incomputing, and a product pitching itshows how far the bar has fallen. Last stated 3 months ago 26 Aug 2026 CM Casey Muratori — holds since 2026-08-26 — tap for who they are Same subject: A profiler measures time already spent in an operation, not how much time would be saved by optimizing it. — tap to centre the map on it A profiler measures time alreadyspent in an operation, not how muchtime would be saved by optimizingit. Last stated 3 years ago 18 Dec 2023 NE Nelson Elhage — holds since 2023-12-18 — tap for who they are Same subject: A tool's speed comes primarily from its design decisions rather than the programming language it is written in. — tap to centre the map on it A tool's speed comes primarily fromits design decisions rather than theprogramming language it is writtenin. Last stated 11 months ago 29 Dec 2025 AP Aaron Patterson — holds since 2025-12-29 — tap for who they are Same subject: For GPU performance, flops do not matter so much as the whole memory access pattern does, and torch.compile cannot find every optimization. — tap to centre the map on it For GPU performance, flops do notmatter so much as the whole memoryaccess pattern does, andtorch.compile cannot find everyoptimization. Last stated 2 years ago 9 Jun 2024 AK Andrej Karpathy — holds since 2024-06-09 — tap for who they are Same subject: Compiler mechanisms that force a required optimization to occur, rather than relying on incidental compiler behavior, are a more robust way to write performance-sensitive code. — tap to centre the map on it Compiler mechanisms that force arequired optimization to occur,rather than relying on incidentalcompiler behavior, are a more robustway to write performance-sensitivecode. Last stated 2 years ago 9 Mar 2025 NE Nelson Elhage — holds since 2025-03-09 — tap for who they are Same subject: What clean-code polymorphism really costs you is not the virtual call but the optimizations the compiler can no longer perform once it cannot see what runs. — tap to centre the map on it What clean-code polymorphism reallycosts you is not the virtual callbut the optimizations the compilercan no longer perform once it cannotsee what runs. Last stated 3 months ago 26 Aug 2026 CM Casey Muratori — holds since 2026-08-26 — tap for who they are Same subject: A compiler that also serves an IDE has to be lazy and deferred throughout — a different way of building compilers from the one the textbooks teach. — tap to centre the map on it A compiler that also serves an IDEhas to be lazy and deferredthroughout — a different way ofbuilding compilers from the one thetextbooks teach. Last stated 6 months ago 13 May 2026 AH Anders Hejlsberg — holds since 2026-05-13 — tap for who they are
same subject or similar wordinga cloud: claims about one subject, named for itbar: when it was last stated, on a scale from 2015 to today — full is todaya face: someone who holds the claim — tap it for who they are

At the centre Extra softmax operations should be avoided in training code because softmax is expensive at scale. Last stated 10 Oct 2026 · a month ago Holds Sebastian Raschka Read this korrent →