Tap a claim on the ring to put it at the centre.
← Extra softmax operations should be avoided in training code because softmax is expensive at scale.
17 connected korrents · 14 moments from 1 Dec 2023 to 10 Oct 2026.
Everything filed under software performance
software performance
Everything filed under scaling laws
scaling laws
Everything filed under neural networks
neural networks
Everything filed under software quality
software quality
Everything filed under design
design
Everything filed under AI alignment
AI alignment
Everything filed under developer tools
developer tools
Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject
Read this korrent: Extra softmax operations should be avoided in training code because softmax is expensive at scale.
Extra softmax operations should be avoided in training code because softmax is expensive at scale.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: Training in fp16 required gradient scaling, which is annoying and an extra source of state and complexity. — tap to centre the map on it
Training in fp16 required gradient scaling, which is annoying and an extra source of state and complexity.
Last stated 2 years ago
9 Jun 2024
AK
Andrej Karpathy — holds since 2024-06-09 — tap for who they are
Same subject: When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once. — tap to centre the map on it
When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once.
Last stated 3 years ago
1 Dec 2023
SR
Sebastian Ruder — holds since 2023-12-01 — tap for who they are
Same subject: Aiming for superhuman code quality may not be a realistic path. — tap to centre the map on it
Aiming for superhuman code quality may not be a realistic path.
Last stated 5 months ago
30 Jun 2026
AT
Adam Tornhill — holds since 2026-06-30 — tap for who they are
Same subject: When a KL loss term is not needed, it is just overhead, since it requires keeping an extra reference model in memory. — tap to centre the map on it
When a KL loss term is not needed, it is just overhead, since it requires keeping an extra reference model in memory.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: Training is no longer limited by data but by compute, because most of the data models learn from is now synthetic. — tap to centre the map on it
Training is no longer limited by data but by compute, because most of the data models learn from is now synthetic.
Last stated 8 months ago
23 Mar 2026
JH
Jensen Huang — holds since 2026-03-23 — tap for who they are
Same subject: Figma is still important, but using it to dial in full fidelity is probably the wrong use, because it is not the actual code base. — tap to centre the map on it
Figma is still important, but using it to dial in full fidelity is probably the wrong use, because it is not the actual code base.
Last stated 8 months ago
12 Mar 2026
JP
Josh Puckett — holds since 2026-03-12 — tap for who they are
Same subject: Language models should be thought of as somewhat out of date, because pre-training is too expensive to repeat often. — tap to centre the map on it
Language models should be thought of as somewhat out of date, because pre-training is too expensive to repeat often.
Last stated 2 years ago
27 Feb 2025
AK
Andrej Karpathy — holds since 2025-02-27 — tap for who they are
Same subject: AI systems keep trying hard outside training because a model that only exerted itself when it detected training would be useless and would be selected away. — tap to centre the map on it
AI systems keep trying hard outside training because a model that only exerted itself when it detected training would be useless and would be selected away.
Last stated 3 months ago
1 Sept 2026
AC
Ajeya Cotra — holds since 2026-09-01 — tap for who they are
Same subject: "Is the page loaded?" was never a real question, and web performance only became measurable once it was broken into the distinct moments a user actually notices. — tap to centre the map on it
"Is the page loaded?" was never a real question, and web performance only became measurable once it was broken into the distinct moments a user actually notices.
Last stated 3 months ago
19 Aug 2026
AO
Addy Osmani — holds since 2026-08-19 — tap for who they are
Same subject: "Optimize it later" would be safe advice only if every engineer already knew how to avoid the architectural mistakes that no later optimization can remove. — tap to centre the map on it
"Optimize it later" would be safe advice only if every engineer already knew how to avoid the architectural mistakes that no later optimization can remove.
Last stated 3 months ago
26 Aug 2026
CM
Casey Muratori — holds since 2026-08-26 — tap for who they are
Same subject: A 300-millisecond response budget is not an achievement but an admission: 300 milliseconds is an eternity in computing, and a product pitching it shows how far the bar has fallen. — tap to centre the map on it
A 300-millisecond response budget is not an achievement but an admission: 300 milliseconds is an eternity in computing, and a product pitching it shows how far the bar has fallen.
Last stated 3 months ago
26 Aug 2026
CM
Casey Muratori — holds since 2026-08-26 — tap for who they are
Same subject: A profiler measures time already spent in an operation, not how much time would be saved by optimizing it. — tap to centre the map on it
A profiler measures time already spent in an operation, not how much time would be saved by optimizing it.
Last stated 3 years ago
18 Dec 2023
NE
Nelson Elhage — holds since 2023-12-18 — tap for who they are
Same subject: A tool's speed comes primarily from its design decisions rather than the programming language it is written in. — tap to centre the map on it
A tool's speed comes primarily from its design decisions rather than the programming language it is written in.
Last stated 11 months ago
29 Dec 2025
AP
Aaron Patterson — holds since 2025-12-29 — tap for who they are
Same subject: For GPU performance, flops do not matter so much as the whole memory access pattern does, and torch.compile cannot find every optimization. — tap to centre the map on it
For GPU performance, flops do not matter so much as the whole memory access pattern does, and torch.compile cannot find every optimization.
Last stated 2 years ago
9 Jun 2024
AK
Andrej Karpathy — holds since 2024-06-09 — tap for who they are
Same subject: Compiler mechanisms that force a required optimization to occur, rather than relying on incidental compiler behavior, are a more robust way to write performance-sensitive code. — tap to centre the map on it
Compiler mechanisms that force a required optimization to occur, rather than relying on incidental compiler behavior, are a more robust way to write performance-sensitive code.
Last stated 2 years ago
9 Mar 2025
NE
Nelson Elhage — holds since 2025-03-09 — tap for who they are
Same subject: What clean-code polymorphism really costs you is not the virtual call but the optimizations the compiler can no longer perform once it cannot see what runs. — tap to centre the map on it
What clean-code polymorphism really costs you is not the virtual call but the optimizations the compiler can no longer perform once it cannot see what runs.
Last stated 3 months ago
26 Aug 2026
CM
Casey Muratori — holds since 2026-08-26 — tap for who they are
Same subject: A compiler that also serves an IDE has to be lazy and deferred throughout — a different way of building compilers from the one the textbooks teach. — tap to centre the map on it
A compiler that also serves an IDE has to be lazy and deferred throughout — a different way of building compilers from the one the textbooks teach.
Last stated 6 months ago
13 May 2026
AH
Anders Hejlsberg — holds since 2026-05-13 — tap for who they are
same subject or similar wording a cloud: claims about one subject, named for it bar: when it was last stated, on a scale from 2015 to today — full is today a face: someone who holds the claim — tap it for who they are
At the centre
Extra softmax operations should be avoided in training code because softmax is expensive at scale.
Last stated 10 Oct 2026 · a month ago
Holds Sebastian Raschka
Read this korrent →
Similar wording
Training in fp16 required gradient scaling, which is annoying and an extra source of state and complexity.
Last stated 9 Jun 2024 · 2 years ago
Holds Andrej Karpathy
Similar wording
When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once.
Last stated 1 Dec 2023 · 3 years ago
Holds Sebastian Ruder
Similar wording
Aiming for superhuman code quality may not be a realistic path.
Last stated 30 Jun 2026 · 5 months ago
Holds Adam Tornhill
Similar wording
When a KL loss term is not needed, it is just overhead, since it requires keeping an extra reference model in memory.
Last stated 10 Oct 2026 · a month ago
Holds Sebastian Raschka
Similar wording
Training is no longer limited by data but by compute, because most of the data models learn from is now synthetic.
Last stated 23 Mar 2026 · 8 months ago
Holds Jensen Huang
Similar wording
Figma is still important, but using it to dial in full fidelity is probably the wrong use, because it is not the actual code base.
Last stated 12 Mar 2026 · 8 months ago
Holds Josh Puckett
Similar wording
Language models should be thought of as somewhat out of date, because pre-training is too expensive to repeat often.
Last stated 27 Feb 2025 · 2 years ago
Holds Andrej Karpathy
Similar wording
AI systems keep trying hard outside training because a model that only exerted itself when it detected training would be useless and would be selected away.
Last stated 1 Sept 2026 · 3 months ago
Holds Ajeya Cotra
Same subject: software performance
"Is the page loaded?" was never a real question, and web performance only became measurable once it was broken into the distinct moments a user actually notices.
Last stated 19 Aug 2026 · 3 months ago
Holds Addy Osmani
Same subject: software performance
"Optimize it later" would be safe advice only if every engineer already knew how to avoid the architectural mistakes that no later optimization can remove.
Last stated 26 Aug 2026 · 3 months ago
Holds Casey Muratori
Same subject: software performance
A 300-millisecond response budget is not an achievement but an admission: 300 milliseconds is an eternity in computing, and a product pitching it shows how far the bar has fallen.
Last stated 26 Aug 2026 · 3 months ago
Holds Casey Muratori
Same subject: developer tools
A profiler measures time already spent in an operation, not how much time would be saved by optimizing it.
Last stated 18 Dec 2023 · 3 years ago
Holds Nelson Elhage
Same subject: developer tools
A tool's speed comes primarily from its design decisions rather than the programming language it is written in.
Last stated 29 Dec 2025 · 11 months ago
Holds Aaron Patterson
Same subject: developer tools
For GPU performance, flops do not matter so much as the whole memory access pattern does, and torch.compile cannot find every optimization.
Last stated 9 Jun 2024 · 2 years ago
Holds Andrej Karpathy
Same subject: compilers
Compiler mechanisms that force a required optimization to occur, rather than relying on incidental compiler behavior, are a more robust way to write performance-sensitive code.
Last stated 9 Mar 2025 · 2 years ago
Holds Nelson Elhage
Same subject: compilers
What clean-code polymorphism really costs you is not the virtual call but the optimizations the compiler can no longer perform once it cannot see what runs.
Last stated 26 Aug 2026 · 3 months ago
Holds Casey Muratori
Same subject: compilers
A compiler that also serves an IDE has to be lazy and deferred throughout — a different way of building compilers from the one the textbooks teach.
Last stated 13 May 2026 · 6 months ago
Holds Anders Hejlsberg