Tap a claim on the ring to put it at the centre.
← When a KL loss term is not needed, it is just overhead, since it…
17 connected korrents · 14 moments from 8 Aug 2023 to 5 Nov 2026.
Everything filed under software performance
software performance
Everything filed under LLMs
LLMs
Everything filed under AI alignment
AI alignment
Everything filed under reinforcement learning
reinforcement learning
Everything filed under developer tools
developer tools
Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject
Read this korrent: When a KL loss term is not needed, it is just overhead, since it requires keeping an extra reference model in memory.
When a KL loss term is not needed, it is just overhead, since it requires keeping an extra reference model in memory.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: A KL loss term is very important in domains with a lot of reward hacking but usually not necessary for math models. — tap to centre the map on it
A KL loss term is very important in domains with a lot of reward hacking but usually not necessary for math models.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: The KL term in GRPO is not essential; whether to use it is a hyperparameter to decide case by case, and on math models may do better without it. — tap to centre the map on it
The KL term in GRPO is not essential; whether to use it is a hyperparameter to decide case by case, and on math models may do better without it.
Last stated 2 months ago
3 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-03 — tap for who they are
Same subject: At the level of loss functions, language-model pre-training is remarkably simple, though running it at scale is a complete engineering feat. — tap to centre the map on it
At the level of loss functions, language-model pre-training is remarkably simple, though running it at scale is a complete engineering feat.
Last stated 4 months ago
16 Jul 2026
GS
Grant Sanderson — holds since 2026-07-16 — tap for who they are
Same subject: Language models should be thought of as somewhat out of date, because pre-training is too expensive to repeat often. — tap to centre the map on it
Language models should be thought of as somewhat out of date, because pre-training is too expensive to repeat often.
Last stated 2 years ago
27 Feb 2025
AK
Andrej Karpathy — holds since 2025-02-27 — tap for who they are
Same subject: The argument that RAG and long context make fine-tuning unnecessary holds when information is missing, but not when the model struggles with the task itself. — tap to centre the map on it
The argument that RAG and long context make fine-tuning unnecessary holds when information is missing, but not when the model struggles with the task itself.
Last stated 2 weeks ago
5 Nov 2026
AR
Aishwarya Naresh Reganti — holds since 2026-11-05 — tap for who they are
Same subject: The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out. — tap to centre the map on it
The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.
Last stated a year ago
17 Oct 2025
AK
Andrej Karpathy — holds since 2025-10-17 — tap for who they are
Same subject: Given good data, room to operate and sound numerical conditioning, models will learn on their own. — tap to centre the map on it
Given good data, room to operate and sound numerical conditioning, models will learn on their own.
Last stated 3 years ago
8 Aug 2023
DA
Dario Amodei — holds since 2023-08-08 — tap for who they are
Same subject: In reinforcement learning for LLMs, the loss is not as informative as in classic deep learning or pre-training. — tap to centre the map on it
In reinforcement learning for LLMs, the loss is not as informative as in classic deep learning or pre-training.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: "Is the page loaded?" was never a real question, and web performance only became measurable once it was broken into the distinct moments a user actually notices. — tap to centre the map on it
"Is the page loaded?" was never a real question, and web performance only became measurable once it was broken into the distinct moments a user actually notices.
Last stated 3 months ago
19 Aug 2026
AO
Addy Osmani — holds since 2026-08-19 — tap for who they are
Same subject: "Optimize it later" would be safe advice only if every engineer already knew how to avoid the architectural mistakes that no later optimization can remove. — tap to centre the map on it
"Optimize it later" would be safe advice only if every engineer already knew how to avoid the architectural mistakes that no later optimization can remove.
Last stated 3 months ago
26 Aug 2026
CM
Casey Muratori — holds since 2026-08-26 — tap for who they are
Same subject: A 300-millisecond response budget is not an achievement but an admission: 300 milliseconds is an eternity in computing, and a product pitching it shows how far the bar has fallen. — tap to centre the map on it
A 300-millisecond response budget is not an achievement but an admission: 300 milliseconds is an eternity in computing, and a product pitching it shows how far the bar has fallen.
Last stated 3 months ago
26 Aug 2026
CM
Casey Muratori — holds since 2026-08-26 — tap for who they are
Same subject: A profiler measures time already spent in an operation, not how much time would be saved by optimizing it. — tap to centre the map on it
A profiler measures time already spent in an operation, not how much time would be saved by optimizing it.
Last stated 3 years ago
18 Dec 2023
NE
Nelson Elhage — holds since 2023-12-18 — tap for who they are
Same subject: A tool's speed comes primarily from its design decisions rather than the programming language it is written in. — tap to centre the map on it
A tool's speed comes primarily from its design decisions rather than the programming language it is written in.
Last stated 11 months ago
29 Dec 2025
AP
Aaron Patterson — holds since 2025-12-29 — tap for who they are
Same subject: For GPU performance, flops do not matter so much as the whole memory access pattern does, and torch.compile cannot find every optimization. — tap to centre the map on it
For GPU performance, flops do not matter so much as the whole memory access pattern does, and torch.compile cannot find every optimization.
Last stated 2 years ago
9 Jun 2024
AK
Andrej Karpathy — holds since 2024-06-09 — tap for who they are
Same subject: Compiler mechanisms that force a required optimization to occur, rather than relying on incidental compiler behavior, are a more robust way to write performance-sensitive code. — tap to centre the map on it
Compiler mechanisms that force a required optimization to occur, rather than relying on incidental compiler behavior, are a more robust way to write performance-sensitive code.
Last stated 2 years ago
9 Mar 2025
NE
Nelson Elhage — holds since 2025-03-09 — tap for who they are
Same subject: What clean-code polymorphism really costs you is not the virtual call but the optimizations the compiler can no longer perform once it cannot see what runs. — tap to centre the map on it
What clean-code polymorphism really costs you is not the virtual call but the optimizations the compiler can no longer perform once it cannot see what runs.
Last stated 3 months ago
26 Aug 2026
CM
Casey Muratori — holds since 2026-08-26 — tap for who they are
Same subject: A compiler that also serves an IDE has to be lazy and deferred throughout — a different way of building compilers from the one the textbooks teach. — tap to centre the map on it
A compiler that also serves an IDE has to be lazy and deferred throughout — a different way of building compilers from the one the textbooks teach.
Last stated 6 months ago
13 May 2026
AH
Anders Hejlsberg — holds since 2026-05-13 — tap for who they are
same subject or similar wording a cloud: claims about one subject, named for it bar: when it was last stated, on a scale from 2015 to today — full is today a face: someone who holds the claim — tap it for who they are
At the centre
When a KL loss term is not needed, it is just overhead, since it requires keeping an extra reference model in memory.
Last stated 10 Oct 2026 · a month ago
Holds Sebastian Raschka
Read this korrent →
Similar wording
A KL loss term is very important in domains with a lot of reward hacking but usually not necessary for math models.
Last stated 10 Oct 2026 · a month ago
Holds Sebastian Raschka
Similar wording
The KL term in GRPO is not essential; whether to use it is a hyperparameter to decide case by case, and on math models may do better without it.
Last stated 3 Oct 2026 · 2 months ago
Holds Sebastian Raschka
Similar wording
At the level of loss functions, language-model pre-training is remarkably simple, though running it at scale is a complete engineering feat.
Last stated 16 Jul 2026 · 4 months ago
Holds Grant Sanderson
Similar wording
Language models should be thought of as somewhat out of date, because pre-training is too expensive to repeat often.
Last stated 27 Feb 2025 · 2 years ago
Holds Andrej Karpathy
Similar wording
The argument that RAG and long context make fine-tuning unnecessary holds when information is missing, but not when the model struggles with the task itself.
Last stated 5 Nov 2026 · 2 weeks ago
Holds Aishwarya Naresh Reganti
Similar wording
The knowledge a model soaks up in pre-training is holding it back; what we actually want is the intelligence with the knowledge stripped out.
Last stated 17 Oct 2025 · a year ago
Holds Andrej Karpathy
Similar wording
Given good data, room to operate and sound numerical conditioning, models will learn on their own.
Last stated 8 Aug 2023 · 3 years ago
Holds Dario Amodei
Similar wording
In reinforcement learning for LLMs, the loss is not as informative as in classic deep learning or pre-training.
Last stated 10 Oct 2026 · a month ago
Holds Sebastian Raschka
Same subject: software performance
"Is the page loaded?" was never a real question, and web performance only became measurable once it was broken into the distinct moments a user actually notices.
Last stated 19 Aug 2026 · 3 months ago
Holds Addy Osmani
Same subject: software performance
"Optimize it later" would be safe advice only if every engineer already knew how to avoid the architectural mistakes that no later optimization can remove.
Last stated 26 Aug 2026 · 3 months ago
Holds Casey Muratori
Same subject: software performance
A 300-millisecond response budget is not an achievement but an admission: 300 milliseconds is an eternity in computing, and a product pitching it shows how far the bar has fallen.
Last stated 26 Aug 2026 · 3 months ago
Holds Casey Muratori
Same subject: developer tools
A profiler measures time already spent in an operation, not how much time would be saved by optimizing it.
Last stated 18 Dec 2023 · 3 years ago
Holds Nelson Elhage
Same subject: developer tools
A tool's speed comes primarily from its design decisions rather than the programming language it is written in.
Last stated 29 Dec 2025 · 11 months ago
Holds Aaron Patterson
Same subject: developer tools
For GPU performance, flops do not matter so much as the whole memory access pattern does, and torch.compile cannot find every optimization.
Last stated 9 Jun 2024 · 2 years ago
Holds Andrej Karpathy
Same subject: compilers
Compiler mechanisms that force a required optimization to occur, rather than relying on incidental compiler behavior, are a more robust way to write performance-sensitive code.
Last stated 9 Mar 2025 · 2 years ago
Holds Nelson Elhage
Same subject: compilers
What clean-code polymorphism really costs you is not the virtual call but the optimizations the compiler can no longer perform once it cannot see what runs.
Last stated 26 Aug 2026 · 3 months ago
Holds Casey Muratori
Same subject: compilers
A compiler that also serves an IDE has to be lazy and deferred throughout — a different way of building compilers from the one the textbooks teach.
Last stated 13 May 2026 · 6 months ago
Holds Anders Hejlsberg