Tap a claim on the ring to put it at the centre.
← The common explanation of cross-entropy loss as comparison with a…
17 connected korrents · 13 moments from 10 Dec 2015 to 17 Oct 2026.
Everything filed under neural networks
neural networks
Everything filed under LLMs
LLMs
Everything filed under mathematics
mathematics
Everything filed under physics
physics
Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject
Read this korrent: The common explanation of cross-entropy loss as comparison with a one-hot distribution is lame because it does not explain why cross-entropy is used.
The common explanation of cross-entropy loss as comparison with a one-hot distribution is lame because it does not explain why cross-entropy is used.
Last stated 4 months ago
16 Jul 2026
GS
Grant Sanderson — holds since 2026-07-16 — tap for who they are
Same subject: Wherever cross-entropy appears, there is a way to connect it to compression, just as pi always hides a connection to circles. — tap to centre the map on it
Wherever cross-entropy appears, there is a way to connect it to compression, just as pi always hides a connection to circles.
Last stated 4 months ago
16 Jul 2026
GS
Grant Sanderson — holds since 2026-07-16 — tap for who they are
Same subject: Scaling up GPT will never produce AGI, because a model trained on cross-entropy loss cannot get there; reinforcement learning in rich environments is required. — tap to centre the map on it
Scaling up GPT will never produce AGI, because a model trained on cross-entropy loss cannot get there; reinforcement learning in rich environments is required.
Last stated 3 years ago
29 Jun 2023
GH
George Hotz — holds since 2023-06-29 — tap for who they are
Same subject: Soft distillation is not hard to implement; few people use it mainly because for many models the full probability distribution is not accessible. — tap to centre the map on it
Soft distillation is not hard to implement; few people use it mainly because for many models the full probability distribution is not accessible.
Last stated a month ago
17 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-17 — tap for who they are
Same subject: Zero token entropy during RL training is undesirable because the model is too confident and never explores other options. — tap to centre the map on it
Zero token entropy during RL training is undesirable because the model is too confident and never explores other options.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: Judging one model worse than another from a one-shot, probabilistic output on a given day is not a good way to evaluate it. — tap to centre the map on it
Judging one model worse than another from a one-shot, probabilistic output on a given day is not a good way to evaluate it.
Last stated 6 months ago
19 May 2026
TG
Tommy Geoco — holds since 2026-05-19 — tap for who they are
Same subject: Non-equilibrium physics will not explain life, and entropy has almost nothing to do with it. — tap to centre the map on it
Non-equilibrium physics will not explain life, and entropy has almost nothing to do with it.
Last stated 2 years ago
13 Jun 2024
SW
Sara Walker — holds since 2024-06-13 — tap for who they are
Same subject: For a Qwen-sized vocabulary, a moderate entropy of around one to three is probably good, while zero or the maximum of about 11 would be bad. — tap to centre the map on it
For a Qwen-sized vocabulary, a moderate entropy of around one to three is probably good, while zero or the maximum of about 11 would be bad.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: In reinforcement learning for LLMs, the loss is not as informative as in classic deep learning or pre-training. — tap to centre the map on it
In reinforcement learning for LLMs, the loss is not as informative as in classic deep learning or pre-training.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: A curve, which is what a deep learning model is, is a very bad substrate for discrete computation. — tap to centre the map on it
A curve, which is what a deep learning model is, is a very bad substrate for discrete computation.
Last stated 2 years ago
6 Nov 2024
FC
François Chollet — holds since 2024-11-06 — tap for who they are
Same subject: A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable. — tap to centre the map on it
A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable.
Last stated 11 years ago
10 Dec 2015
JS
Jian Sun — holds since 2015-12-10 — tap for who they are
KH
Kaiming He — holds since 2015-12-10 — tap for who they are
Same subject: A project-level agent memory file works like a neural net, steering the agent much as a fine-tuned LoRA steers a model. — tap to centre the map on it
A project-level agent memory file works like a neural net, steering the agent much as a fine-tuned LoRA steers a model.
Last stated 3 months ago
23 Aug 2026
KC
Kun Chen — holds since 2026-08-23 — tap for who they are
Same subject: A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it. — tap to centre the map on it
A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it.
Last stated 9 years ago
9 Mar 2018
MC
Michael Carbin — holds since 2018-03-09 — tap for who they are
JF
Jonathan Frankle — holds since 2018-03-09 — tap for who they are
Same subject: A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work. — tap to centre the map on it
A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work.
Last stated 9 years ago
9 Mar 2018
MC
Michael Carbin — holds since 2018-03-09 — tap for who they are
JF
Jonathan Frankle — holds since 2018-03-09 — tap for who they are
Same subject: Intelligence is the execution of small, simplicity-weighted programs found by search over smooth loss landscapes in overparameterized networks, approaching Bayes-optimal reasoning as scale increases. — tap to centre the map on it
Intelligence is the execution of small, simplicity-weighted programs found by search over smooth loss landscapes in overparameterized networks, approaching Bayes-optimal reasoning as scale increases.
Last stated 5 years ago
11 Jun 2021
GW
Gwern — holds since 2021-06-11 — tap for who they are
Same subject: A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. — tap to centre the map on it
A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.
Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
Same subject: Linear probes are very strong classifiers, and attention-head probes are even stronger. — tap to centre the map on it
Linear probes are very strong classifiers, and attention-head probes are even stronger.
Last stated 11 months ago
20 Dec 2025
NN
Neel Nanda — holds since 2025-12-20 — tap for who they are
Same subject: Self-attention can yield more interpretable models. — tap to centre the map on it
Self-attention can yield more interpretable models.
Last stated 9 years ago
12 Jun 2017
IP
Illia Polosukhin — holds since 2017-06-12 — tap for who they are
JU
Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are
NP
Niki Parmar — holds since 2017-06-12 — tap for who they are
AV
Ashish Vaswani — holds since 2017-06-12 — tap for who they are
NS
Noam Shazeer — holds since 2017-06-12 — tap for who they are
AG
Aidan N. Gomez — holds since 2017-06-12 — tap for who they are
+2
2 more
same subject or similar wording a cloud: claims about one subject, named for it bar: when it was last stated, on a scale from 2015 to today — full is today a face: someone who holds the claim — tap it for who they are
At the centre
The common explanation of cross-entropy loss as comparison with a one-hot distribution is lame because it does not explain why cross-entropy is used.
Last stated 16 Jul 2026 · 4 months ago
Holds Grant Sanderson
Read this korrent →
Similar wording
Wherever cross-entropy appears, there is a way to connect it to compression, just as pi always hides a connection to circles.
Last stated 16 Jul 2026 · 4 months ago
Holds Grant Sanderson
Similar wording
Scaling up GPT will never produce AGI, because a model trained on cross-entropy loss cannot get there; reinforcement learning in rich environments is required.
Last stated 29 Jun 2023 · 3 years ago
Holds George Hotz
Similar wording
Soft distillation is not hard to implement; few people use it mainly because for many models the full probability distribution is not accessible.
Last stated 17 Oct 2026 · a month ago
Holds Sebastian Raschka
Similar wording
Zero token entropy during RL training is undesirable because the model is too confident and never explores other options.
Last stated 10 Oct 2026 · a month ago
Holds Sebastian Raschka
Similar wording
Judging one model worse than another from a one-shot, probabilistic output on a given day is not a good way to evaluate it.
Last stated 19 May 2026 · 6 months ago
Holds Tommy Geoco
Similar wording
Non-equilibrium physics will not explain life, and entropy has almost nothing to do with it.
Last stated 13 Jun 2024 · 2 years ago
Holds Sara Walker
Similar wording
For a Qwen-sized vocabulary, a moderate entropy of around one to three is probably good, while zero or the maximum of about 11 would be bad.
Last stated 10 Oct 2026 · a month ago
Holds Sebastian Raschka
Similar wording
In reinforcement learning for LLMs, the loss is not as informative as in classic deep learning or pre-training.
Last stated 10 Oct 2026 · a month ago
Holds Sebastian Raschka
Same subject: neural networks
A curve, which is what a deep learning model is, is a very bad substrate for discrete computation.
Last stated 6 Nov 2024 · 2 years ago
Holds François Chollet
Same subject: neural networks
A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable.
Last stated 10 Dec 2015 · 11 years ago
Holds Jian Sun Kaiming He
Same subject: neural networks
A project-level agent memory file works like a neural net, steering the agent much as a fine-tuned LoRA steers a model.
Last stated 23 Aug 2026 · 3 months ago
Holds Kun Chen
Same subject: pruning
A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it.
Last stated 9 Mar 2018 · 9 years ago
Holds Michael Carbin Jonathan Frankle
Same subject: pruning
A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work.
Last stated 9 Mar 2018 · 9 years ago
Holds Michael Carbin Jonathan Frankle
Same subject: pruning
Intelligence is the execution of small, simplicity-weighted programs found by search over smooth loss landscapes in overparameterized networks, approaching Bayes-optimal reasoning as scale increases.
Last stated 11 Jun 2021 · 5 years ago
Holds Gwern
Same subject: transformers
A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution.
Last stated 12 Jun 2017 · 9 years ago
Holds Illia Polosukhin Jakob Uszkoreit Niki Parmar Ashish Vaswani Noam Shazeer Aidan N. Gomez Llion Jones Łukasz Kaiser
Same subject: transformers
Linear probes are very strong classifiers, and attention-head probes are even stronger.
Last stated 20 Dec 2025 · 11 months ago
Holds Neel Nanda
Same subject: transformers
Self-attention can yield more interpretable models.
Last stated 12 Jun 2017 · 9 years ago
Holds Illia Polosukhin Jakob Uszkoreit Niki Parmar Ashish Vaswani Noam Shazeer Aidan N. Gomez Llion Jones Łukasz Kaiser