korrents

On the map

Tap a claim on the ring to put it at the centre.

← The common explanation of cross-entropy loss as comparison with a…

17 connected korrents · 13 moments from 10 Dec 2015 to 17 Oct 2026.

Everything filed under neural networks neural networks Everything filed under LLMs LLMs Everything filed under mathematics mathematics Everything filed under physics physics Same subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subject Read this korrent: The common explanation of cross-entropy loss as comparison with a one-hot distribution is lame because it does not explain why cross-entropy is used. The common explanation of cross-entropyloss as comparison with a one-hotdistribution is lame because it does notexplain why cross-entropy is used. Last stated 4 months ago 16 Jul 2026 GS Grant Sanderson — holds since 2026-07-16 — tap for who they are Same subject: Wherever cross-entropy appears, there is a way to connect it to compression, just as pi always hides a connection to circles. — tap to centre the map on it Wherever cross-entropy appears,there is a way to connect it tocompression, just as pi always hidesa connection to circles. Last stated 4 months ago 16 Jul 2026 GS Grant Sanderson — holds since 2026-07-16 — tap for who they are Same subject: Scaling up GPT will never produce AGI, because a model trained on cross-entropy loss cannot get there; reinforcement learning in rich environments is required. — tap to centre the map on it Scaling up GPT will never produceAGI, because a model trained oncross-entropy loss cannot get there;reinforcement learning in richenvironments is required. Last stated 3 years ago 29 Jun 2023 GH George Hotz — holds since 2023-06-29 — tap for who they are Same subject: Soft distillation is not hard to implement; few people use it mainly because for many models the full probability distribution is not accessible. — tap to centre the map on it Soft distillation is not hard toimplement; few people use it mainlybecause for many models the fullprobability distribution is notaccessible. Last stated a month ago 17 Oct 2026 SR Sebastian Raschka — holds since 2026-10-17 — tap for who they are Same subject: Zero token entropy during RL training is undesirable because the model is too confident and never explores other options. — tap to centre the map on it Zero token entropy during RLtraining is undesirable because themodel is too confident and neverexplores other options. Last stated a month ago 10 Oct 2026 SR Sebastian Raschka — holds since 2026-10-10 — tap for who they are Same subject: Judging one model worse than another from a one-shot, probabilistic output on a given day is not a good way to evaluate it. — tap to centre the map on it Judging one model worse than anotherfrom a one-shot, probabilisticoutput on a given day is not a goodway to evaluate it. Last stated 6 months ago 19 May 2026 TG Tommy Geoco — holds since 2026-05-19 — tap for who they are Same subject: Non-equilibrium physics will not explain life, and entropy has almost nothing to do with it. — tap to centre the map on it Non-equilibrium physics will notexplain life, and entropy has almostnothing to do with it. Last stated 2 years ago 13 Jun 2024 SW Sara Walker — holds since 2024-06-13 — tap for who they are Same subject: For a Qwen-sized vocabulary, a moderate entropy of around one to three is probably good, while zero or the maximum of about 11 would be bad. — tap to centre the map on it For a Qwen-sized vocabulary, amoderate entropy of around one tothree is probably good, while zeroor the maximum of about 11 would bebad. Last stated a month ago 10 Oct 2026 SR Sebastian Raschka — holds since 2026-10-10 — tap for who they are Same subject: In reinforcement learning for LLMs, the loss is not as informative as in classic deep learning or pre-training. — tap to centre the map on it In reinforcement learning for LLMs,the loss is not as informative as inclassic deep learning orpre-training. Last stated a month ago 10 Oct 2026 SR Sebastian Raschka — holds since 2026-10-10 — tap for who they are Same subject: A curve, which is what a deep learning model is, is a very bad substrate for discrete computation. — tap to centre the map on it A curve, which is what a deeplearning model is, is a very badsubstrate for discrete computation. Last stated 2 years ago 6 Nov 2024 FC François Chollet — holds since 2024-11-06 — tap for who they are Same subject: A layer should learn a residual with reference to its own input rather than an unreferenced function, which is what makes great depth trainable. — tap to centre the map on it A layer should learn a residual withreference to its own input ratherthan an unreferenced function, whichis what makes great depth trainable. Last stated 11 years ago 10 Dec 2015 JS Jian Sun — holds since 2015-12-10 — tap for who they are KH Kaiming He — holds since 2015-12-10 — tap for who they are Same subject: A project-level agent memory file works like a neural net, steering the agent much as a fine-tuned LoRA steers a model. — tap to centre the map on it A project-level agent memory fileworks like a neural net, steeringthe agent much as a fine-tuned LoRAsteers a model. Last stated 3 months ago 23 Aug 2026 KC Kun Chen — holds since 2026-08-23 — tap for who they are Same subject: A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it. — tap to centre the map on it A small enough winning ticket learnsfaster than the network it was cutout of, and ends up more accuratethan it. Last stated 9 years ago 9 Mar 2018 MC Michael Carbin — holds since 2018-03-09 — tap for who they are JF Jonathan Frankle — holds since 2018-03-09 — tap for who they are Same subject: A winning ticket wins on its initial weights: the connections it keeps started at values that happen to make training work. — tap to centre the map on it A winning ticket wins on its initialweights: the connections it keepsstarted at values that happen tomake training work. Last stated 9 years ago 9 Mar 2018 MC Michael Carbin — holds since 2018-03-09 — tap for who they are JF Jonathan Frankle — holds since 2018-03-09 — tap for who they are Same subject: Intelligence is the execution of small, simplicity-weighted programs found by search over smooth loss landscapes in overparameterized networks, approaching Bayes-optimal reasoning as scale increases. — tap to centre the map on it Intelligence is the execution ofsmall, simplicity-weighted programsfound by search over smooth losslandscapes in overparameterizednetworks, approaching Bayes-optimalreasoning as scale increases. Last stated 5 years ago 11 Jun 2021 GW Gwern — holds since 2021-06-11 — tap for who they are Same subject: A transduction model can rely entirely on self-attention for input and output representations without sequence-aligned RNNs or convolution. — tap to centre the map on it A transduction model can relyentirely on self-attention for inputand output representations withoutsequence-aligned RNNs orconvolution. Last stated 9 years ago 12 Jun 2017 IP Illia Polosukhin — holds since 2017-06-12 — tap for who they are JU Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are NP Niki Parmar — holds since 2017-06-12 — tap for who they are AV Ashish Vaswani — holds since 2017-06-12 — tap for who they are NS Noam Shazeer — holds since 2017-06-12 — tap for who they are AG Aidan N. Gomez — holds since 2017-06-12 — tap for who they are +2 2 more Same subject: Linear probes are very strong classifiers, and attention-head probes are even stronger. — tap to centre the map on it Linear probes are very strongclassifiers, and attention-headprobes are even stronger. Last stated 11 months ago 20 Dec 2025 NN Neel Nanda — holds since 2025-12-20 — tap for who they are Same subject: Self-attention can yield more interpretable models. — tap to centre the map on it Self-attention can yield moreinterpretable models. Last stated 9 years ago 12 Jun 2017 IP Illia Polosukhin — holds since 2017-06-12 — tap for who they are JU Jakob Uszkoreit — holds since 2017-06-12 — tap for who they are NP Niki Parmar — holds since 2017-06-12 — tap for who they are AV Ashish Vaswani — holds since 2017-06-12 — tap for who they are NS Noam Shazeer — holds since 2017-06-12 — tap for who they are AG Aidan N. Gomez — holds since 2017-06-12 — tap for who they are +2 2 more
same subject or similar wordinga cloud: claims about one subject, named for itbar: when it was last stated, on a scale from 2015 to today — full is todaya face: someone who holds the claim — tap it for who they are

At the centre The common explanation of cross-entropy loss as comparison with a one-hot distribution is lame because it does not explain why cross-entropy is used. Last stated 16 Jul 2026 · 4 months ago Holds Grant Sanderson Read this korrent →