Our readingWhat is a korrent?
Distillation gives a small model a much richer training signal than ordinary pre-training, which weights only the one true next token.
Drawn from what Grant Sanderson said
What this subject means
scaling laws Whether more compute keeps buying more capability, and where the curve now bends -- pre-training, post-training, test-time.