Our readingWhat is a korrent?
Distillation carries a higher risk of overfitting, so it is not a free lunch.
Drawn from what Sebastian Raschka said
SubjectLLMs
Our readingWhat is a korrent?
Drawn from what Sebastian Raschka said
So you can get a lot of performance from distillation by trying to mimic the teacher. But then, of course, there's a more risk of overfitting as well. So it's not like a completely free lunch there.
Where this was said
Watch from 0:22:10 plays here↗Build A Reasoning Model From Scratch 8: Distillationyoutube.com
All 23 korrents from this recordingthis one is 11th
Added to korrents 9 Nov 2026 · How quotes work · Something wrong? Tell us
Do you hold this korrent?Do you also believe this?
I hold this tooShared subject first, then similar wording. They may agree or disagree.