korrents

korrents · video

Build A Reasoning Model From Scratch 8: Distillation

Sebastian Raschka · 1h 21m · youtube.com ↗

23 korrents from this recording

1h
Sebastian Raschka did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

  1. 0:01:10 · watch on youtube.com↗

    It's essentially if reinforcement learning is too expensive, which it is for many people, we can use model distillation to achieve really good results. Maybe not quite as good, but quite good results, a bit more cheaply.
  2. 6 min later
  3. 0:06:55 · watch on youtube.com↗

    Now this, of course, like I mentioned before, provides us with more information, but it's not super hard to implement it. The tricky part why not many people do that in practice is that for many models, we don't have access to this information.
  4. 2 min later
  5. 0:08:47 · watch on youtube.com↗

    Of course, there are hacks and workaround you can kind of synchronize these vocabularies and kind of make them match up and stuff like that. But it's it's super hacky and tricky. And so most people in practice, they just do hard distillation because it's just easier than doing this.
  6. 1 min later
  7. 0:09:45 · watch on youtube.com↗

    And then again, if you want to distill a model from Claude or OpenAI, which is I think not allowed, so I'm not urging you to do that.
  8. 0:10:03 · watch on youtube.com↗

    So long story short, hard distillation is by far kind of like the easiest because it's the easiest in terms of data collection.
  9. 3 min later
  10. 0:13:24 · watch on youtube.com↗

    So you can do it even on a different machine ahead of time, collect everything and then train. But on-policy would be a bit, I guess, even more accurate or better because it's always with the most up-to-date version of your student model.
  11. 3 min later
  12. 0:16:04 · watch on youtube.com↗

    But yeah, again, this is like empirical. So you will have to run these experiments, but both are really fine approaches. There's nothing wrong with hard distillation.
  13. 1 min later
  14. 0:17:33 · watch on youtube.com↗

    So distillation is actually simpler than reinforcement learning. The reason why we did this that way is reinforcement learning really is what is more interesting. That is like the state of the art. And now we are doing an approximation that is more efficient.
  15. 4 min later
  16. 0:21:49 · watch on youtube.com↗

    They complain while people steal our reasoning traces and do distillation, because distillation will eventually get you there quicker if you already have the answers. It's cheaper than reinforcement learning.
  17. 0:22:01 · watch on youtube.com↗

    And I would say the best distilled models are still not as good as the state-of-the-art reasoning models, but it's maybe 80, 90% there.
  18. 0:22:10 · watch on youtube.com↗

    So you can get a lot of performance from distillation by trying to mimic the teacher. But then, of course, there's a more risk of overfitting as well. So it's not like a completely free lunch there.
  19. 7 min later
  20. 0:29:02 · watch on youtube.com↗

    And so you see, this is also one of the earlier reasoning models, like newer models don't usually do that as often anymore, but it's kind of funny to see the model arguing through this
  21. 15 min later
  22. 0:44:07 · watch on youtube.com↗

    And then in my opinion, it's always a good idea. I, it's never a bad idea to check your data because data is often the most precious thing. Like if something is wrong in the data, you will also get really poor training results.
  23. 4 min later
  24. 0:48:31 · watch on youtube.com↗

    But long story short, it is more common to only train on the answer tokens rather than the prompt tokens. It's just like also a common convention.
  25. 14 min later
  26. 1:02:49 · watch on youtube.com↗

    Usually it's a good idea to shuffle between those epochs to just prevent cycles. It's just better for the optimization.
  27. 1 min later
  28. 1:03:32 · watch on youtube.com↗

    So really the core difference is, sure, we have multiple epochs. We could also do that in RLVR, but RLVR is so expensive that we usually don't do multiple epochs.
  29. 12 min later
  30. 1:15:11 · watch on youtube.com↗

    It's even better than with a DeepSeek R1. I would say that's likely because it's from the same model family and it's a bit more familiar with a prompt format or the answer structure and so forth.
  31. 1 min later
  32. 1:16:07 · watch on youtube.com↗

    if you train a large state of the art model, you know, like DeepSeek R1 671 billion sure use reasoning, use RLVR for this reasoning training, but if you're developing a smaller model, you can get better results on the smaller model with distillation by distilling from a bigger one than doing RLVR from scratch on a small model.
  33. 1 min later
  34. 1:17:30 · watch on youtube.com↗

    So the DeepSeekMath version two paper is also super interesting because they train an additional grader. So it is kind of like a process reward model, which I personally find interesting.
  35. 1:17:41 · watch on youtube.com↗

    Also, most reasoning models are still trained on code and math. So it would be also interesting to go beyond those domains. So applying it to other domains in let's say biology or physics,
  36. 1:17:55 · watch on youtube.com↗

    if you think about it, you can apply the RLVR to anything where you have a verifiable reward, right? And so you just have to be a bit careful that it doesn't do the reward hacking.
  37. 1 min later
  38. 1:18:28 · watch on youtube.com↗

    So instead of just developing a reasoning model that always reasons, which can be wasteful if you have a very simple prompt, we can also develop reasoning models that have different effort levels like light, medium, high, extra high, max, and ultra.
  39. 1 min later
  40. 1:19:14 · watch on youtube.com↗

    I try to stick to the fundamentals. So everything you see here is like the foundation and then there are always newer techniques on top of it. But once you understand these foundations, it's easier to understand these other yeah, concepts on top of it,