-
0:01:10 · watch on youtube.com↗
Their wordsIt's essentially if reinforcement learning is too expensive, which it is for many people, we can use model distillation to achieve really good results. Maybe not quite as good, but quite good results, a bit more cheaply.
- 6 min later
-
0:06:55 · watch on youtube.com↗
Their wordsNow this, of course, like I mentioned before, provides us with more information, but it's not super hard to implement it. The tricky part why not many people do that in practice is that for many models, we don't have access to this information.
- 2 min later
-
0:08:47 · watch on youtube.com↗
Their wordsOf course, there are hacks and workaround you can kind of synchronize these vocabularies and kind of make them match up and stuff like that. But it's it's super hacky and tricky. And so most people in practice, they just do hard distillation because it's just easier than doing this.
- 1 min later
-
0:09:45 · watch on youtube.com↗
Their wordsAnd then again, if you want to distill a model from Claude or OpenAI, which is I think not allowed, so I'm not urging you to do that.
-
0:10:03 · watch on youtube.com↗
Their wordsSo long story short, hard distillation is by far kind of like the easiest because it's the easiest in terms of data collection.
- 3 min later
-
0:13:24 · watch on youtube.com↗
Their wordsSo you can do it even on a different machine ahead of time, collect everything and then train. But on-policy would be a bit, I guess, even more accurate or better because it's always with the most up-to-date version of your student model.
- 3 min later
-
0:16:04 · watch on youtube.com↗
Their wordsBut yeah, again, this is like empirical. So you will have to run these experiments, but both are really fine approaches. There's nothing wrong with hard distillation.
- 1 min later
-
0:17:33 · watch on youtube.com↗
Their wordsSo distillation is actually simpler than reinforcement learning. The reason why we did this that way is reinforcement learning really is what is more interesting. That is like the state of the art. And now we are doing an approximation that is more efficient.
- 4 min later
-
0:21:49 · watch on youtube.com↗
Their wordsThey complain while people steal our reasoning traces and do distillation, because distillation will eventually get you there quicker if you already have the answers. It's cheaper than reinforcement learning.
-
0:22:01 · watch on youtube.com↗
Their wordsAnd I would say the best distilled models are still not as good as the state-of-the-art reasoning models, but it's maybe 80, 90% there.
-
0:22:10 · watch on youtube.com↗
Their wordsSo you can get a lot of performance from distillation by trying to mimic the teacher. But then, of course, there's a more risk of overfitting as well. So it's not like a completely free lunch there.
- 7 min later
-
0:29:02 · watch on youtube.com↗
Their wordsAnd so you see, this is also one of the earlier reasoning models, like newer models don't usually do that as often anymore, but it's kind of funny to see the model arguing through this
- 15 min later
-
0:44:07 · watch on youtube.com↗
Their wordsAnd then in my opinion, it's always a good idea. I, it's never a bad idea to check your data because data is often the most precious thing. Like if something is wrong in the data, you will also get really poor training results.
- 4 min later
-
0:48:31 · watch on youtube.com↗
Their wordsBut long story short, it is more common to only train on the answer tokens rather than the prompt tokens. It's just like also a common convention.
- 14 min later
-
1:02:49 · watch on youtube.com↗
Their wordsUsually it's a good idea to shuffle between those epochs to just prevent cycles. It's just better for the optimization.
- 1 min later
-
1:03:32 · watch on youtube.com↗
Their wordsSo really the core difference is, sure, we have multiple epochs. We could also do that in RLVR, but RLVR is so expensive that we usually don't do multiple epochs.
- 12 min later
-
1:15:11 · watch on youtube.com↗
Their wordsIt's even better than with a DeepSeek R1. I would say that's likely because it's from the same model family and it's a bit more familiar with a prompt format or the answer structure and so forth.
- 1 min later
-
1:16:07 · watch on youtube.com↗
Their wordsif you train a large state of the art model, you know, like DeepSeek R1 671 billion sure use reasoning, use RLVR for this reasoning training, but if you're developing a smaller model, you can get better results on the smaller model with distillation by distilling from a bigger one than doing RLVR from scratch on a small model.
- 1 min later
-
1:17:30 · watch on youtube.com↗
Their wordsSo the DeepSeekMath version two paper is also super interesting because they train an additional grader. So it is kind of like a process reward model, which I personally find interesting.
-
1:17:41 · watch on youtube.com↗
Their wordsAlso, most reasoning models are still trained on code and math. So it would be also interesting to go beyond those domains. So applying it to other domains in let's say biology or physics,
-
1:17:55 · watch on youtube.com↗
Their wordsif you think about it, you can apply the RLVR to anything where you have a verifiable reward, right? And so you just have to be a bit careful that it doesn't do the reward hacking.
- 1 min later
-
1:18:28 · watch on youtube.com↗
Their wordsSo instead of just developing a reasoning model that always reasons, which can be wasteful if you have a very simple prompt, we can also develop reasoning models that have different effort levels like light, medium, high, extra high, max, and ultra.
- 1 min later
-
1:19:14 · watch on youtube.com↗
Their wordsI try to stick to the fundamentals. So everything you see here is like the foundation and then there are always newer techniques on top of it. But once you understand these foundations, it's easier to understand these other yeah, concepts on top of it,