korrents

korrents · video

Build A Reasoning Model From Scratch 7: Reinforcement Learning 2 (Tracking, KL Terms, And More)

Sebastian Raschka · 1h 46m · youtube.com ↗

20 korrents from this recording

1h
Sebastian Raschka did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

  1. 0:02:27 · watch on youtube.com↗

    So first we have the loss, I should say for reinforcement learning with verifiable rewards or also with human feedback, like the loss is not quite as informative as the one that we would have in, let's say, classic deep learning for training or when we're training a classifier
  2. 1 min later
  3. 0:03:14 · watch on youtube.com↗

    So what we want to mainly see is that the loss is kind of stable around a certain value. It might fluctuate, but what we don't want to see is like large loss spikes.
  4. 2 min later
  5. 0:05:31 · watch on youtube.com↗

    I wouldn't necessarily from one example deduce that these intermediate steps are the ones that made the answer correct. But on average, we often see that longer answers are indeed correlated to answer correctness to a certain degree.
  6. 2 min later
  7. 0:07:56 · watch on youtube.com↗

    Then we will make some modifications to the algorithm to hopefully make it even better. For example, clipped policy ratios, they are technically essential in GRPO.
  8. 2 min later
  9. 0:09:29 · watch on youtube.com↗

    But personally, if I'm running things for like longer, more than, let's say, 10 or 20 minutes, I usually prefer using Python scripts. I run those primarily on a separate computer.
  10. 3 min later
  11. 0:12:19 · watch on youtube.com↗

    I personally think it's always a good idea if you're downloading something from the Internet to just briefly double check that it is indeed doing what you expect, because nowadays, you know, with in the age of A.I., it can happen that.
  12. 9 min later
  13. 0:20:58 · watch on youtube.com↗

    If it reaches one, and it's like one all the time, we might also as well stop the training, because if it's already answering everything correctly, there's no learning signal anymore.
  14. 2 min later
  15. 0:23:18 · watch on youtube.com↗

    Also the rollout number is very small here, only four. Everything is kept small for efficiency. So whatever you can increase, I would suggest increasing.
  16. 11 min later
  17. 0:34:27 · watch on youtube.com↗

    What's more interesting is the standard deviation. We want it to be high. Ideally, that means the data is spread out quite a bit.
  18. 9 min later
  19. 0:43:14 · watch on youtube.com↗

    And so technically we always want to avoid softmaxes in, in training code or anywhere. I mean, sometimes it's unavoidable, but we don't want to like have extra softmaxes, for example.
  20. 1 min later
  21. 0:44:43 · watch on youtube.com↗

    if the entropy is zero, that means we have probability of one for a certain token, the model is almost too confident, which means it never explores any other options really. So it will always deterministically pick the next token.
  22. 3 min later
  23. 0:47:14 · watch on youtube.com↗

    So we want maybe in an entropy that is around one, I would say it's good in this case, like zero would be bad. Um, 11 would be certainly also very bad. So like a moderate entropy of like, yeah, one, two, three, something like that.
  24. 17 min later
  25. 1:04:37 · watch on youtube.com↗

    So I thought that might be the one using as a reference, but in practice, that's kind of large. So usually smaller values could be, for example, 0.1 and so forth. So 10, 10 is relatively large.
  26. 9 min later
  27. 1:13:13 · watch on youtube.com↗

    So the verifier is kind of brittle in that sense. So as soon as you generate a text that doesn't look like human generated text, like just gobbledygook text, it will pass as non-AI generated. And so it was easy to fool the AI detector.
  28. 1 min later
  29. 1:14:17 · watch on youtube.com↗

    Now in domains where there's a lot of reward hacking, this is very important, but in other domains, for example, our math verifier, this is usually not necessary. So personally, when I trained math models, so models that generate math answers correctly, I found that it's usually not that necessary.
  30. 1:14:45 · watch on youtube.com↗

    They also omitted this KL loss term because it's just making the training more expensive. You have an additional reference model. You have to keep in memory and compute scores for that. And if you don't need that, it's just additional overhead.
  31. 14 min later
  32. 1:28:59 · watch on youtube.com↗

    but in practice, I mean, this would work, but it would be relatively, um, brittle because the model has not seen those during pre-training. So in an ideal case scenario, I would personally recommend taking the base model and do a bit more continued pre-training or supervised fine tuning with think tokens present.
  33. 9 min later
  34. 1:37:43 · watch on youtube.com↗

    And so what's likely happening here is that the model overemphasizes the format reward, so it's trying to get the think tags right, but then the real reward, the correctness reward gets neglected.
  35. 1 min later
  36. 1:38:31 · watch on youtube.com↗

    So instead of just brute force adding the format reward in every case, I would only personally make the modification that the format reward is only added if the answer is already correct.
  37. 3 min later
  38. 1:41:41 · watch on youtube.com↗

    And if you see instability, it's not necessarily that the algorithm is wrong. It can just be that the settings are just, everything is on a scale that is too small to see the benefits of that.