korrents

korrents · video

Build A Reasoning Model From Scratch 6: Reinforcement Learning 1 (Implementing GRPO for RLVR)

Sebastian Raschka · 1h 26m · youtube.com ↗

17 korrents from this recording

1h
Sebastian Raschka did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

  1. 0:02:29 · watch on youtube.com↗

    I would rather not call this a reasoning model if it has not been trained with reinforcement learning with verifiable rewards.
  2. 1 min later
  3. 0:03:23 · watch on youtube.com↗

    I wouldn't call it thought pattern, because it makes it sound like they are thinking like humans, which they don't. I mean, they're still large language models and not human brains or something.
  4. 1 min later
  5. 0:04:49 · watch on youtube.com↗

    But what I wanted to say there is that in general, the more capable a model is, the less reasoning it kind of needs to do explicitly.
  6. 1 min later
  7. 0:06:00 · watch on youtube.com↗

    So I wouldn't say it's like a malicious or intentional thing that the model is doing. But if the model is just like, you know, smarter, it just needs less of this backtracking and self-correction.
  8. 1 min later
  9. 0:06:54 · watch on youtube.com↗

    It's just like an observation to make. The reasoning chain is useful though, because it can help people understand what the model is doing.
  10. 1 min later
  11. 0:07:49 · watch on youtube.com↗

    So as we will see, the model is only trained on the final result, because as soon as we try to optimize the explanations, the model might also learn how to cheat more efficiently by hiding this intermediate chain.
  12. 1 min later
  13. 0:09:13 · watch on youtube.com↗

    By default, when you're using something like ChatGPT, they are usually filtering out everything in these think tags and show you only a summary of that. So I guess the intention is that people then can't steal these thinking traces to distill or train their own model, basically.
  14. 11 min later
  15. 0:19:51 · watch on youtube.com↗

    But you can imagine if you are having to rank answers, and you have to only so much time to rank them, you can't always double check which one is the correct answer. You're sometimes also just ranking them by, well, I like this explanation better, for example.
  16. 2 min later
  17. 0:21:33 · watch on youtube.com↗

    Now in reinforcement learning with verifiable rewards, we don't need the reward model. So it's actually much simpler. And that is also why it's so attractive and nice to use in practice,
  18. 2 min later
  19. 0:23:18 · watch on youtube.com↗

    One popular one is GRPO. So GRPO is, I would say it's a bit nicer and simpler than the classic PPO.
  20. 4 min later
  21. 0:27:06 · watch on youtube.com↗

    because it is one of the biggest topics in LLM fields where reinforcement learning is one of the cornerstone algorithms. And it does make sense to invest some time to understand it, because it's kind of like a important foundation.
  22. 6 min later
  23. 0:32:49 · watch on youtube.com↗

    There are a lot of papers or examples showing that actually without KL term on math models perform better. So it's not even essential to do that. It's kind of like a hyper parameter, whether to use it or not. It needs to be decided on a case by case basis.
  24. 11 min later
  25. 0:44:02 · watch on youtube.com↗

    It's actually useful to include end of text for the reinforcement learning because otherwise the model might unlearn to generate those.
  26. 10 min later
  27. 0:53:33 · watch on youtube.com↗

    Yeah, so if we drop the one over T, the token normalization, if we just have the sum, the sum log probability, that would be the sequence level one. And yeah, it's almost like a hyperparameter. And there are good reasons for each of those.
  28. 20 min later
  29. 1:13:48 · watch on youtube.com↗

    The optimizer, we use AdamW. That's a common, still one of the most widely used optimizers. Sometimes nowadays, also the Muon optimizer is pretty popular, except for the embedding layers.
  30. 12 min later
  31. 1:25:38 · watch on youtube.com↗

    But it's always like the I guess, the challenge for a book or for educational purposes is making it efficient, but also making it still easy to read and follow because usually optimization making things efficient means also making things more complicated.
  32. 1:25:59 · watch on youtube.com↗

    the original GPT-2 architecture, very simple, but also very expensive, if you would scale that up. So nowadays, LLMs get also very complicated, but topic for another time.