korrents

korrents · video

Let's reproduce GPT-2 (124M)

Andrej Karpathy · 4h 01m · youtube.com ↗

14 korrents from this recording

1h2h3h4h
Andrej Karpathy did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

  1. 1:32:19 · watch on youtube.com↗

    but empirically when you actually train with this you basically can't tell the difference so the reason I like tf32 is because if you can tolerate a little bit of a Precision fudge um then this is free
  2. 10 min later
  3. 1:41:56 · watch on youtube.com↗

    everyone started to train in fp16 but everyone had to use all these gradient scaling operations which are kind of annoying and it's an additional source of state and complexity
  4. 20 min later
  5. 2:01:28 · watch on youtube.com↗

    what's remarkable about it is that uh flash attention actually if you just count the number of flops flash attention does more flops than this attention here but flash attention is actually significantly faster
  6. 3 min later
  7. 2:04:42 · watch on youtube.com↗

    great example I think of being aware of um memory hierarchy the fact that flops don't matter uh the entire memory access pattern matters and that torch compile is amazing but there are many optimizations that are still available to us that potentially torch compile cannot find
  8. 2 min later
  9. 2:07:10 · watch on youtube.com↗

    so 64 is a beautiful nice number 128 is even nicer 256 is beautiful what makes these numbers beautiful is that there are many powers of two inside them you can divide by two many times
  10. 11 min later
  11. 2:18:23 · watch on youtube.com↗

    sometimes you can get unlucky during your optimization maybe it's a bad data batch or something like that and if you get very unlucky in the batch you might get really high loss and really high loss could lead to a really high gradient and this could basically uh shock your model and shock the optimization
  12. 5 min later
  13. 2:22:57 · watch on youtube.com↗

    I just don't really love using that code because honestly it's like five lines of code and I fully understand what's happening inside these lines so I don't love to use abstractions
  14. 3 min later
  15. 2:26:10 · watch on youtube.com↗

    people have come up with all kinds of uh other learning rate schedules um and this is kind of like an active area of uh research as to which one is the most effective at train these networks
  16. 21 min later
  17. 2:47:16 · watch on youtube.com↗

    for this we are going to be using the distributed data parallel from pytorch there's also a legacy data parallel which I recommend you not use
  18. 16 min later
  19. 3:03:32 · watch on youtube.com↗

    I just really don't love this I I just really don't like it uh the fact that you have to copy paste your code here and use a context manager and this is just super ugly
  20. 8 min later
  21. 3:11:11 · watch on youtube.com↗

    basically it's not a very high quality data set all by itself because it is extremely noisy this is a completely random subset of the internet and it's much worse than you think
  22. 9 min later
  23. 3:20:07 · watch on youtube.com↗

    I think 715 by the way is very uh mild and this could be made significantly more aggressive probably even like 100 is good enough
  24. 11 min later
  25. 3:31:11 · watch on youtube.com↗

    basically the 4% the last 4% is probably errors in the data set or the questions are really really hard and so basically this data set is kind of crushed with respect to language models
  26. 14 min later
  27. 3:45:32 · watch on youtube.com↗

    fine web edu is all English it's not multilingual and there's not that much math and code um and so math and code and multilingual could have been stealing capacity from the original gpt2 model