korrents

korrents · video

But what is cross-entropy? | Compression is Intelligence Part 2

Grant Sanderson · 33m · youtube.com ↗

8 korrents from this recording

Grant Sanderson did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

  1. 1:42 · watch on youtube.com↗

    What I like about this paper is what a pure example it is of when the theory and the engineering behind compression can be surprisingly useful to tasks which would seem to fall in the domain of machine learning and artificial intelligence.
  2. 1 min later
  3. 2:34 · watch on youtube.com↗

    But whenever you see the same formula pop up in two separate contexts, it's math's way of kind of winking at you and hinting at a connection.
  4. 2 min later
  5. 4:33 · watch on youtube.com↗

    I mentioned how I kind of wish that history had unfolded in such a way that we call it the log base 1 half of the probability, since it's really just asking how many times do you chop things in half to get down to a certain amount.
  6. 12 min later
  7. 16:53 · watch on youtube.com↗

    But the point I want to emphasize is that for you, the engineer, trying to train a model, once you define a good loss function, you're essentially done. The standard machinery of deep learning takes it from there.
  8. 3 min later
  9. 20:13 · watch on youtube.com↗

    and getting this to run at the mind-boggling scales used in practice is a complete engineering feat. But at the level of abstraction of just thinking of loss functions, it really is remarkably simple. It really is just the average information per token
  10. 1 min later
  11. 21:22 · watch on youtube.com↗

    But if that's all there was to it, this would be very lame, at least in my opinion. This does nothing to explain why you're using cross-entropy in the first place, and if it all just evaporates away anyway, why not just be straightforward?
  12. 6 min later
  13. 27:28 · watch on youtube.com↗

    When you think about it, this gives you something much, much richer than ordinary pre-training, since pre-training is just giving full weight to only one true next token, the one that happens to be there.
  14. 1 min later
  15. 28:56 · watch on youtube.com↗

    In the same way that whenever you see pi show up in math, even when it seems unrelated to circles, there's always going to be some connection to circles hiding there, I think whenever you see cross-entropy pop up, there will be a way to connect it to compression.