korrents

Our readingWhat is a korrent?

Gradient norm clipping guards against an unlucky bad batch producing a huge gradient that shocks the optimization.

Drawn from what Andrej Karpathy said

optimizers
What this subject means

optimizers The algorithms that actually move a model's weights during training, and the trade-offs between them.

What Andrej Karpathy actually said

  1. Andrej Karpathy Founding member of OpenAI and former director of AI at Tesla
    sometimes you can get unlucky during your optimization maybe it's a bad data batch or something like that and if you get very unlucky in the batch you might get really high loss and really high loss could lead to a really high gradient and this could basically uh shock your model and shock the optimization

    Watch from 2:18:23 plays here

    ↗Let's reproduce GPT-2 (124M)youtube.com

    9 Jun 2024 · video · 4h 01m · spoken · machine transcript

    All 14 korrents from this recordingthis one is 6th

Added to korrents 9 Nov 2026 · How quotes work · Something wrong? Tell us

Do you hold this korrent?Do you also believe this?

I hold this too

Related korrents

Shared subject first, then similar wording. They may agree or disagree.

See this claim on the map