Our readingWhat is a korrent?
Gradient norm clipping guards against an unlucky bad batch producing a huge gradient that shocks the optimization.
Drawn from what Andrej Karpathy said
Subjectoptimizers
What this subject means
optimizers The algorithms that actually move a model's weights during training, and the trade-offs between them.