korrents

optimizers

The algorithms that actually move a model's weights during training, and the trade-offs between them.

What people on korrents have said about optimizers, newest first — 9 positions from 3 people.

FilterEveryone, all time
Who Everyone DKDiederik P. Kingma 4 JBJimmy Ba 4 RSRichard Sutton 1
When All time Past 1 yearPast 3 yearsPast 5 years
  1. RS

    Richard Sutton quoted

    Our readingGradient descent finds a solution to the problems a model has seen; nothing in the algorithm makes it pick the one that generalises well.

    Their wordsWell, there's nothing in them which will cause it to generalize. Well, the gradient descent will cause them to find a solution to the problems they've seen. And if there's only one way to solve them, you know, they they'll do it. But there are many ways to solve it. Some which generalize well, some which generalize poorly. There's nothing in them in the algorithms that will cause them to generalize well.

    ↗Richard Sutton – Father of RL thinks LLMs are a dead endyoutube.com 17th of 25 in this recording

    Interesting
  2. 11 years earlier
  3. DK

    Diederik P. Kingma quoted

    Our readingStochastic optimization can be done well using only adaptive estimates of the gradient's lower-order moments.

    Their wordsWe introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.

    ↗Kingma & Ba, "Adam: A Method for Stochastic Optimization" (arXiv)arxiv.org 1st of 8 in this piece

    Interesting
  5. JB

    Jimmy Ba quoted

    Our readingStochastic optimization can be done well using only adaptive estimates of the gradient's lower-order moments.

    Their wordsWe introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.

    ↗Kingma & Ba, "Adam: A Method for Stochastic Optimization" (arXiv)arxiv.org 2nd of 8 in this piece

    Interesting
  7. DK

    Diederik P. Kingma quoted

    Our readingAn optimizer that is invariant to rescaling the gradients and cheap in memory is what makes very large models practical to train.

    Their wordsThe method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters.

    ↗Kingma & Ba, "Adam: A Method for Stochastic Optimization" (arXiv)arxiv.org 3rd of 8 in this piece

    Interesting
  9. JB

    Jimmy Ba quoted

    Our readingAn optimizer that is invariant to rescaling the gradients and cheap in memory is what makes very large models practical to train.

    Their wordsThe method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters.

    ↗Kingma & Ba, "Adam: A Method for Stochastic Optimization" (arXiv)arxiv.org 4th of 8 in this piece

    Interesting
  11. DK

    Diederik P. Kingma quoted

    Our readingThe same optimizer should handle objectives that move under it and gradients that are noisy or sparse.

    Their wordsThe method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients.

    ↗Kingma & Ba, "Adam: A Method for Stochastic Optimization" (arXiv)arxiv.org 5th of 8 in this piece

    Interesting
  13. JB

    Jimmy Ba quoted

    Our readingThe same optimizer should handle objectives that move under it and gradients that are noisy or sparse.

    Their wordsThe method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients.

    ↗Kingma & Ba, "Adam: A Method for Stochastic Optimization" (arXiv)arxiv.org 6th of 8 in this piece

    Interesting
  15. DK

    Diederik P. Kingma quoted

    Our readingAn optimizer's hyper-parameters should mean something a practitioner can reason about, and should rarely need tuning.

    Their wordsThe hyper-parameters have intuitive interpretations and typically require little tuning.

    ↗Kingma & Ba, "Adam: A Method for Stochastic Optimization" (arXiv)arxiv.org 7th of 8 in this piece

    Interesting
  17. JB

    Jimmy Ba quoted

    Our readingAn optimizer's hyper-parameters should mean something a practitioner can reason about, and should rarely need tuning.

    Their wordsThe hyper-parameters have intuitive interpretations and typically require little tuning.

    ↗Kingma & Ba, "Adam: A Method for Stochastic Optimization" (arXiv)arxiv.org 8th of 8 in this piece

    Interesting