Understanding LSTM Networks

Christopher Olah · 27 Aug 2015 · colah.github.io ↗

2 korrents from this piece

Christopher Olah did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

Our reading
Standard recurrent neural networks are theoretically capable of representing long-term dependencies but fail to learn them from data in practice.

Their wordsIn theory, RNNs are absolutely capable of handling such "long-term dependencies." A human could carefully pick parameters for them to solve toy problems of this form. Sadly, in practice, RNNs don't seem to be able to learn them.
Our reading
LSTM architectures are designed so that retaining information over long time spans is their default behavior rather than something they must struggle to learn.

Their wordsLSTMs are explicitly designed to avoid the long-term dependency problem. Remembering information for long periods of time is practically their default behavior, not something they struggle to learn!