What Christopher Olah thinks about neural networks

@chris-olah · 11 positions · 0 changes of mind

Interpretability researcher at Anthropic, which he co-founded; before that at OpenAI and Google Brain.

Everything they publish, on ppll ↗

Christopher Olah did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

2 dated positions, 2015, in their own words. Our reading of what Christopher Olah has said — not written or endorsed by them.

2 positions so far — this page is not yet offered to search engines.

1. 27 Aug 2015

   Our readingStandard recurrent neural networks are theoretically capable of representing long-term dependencies but fail to learn them from data in practice.

   Their wordsIn theory, RNNs are absolutely capable of handling such "long-term dependencies." A human could carefully pick parameters for them to solve toy problems of this form. Sadly, in practice, RNNs don't seem to be able to learn them.

   ↗Understanding LSTM Networkscolah.github.io 1st of 2 in this piece

3. 27 Aug 2015

   Our readingLSTM architectures are designed so that retaining information over long time spans is their default behavior rather than something they must struggle to learn.

   Their wordsLSTMs are explicitly designed to avoid the long-term dependency problem. Remembering information for long periods of time is practically their default behavior, not something they struggle to learn!

   ↗Understanding LSTM Networkscolah.github.io 2nd of 2 in this piece