What Kenton Lee thinks about scaling laws

@kenton-lee · 4 positions · On ppll ↗

One of the four authors of the 2018 paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", at Google AI Language, the affiliation printed on the paper.

Kenton Lee did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter scaling laws · Newest first

Topic

All 4LLMs 3neural networks 2scaling laws 1transformers 1

Sort

NewestOldest

Only Kenton Lee on scaling laws. See everyone on scaling laws

1 dated position, 2018, in their own words. Our reading of what Kenton Lee has said — not written or endorsed by them.

1 position so far — this page is not yet offered to search engines.

1. 11 Oct 2018

   Our readingCurrent techniques restrict pre-trained representation power because standard language models are unidirectional.

   LLMs

   Their wordsWe argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

   ↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org All 16 from this paper