Current techniques restrict pre-trained representation power because standard language models are unidirectional.

Drawn from what Kristina Toutanova, Kenton Lee and 2 others said

What this subject means

scaling laws Whether more compute keeps buying more capability, and where the curve now bends -- pre-training, post-training, test-time.

What they actually said

Word for word, with the source under each one. They did not write this page.

Kristina Toutanova 11 Oct 2018 We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training. arxiv.org

One of the four authors of the 2018 paper "BERT:…

Their wordsWe argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

Where this was said

↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org

All 16 korrents from this piecethis one is 4th
Kenton Lee 11 Oct 2018 We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training. arxiv.org

One of the four authors of the 2018 paper "BERT:…

Their wordsWe argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

Where this was said

↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org

All 16 korrents from this piecethis one is 3rd
Ming-Wei Chang 11 Oct 2018 We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training. arxiv.org

One of the four authors of the 2018 paper "BERT:…

Their wordsWe argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

Where this was said

↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org

All 16 korrents from this piecethis one is 2nd
Jacob Devlin 11 Oct 2018 We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training. arxiv.org

One of the four authors of the 2018 paper "BERT:…

Their wordsWe argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

Where this was said

↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org

All 16 korrents from this piecethis one is 1st

Added to korrents 11 Oct 2018 · How quotes work · Something wrong? Tell us

Do you hold this korrent?Do you also believe this?

Related korrents

Our reading — they may agree, disagree or merely touch the same thing. Closest first: a shared subject counts for most, then how near the wording is.

On the map

Loading the map… or open it on its own page

Open the map on its own page →