korrents

Our readingWhat is a korrent?

Current techniques restrict pre-trained representation power because standard language models are unidirectional.

Drawn from what Kristina Toutanova, Kenton Lee and 2 others said

SubjectsLLMsscaling laws
What this subject means

scaling laws Whether more compute keeps buying more capability, and where the curve now bends -- pre-training, post-training, test-time.

What they actually said

  1. Kristina Toutanova One of the four authors of the 2018 paper "BERT:… We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training. BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org · 11 Oct 2018All 16 korrents from this piece
    Their wordsWe argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

    Where this was said

    ↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org

    11 Oct 2018

    All 16 korrents from this piecethis one is 4th

  2. Kenton Lee One of the four authors of the 2018 paper "BERT:… We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training. BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org · 11 Oct 2018All 16 korrents from this piece
    Their wordsWe argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

    Where this was said

    ↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org

    11 Oct 2018

    All 16 korrents from this piecethis one is 3rd

  3. Ming-Wei Chang One of the four authors of the 2018 paper "BERT:… We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training. BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org · 11 Oct 2018All 16 korrents from this piece
    Their wordsWe argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

    Where this was said

    ↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org

    11 Oct 2018

    All 16 korrents from this piecethis one is 2nd

  4. Jacob Devlin One of the four authors of the 2018 paper "BERT:… We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training. BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org · 11 Oct 2018All 16 korrents from this piece
    Their wordsWe argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

    Where this was said

    ↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org

    11 Oct 2018

    All 16 korrents from this piecethis one is 1st

Added to korrents 9 Sept 2026 · How quotes work · Something wrong? Tell us

Do you hold this korrent?Do you also believe this?

I hold this too

Related korrents

Shared subject first, then similar wording. They may agree or disagree.

See this claim on the map