BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
4 korrents from this paper
In plain words
Train a language model to guess randomly hidden words using the full surrounding text so it sees both directions, then lightly adjust it for many different tasks. This lets one model handle question answering and other jobs with almost no custom design and raises scores a lot on eleven language tests, including 7.7 points to 80.5 on the main suite. It proposes a training method that builds on prior language models but pushes against their one-way limits.
Our summary of the paper, not the authors' words — written to be readable without the field's vocabulary, from the stored copy of the paper and nothing else. Drafted with xai:grok-4.5 and checked by a person. The authors' own sentences are the quotes below.
Near this, by wording
- Attention is all you need 2 claims
- Language models are few-shot learners (with 30 co-authors)
- Training language models to follow instructions with human feedback (with 19 co-authors)
- He, Zhang, Ren & Sun, "Deep Residual Learning for Image Recognition" (arXiv)
- Generative Adversarial Networks (with 7 co-authors)
Papers whose claims are worded most like this one's, found by the same hourly pass that draws the map. It is a measure of LANGUAGE, not of agreement or of citation: two papers can be near each other here and flatly contradict one another.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova did not write this page.
Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.