transformers
The architecture that dropped recurrence in favour of attention, and that almost every large model since is built on.
FilterEveryone, all time34 from 16 people
- SR
Sebastian Raschka quoted
AI research engineer working on large language models
Their wordsSo, in short, we can say that using looped transformers can improve model quality at a fixed compute budget if the model is large enough.
↗GPT-6 Astra, Looped Transformers, and Hidden Reasoningmagazine.sebastianraschka.com 3rd of 3 in this piece
- 8 weeks earlier
- DH
Dex Horthy quoted
Founder of the agent company HumanLayer
Their wordsI think context engineering has been so long lived because it's it's grounded in the fundamentals of how transformer attention works and until we have post transformer models or linear attention or whatever it is which who knows when that's going to happen context engineering will be interesting and important to anyone building on AI
↗Context engineering with Dex Horthyyoutube.com 3rd of 26 in this recording
- 2 months earlier
- SR
Sebastian Raschka quoted
Their wordsAs reasoning models and agent workflows keep more tokens around (for longer), KV-cache size, memory traffic, and attention cost quickly become the main constraints, and LLM developers are adding a growing number of architecture tricks to reduce those costs.
↗Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attentionmagazine.sebastianraschka.com 1st of 2 in this piece
- 23 months earlier
- AS
Aravind Srinivas quoted
Co-founder and CEO of Perplexity, an AI answer engine
Their wordsAnd that is the, I would say, it’s almost like the last answer. Nothing has changed since 2017 except maybe a few changes on what the nonlinearities are and how the square descaling should be done. Some of that has changed.
↗Aravind Srinivas: Perplexity CEO on Future of AI, Search & the Internet | Lex Fridman Podcast #434youtube.com 7th of 20 in this recording
- 3 years earlier
- GW
Gwern quoted
Pseudonymous essayist
Their wordsif the marginal return on this research direction had fallen below that of competitors, we would see those neglected directions invade Transformer topics-while we continue to see the reverse, and many applications as yet untouched by all the new approaches, suggesting that we still don't pay enough attention
↗April 2021 newslettergwern.substack.com 1st of 3 in this piece
- 4 months earlier
- GW
Gwern quoted
Their wordsThe fact that the prefix-tuning, by directly optimizing the prompt embeddings, yields better performance than even single optimized text prompts, suggests so.
↗Jan 2021 Gwern.net Newslettergwern.substack.com 3rd of 3 in this piece
- 2 years earlier
- JD
Jacob Devlin quoted
One of the four authors of the 2018 paper "BERT:…
Their wordsSuch restrictions are sub-optimal for sentence-level tasks, and could be very harmful when applying fine-tuning based approaches to token-level tasks such as question answering, where it is crucial to incorporate context from both directions.
↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org All 16 from this paper
- MC
Ming-Wei Chang quoted
One of the four authors of the 2018 paper "BERT:…
Their wordsSuch restrictions are sub-optimal for sentence-level tasks, and could be very harmful when applying fine-tuning based approaches to token-level tasks such as question answering, where it is crucial to incorporate context from both directions.
↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org All 16 from this paper
- KL
Kenton Lee quoted
One of the four authors of the 2018 paper "BERT:…
Their wordsSuch restrictions are sub-optimal for sentence-level tasks, and could be very harmful when applying fine-tuning based approaches to token-level tasks such as question answering, where it is crucial to incorporate context from both directions.
↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org All 16 from this paper
- KT
Kristina Toutanova quoted
One of the four authors of the 2018 paper "BERT:…
Their wordsSuch restrictions are sub-optimal for sentence-level tasks, and could be very harmful when applying fine-tuning based approaches to token-level tasks such as question answering, where it is crucial to incorporate context from both directions.
↗BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingarxiv.org All 16 from this paper
- 16 months earlier
- AV
Ashish Vaswani quoted
One of the eight authors of the 2017 transformer paper…
Their wordsWe propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
- NS
Noam Shazeer quoted
One of the eight authors of the 2017 transformer paper…
Their wordsWe propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
- NP
Niki Parmar quoted
One of the eight authors of the 2017 transformer paper…
Their wordsWe propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
- JU
Jakob Uszkoreit quoted
One of the eight authors of the 2017 transformer paper…
Their wordsWe propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
- LJ
Llion Jones quoted
One of the eight authors of the 2017 transformer paper…
Their wordsWe propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
- AG
Aidan N. Gomez quoted
One of the eight authors of the 2017 transformer paper…
Their wordsWe propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
- ŁK
Łukasz Kaiser quoted
One of the eight authors of the 2017 transformer paper…
Their wordsWe propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
- IP
Illia Polosukhin quoted
One of the eight authors of the 2017 transformer paper…
Their wordsWe propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
- AV
Ashish Vaswani quoted
Their wordsTo the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
- NS
Noam Shazeer quoted
Their wordsTo the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
- NP
Niki Parmar quoted
Their wordsTo the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
- JU
Jakob Uszkoreit quoted
Their wordsTo the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
- LJ
Llion Jones quoted
Their wordsTo the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
- AG
Aidan N. Gomez quoted
Their wordsTo the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
- ŁK
Łukasz Kaiser quoted
Their wordsTo the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
- IP
Illia Polosukhin quoted
Their wordsTo the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
- AV
Ashish Vaswani quoted
Their wordsAs side benefit, self-attention could yield more interpretable models.
- NS
Noam Shazeer quoted
Their wordsAs side benefit, self-attention could yield more interpretable models.
- NP
Niki Parmar quoted
Their wordsAs side benefit, self-attention could yield more interpretable models.
- JU
Jakob Uszkoreit quoted
Their wordsAs side benefit, self-attention could yield more interpretable models.