What Sebastian Raschka thinks about transformers

@sebastian-raschka · 16 positions · On ppll ↗

AI research engineer working on large language models. He writes the Ahead of AI newsletter and is the author of Build a Large Language Model (From Scratch).

Sebastian Raschka did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter transformers · Newest first

Topic

All 16LLMs 13AI agents 2coding agents 2scaling laws 2transformers 2AI writing 1cybersecurity 1developer tools 1neural networks 1

2 more topics

open source 1reinforcement learning 1

Sort

NewestOldest

Only Sebastian Raschka on transformers. See everyone on transformers

2 dated positions, 2026, in their own words. Our reading of what Sebastian Raschka has said — not written or endorsed by them.

2 positions so far — this page is not yet offered to search engines.

1. 9 Sept 2026

   Our readingLooped/recurrent-depth transformer architectures can improve model quality at a fixed compute budget when the model is large enough.

   Their wordsSo, in short, we can say that using looped transformers can improve model quality at a fixed compute budget if the model is large enough.

   ↗GPT-6 Astra, Looped Transformers, and Hidden Reasoningmagazine.sebastianraschka.com 3rd of 3 in this piece

2. 4 months earlier

3. 16 May 2026

   Our readingAs reasoning models and agents retain more tokens for longer, KV-cache size and attention cost become the main constraint on scaling context.

   LLMs

   Their wordsAs reasoning models and agent workflows keep more tokens around (for longer), KV-cache size, memory traffic, and attention cost quickly become the main constraints, and LLM developers are adding a growing number of architecture tricks to reduce those costs.

   ↗Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attentionmagazine.sebastianraschka.com 1st of 2 in this piece