Our readingWhat is a korrent?
Research on transformer-style attention mechanisms has not yet reached diminishing returns compared to alternative approaches.
Drawn from what Gwern said
What these subjects mean
transformers The architecture that dropped recurrence in favour of attention, and that almost every large model since is built on.
innovation Where new things come from and how they spread: invention against diffusion, and being near the factory floor.
Mental modelS-curves and diminishing returns