Our readingWhat is a korrent?
Flash attention does more flops than standard attention yet runs significantly faster.
Drawn from what Andrej Karpathy said
What these subjects mean
transformers The architecture that dropped recurrence in favour of attention, and that almost every large model since is built on.
software performance Speed as a mark of quality: bloat, boot times, and software that got slower on hardware that got faster.