korrents

Our readingWhat is a korrent?

Flash attention does more flops than standard attention yet runs significantly faster.

Drawn from what Andrej Karpathy said

What these subjects mean

transformers The architecture that dropped recurrence in favour of attention, and that almost every large model since is built on.

software performance Speed as a mark of quality: bloat, boot times, and software that got slower on hardware that got faster.

What Andrej Karpathy actually said

  1. Andrej Karpathy Founding member of OpenAI and former director of AI at Tesla
    what's remarkable about it is that uh flash attention actually if you just count the number of flops flash attention does more flops than this attention here but flash attention is actually significantly faster

    Watch from 2:01:28 plays here

    ↗Let's reproduce GPT-2 (124M)youtube.com

    9 Jun 2024 · video · 4h 01m · spoken · machine transcript

    All 14 korrents from this recordingthis one is 3rd

Added to korrents 9 Nov 2026 · How quotes work · Something wrong? Tell us

Do you hold this korrent?Do you also believe this?

I hold this too

Related korrents

Shared subject first, then similar wording. They may agree or disagree.

See this claim on the map